Post Detail

Are your DevOps pipelines taking 10+ or even 20+ minutes to finish? Do your developers need to grab a coffee while waiting for the PR Build to finish? You are not alone! Long running Continuous Integration / Continuous Deployment (or CI/CD for short) pipelines are an innovation killer and more and more companies are loosing thousands of dollars in productivity value every moth because of it.. In this short articles, I will show you three relatively simple steps you can take to bring the CI/CD pipeline duration to single digit duration. The steps to follow are:

Step 1: Question every command
Step 2: Split your pipelines
Step 3: Optimize what is left

Let’s get into some details about these steps:

Step 1: Question every command

It may sound trivial, but you’d be surprised to find out how often pipelines contain several commands/steps that are not providing much value there. Start by reading your CI/CD pipelines and getting deeper understanding of every command in every step. Question every one of them. Check if you have duplicate commands, or commands that are already executed somewhere else as well, if yes, get rid of them. One such example is, it is quite often the case that build commands are run on multiple places, or the same automated tests are run in multiple parts of the process. These duplications are mostly pure waste and they are very easy to be overlooked.

Step 2: Split your pipelines

One of the common pitfalls I see is having CI/CD pipelines that do a lot. One pipeline ruling them all, compiling the code, running the tests, doing static analysis, deploying to various environments, etc. Just as in programming language classes, the pipelines should also have a single responsibility. Make your pipelines specialized for one single business value. I would recommend at least to split your integration pipeline (CI) and deployment pipelines (CD), but depending on the complexity of your solution, you may even want to go even more fine grained splitting. Specialized pipelines with single responsibility provide fast feedback and enable you to eliminate waste. Below I describe a concrete example scenario on how I would do the splitting.

Example scenario: A containerized java micro service application exposing an API, would have the following pipelines:

CI pipeline for branch commits and Pull Requests (PRs)

Triggered after every push to the remote (feature) branch in source control, but also for every PR created, this is the check with the remote repository if “code on my computer would break the remote code”. It should run and finish ideally in less than 5-6 minutes. The pipeline should minimally do the following:

1. Build the code
2. Run unit tests
3. Run integration tests (including persistence tests)

I would highly recommend to add also checks on code standards and vulnerability analysis. The steps you should cover are:

1. Run Linter(s)
2. Run static code analysis
3. Run security check

It should be the rule that this pipeline is required to pass successfully before a change can be merged to master/main branch. The only exception to this rule might be hot-fixes or emergency patches, though I would recommend against it (but if everything is burning, you don’t have much to lose, therefore can tolerate it ;)).

CD pipeline

This pipeline is triggered after a successful merge to the main branch (or manually). The focus of the pipeline is on preparing the deployment and distributing the change, therefore, it should trust the previous pipelines on testing the code and should not repeat any of the steps of CI pipelines. The CD pipeline should:

1. Build the container image
2. Push the image to the container registry
3. Trigger deployment (this could be 1 step if there is one deployment environment, or a multi-step if there are several environments e.g. dev, test, staging, production)

Notice, in this level you don’t run unit tests anymore because you know they are run successfully before, otherwise the merge would not be possible.

For this splitting approach to work, the major requirement is that no one should be able to do direct commit to the deployment or release branches, otherwise untested code may slip into your environments leading to nasty situations. This can be relatively easily established in major code repository solutions by applying security policies on your branches.

Beyond CI/CD pipelines

Unfortunately, many solutions do not have any end-to-end, contract, or smoke tests. If you have them, you are one of the lucky ones. That said, these tests usually are slow and take a lot of time to run. More important, you do not need them to run on every commit, so move them to a specialize pipeline and schedule them to run separately. Depending on the necessity for your solution, this can be after a deployment to every environment.

Step 3: Optimize what is left

After you know that you have eliminated all the waste in the process, then focus on squeezing out all the capacity you can by further optimizing the individual steps of the pipelines. Depending on your tech stack, there are different optimization aspects you would need to check. Some common things to do are:

• Can you run your commands faster? – check the options your tools offer. E.g. in some tech stacks, you have different build commands per dev and prod environments
• Cache, cache, cache. Those container image layers, node packages, nuget packages, ruby gems, libraries, etc., most of them can be cached. Cache everything you can (consciously), this will speed up the pipelines significantly
• Parallelize the automated tests. Many test framework offer parallelization of the execution of automated tests sparing you some minutes
• Adjust the capacity of build workers. Sometimes scaling up your build agent workers can speed up your pipelines significantly

You might need to iterate over this steps in a couple of iterations to achieve the optimal result, but it will pay off. Even if your gain is 5 mins per run, if you run the pipelines twice a day per person and you have 4 devs in the team, you will spare about 14 hours of build time in a month, this is a huge gain.

I hope you have found the tips useful. If you need help in optimizing your pipelines, I’d be happy to help. Book a call with me and we can discuss how can I help you with streamlining your DevOps processes.