Dealing With Flakey CI Commands With a Retry Loop in Bash

| ci | devops |

One of the most frustrating things to deal with in Continuous Integration is flakey commands. Whether it’s flakey tests, or intermittent networking issues, when your build fails for issues outside of your control not only does it cause frustration, it reduces the trust in your CI process.

One strategy for dealing with this type of issue is to introduce some retry logic into your commands. This can easily be accomplished with good old bash.

For example, pretend that I have $FLAKEY_COMMAND and I want to retry it three times before finally failing my build. I could wrap the whole thing up in a bash loop like this.

counter=1
max=3
$FLAKEY_COMMAND

while [[ $? -ne 0 && $counter -lt $max ]]; do
    counter=$((counter+1))
    $FLAKEY_COMMAND
done

This script will run my command, if the exit code (the output of $?) is non zero (i.e something went wrong) and my counter is less than three, then it will retry the command. You can increase or decrease the number of attempts by adjusting the max variable.

This is not a foolproof strategy, but is one approach to handle flakey commands in your CI pipeline.

Thank you for reading! Share your thoughts with me on mastodon or via email.

Check out some more stuff to read down below.

Most popular posts this month

Recent Favorite Blog Posts

This is a collection of the last 8 posts that I bookmarked.

Articles from blogs I follow around the net

Supermicro NVIDIA GB200 NVL72 System at Computex 2024

We checked out the Supermicro NVIDIA GB200 NVL72 rack at Computex 2024. This is a prime example of why power is becoming such a big deal The post Supermicro NVIDIA GB200 NVL72 System at Computex 2024 appeared first on ServeTheHome.

via ServeTheHome July 22, 2024

Forum

Community is one of the main topics I touch on here. My community, my social media is an email and iMessage, but I thought I could do more. I did such an experiment last year and I quit soon after. I don't think I was ready for it. Today I still don&#…

via Michal Zelazny July 21, 2024

Weekly Update 409

It feels weird to be writing anything right now that isn't somehow related to Friday's CrowdStrike incident, but given I recorded this video just a few hours before all hell broke loose, it'll have to wait until next week. This week, the issue…

via Troy Hunt July 21, 2024

Generated by openring