Dealing With Flakey CI Commands With a Retry Loop in Bash
One of the most frustrating things to deal with in Continuous Integration is flakey commands. Whether it’s flakey tests, or intermittent networking issues, when your build fails for issues outside of your control not only does it cause frustration, it reduces the trust in your CI process.
One strategy for dealing with this type of issue is to introduce some retry logic into your commands. This can easily be accomplished with good old bash.
For example, pretend that I have $FLAKEY_COMMAND and I want to retry it three times before finally failing my build. I could wrap the whole thing up in a bash loop like this.
counter=1
max=3
$FLAKEY_COMMAND
while [[ $? -ne 0 && $counter -lt $max ]]; do
counter=$((counter+1))
$FLAKEY_COMMAND
done
This script will run my command, if the exit code (the output of $?) is non zero (i.e something went wrong) and my counter is less than three, then it will retry the command. You can increase or decrease the number of attempts by adjusting the max variable.
This is not a foolproof strategy, but is one approach to handle flakey commands in your CI pipeline.
Thank you for reading! Share your thoughts with me on bluesky, mastodon, or via email.
Check out some more stuff to read down below.
Most popular posts this month
- Now What?
- SQLite DB Migrations with PRAGMA user_version
- Setting up ANTLR4 on Windows
- Dagger Feels Like Magic
- Types and Roles If Not Exists in PostgreSQL
Recent Favorite Blog Posts
This is a collection of the last 8 posts that I bookmarked.
- Useful Bluesky Tools from Robb Knight • Posts • Atom Feed
- Re: Bluesky from Colin Devroe
- From the Red Hell to the Sky of Blue from Straphanger
- We don’t need to use what we make from Derek Sivers blog
- Ubuntu Summit 2024: A joyful experience filled with sorrow from Planet KDE | English
- Sabotage from jwz
- What if My Tribe Is Wrong? from Armin Ronacher's Thoughts and Writings
- It’s the “1998” of the AI Revolution. So Why Can I Safely Ignore It? from The Internet Review
Articles from blogs I follow around the net
[RODEN] Enrique Allen
Roden Readers — The first memory I have of Enrique Allen is from the campus of Stanford. He had just graduated from the d.school and was teaching part-time. We were about to start working together. He was all bounding lightness. That’s the first image: Jum…
via Craig Mod — Writer + Photographer November 21, 202417/11/2024
# Back in May I wrote about being inspired to write a track based on a YouTube comment. I recorded a test not long after and built on that into June. Then the breakdown struck and I had a massive crisis of confidence alongside the depression and anxiety. I …
via Colin Walker - Daily Feed November 21, 2024Cold reading an ADHD affliction
I'm sure there are truly pathological cases of ADHD out there, and maybe taking amphetamines really is a magic pill for some folks. But there clearly is also an entire cottage industry cropping up around convincing perfectly normal people that they …
via David Heinemeier Hansson November 20, 2024Generated by openring