Dealing With Flakey CI Commands With a Retry Loop in Bash
One of the most frustrating things to deal with in Continuous Integration is flakey commands. Whether it’s flakey tests, or intermittent networking issues, when your build fails for issues outside of your control not only does it cause frustration, it reduces the trust in your CI process.
One strategy for dealing with this type of issue is to introduce some retry logic into your commands. This can easily be accomplished with good old bash.
For example, pretend that I have $FLAKEY_COMMAND
and I want to retry it three times before finally failing my build. I could wrap the whole thing up in a bash loop like this.
counter=1
max=3
$FLAKEY_COMMAND
while [[ $? -ne 0 && $counter -lt $max ]]; do
counter=$((counter+1))
$FLAKEY_COMMAND
done
This script will run my command, if the exit code (the output of $?) is non zero (i.e something went wrong) and my counter is less than three, then it will retry the command. You can increase or decrease the number of attempts by adjusting the max
variable.
This is not a foolproof strategy, but is one approach to handle flakey commands in your CI pipeline.
Thank you for reading! Share your thoughts with me on mastodon or via email.
Check out some more stuff to read down below.
Most popular posts this month
- 3 Years
- Moving to New Jersey
- SQLite DB Migrations with PRAGMA user_version
- Setting up ANTLR4 on Windows
- My Custom Miniflux CSS Theme
Recent Favorite Blog Posts
This is a collection of the last 8 posts that I bookmarked.
- The Framework 13 has a new high-res screen! from David Heinemeier Hansson
- 40 Thoughts At 40 from Blog – Brad Frost
- To Blog or to Social-Post from jwz
- Gotchas with SQLite in Production from Anže’s Blog
- Petter Reinholdtsen: More than 200 orphaned Debian packages moved to git, 216 to go from Planet Debian
- Eli Bendersky: You don't need virtualenv in Go from Planet Python
- Introducing Writebook from Jason Fried
- Reasons to use your shell's job control from Julia Evans
Articles from blogs I follow around the net
SEC charges Galois Capital, Galois settles
Eighteen months after the crypto-focused algorithmic trading fund Galois Capital shut down, explaining that they had lost around $40 million in the FTX collapse, the SEC has filed a lawsuit against the firm for failing to properly cu…
via Web3 is Going Just Great September 3, 2024MagSafe Miscellania
Turns out MagSafe is pretty cool
via Robb Knight • Posts • Atom Feed September 3, 2024Progress
As I'm getting older a lot of my social circles are becoming ever more conservative. The focus shifts from building with ambition to fiercly protecting what one has achieved. Shifting the mind on protectionism makes one consider all that can cause dama…
via Armin Ronacher's Thoughts and Writings September 3, 2024Generated by openring