Lev Lazinskiy

Multiline Bash in CircleCI YAML: How I almost invented the slashtag

Warning this post contains a lot of YAML

We talk a lot about push and pray at Dagger but I spent the better part of the last few days saying every prayer I know trying to get my CircleCI config to work properly with multi-line bash commands inside of a single YAML block.

It was particularly painful because this was for a job that only runs on git tags and I was trying to publish a new release, so every time it didn’t work I had to remove the git tag, make the tweak to the yaml file, and then push the same tag back out.

I had to look it up, but the steps to remove a remote and local tag are:

git tag -d $TAG
git push origin --delete $TAG

Then do some work, commit it, rebase it, tag it, and force push it out again.

If this was for a project that actually mattered, I think it would have been much more painful because I would not have rebased and force pushed and instead I would have had at least 6 new patch versions of my program with the CI config file being the only thing that changes.

What I was trying to do

A few weeks ago I published a mastodon module that sends a toot out every time I release a new tagged version of a dagger module in my daggerverse repo. This takes advantage of CircleCI tag filtering to only execute this job when I have a git tag.

This function calls my dagger module with a bunch of parameters and I wanted to split them out into multiple lines so that its easier to read.

The first version that I tried did work, but it did not allow me to add the # symbol because bash (or yaml) was treating this as a comment and breaking. Since I was trying to add the #dagger tag to my toots this needed to be fixed. I was able to get it to work by escaping the tag with \#dagger. This kind of worked, but the escape symbol was not escaped, so my toots looked a bit weird.

works

command: dagger -m github.com/levlaz/daggerverse/mastodon call toot
    --server="https://hachyderm.io"
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
    --client-secret=env:MASTODON_CLIENT_SECRET
    --access-token=env:MASTODON_ACCESS_TOKEN
    --msg="I just published a new version of a Dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"

breaks because # is not escaped

command: dagger -m github.com/levlaz/daggerverse/mastodon call toot
    --server="https://hachyderm.io"
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
    --client-secret=env:MASTODON_CLIENT_SECRET
    --access-token=env:MASTODON_ACCESS_TOKEN
    --msg="I just published a new version of a #dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"

works but /# is not escaped

command: dagger -m github.com/levlaz/daggerverse/mastodon call toot
    --server="https://hachyderm.io"
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
    --client-secret=env:MASTODON_CLIENT_SECRET
    --access-token=env:MASTODON_ACCESS_TOKEN
    --msg="I just published a new version of a /#dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"

What didn’t work

Then I remembered something about the | symbol allowing to do more stuff so I tried that. This didn’t work because my module kept complaining that I was not passing any parameters.

command: |
    dagger -m github.com/levlaz/daggerverse/mastodon call toot
        --server="https://hachyderm.io"
        --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
        --client-secret=env:MASTODON_CLIENT_SECRET
        --access-token=env:MASTODON_ACCESS_TOKEN
        --msg="I just published a new version of a /#dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"    

Next, I remembered that maybe I needed to escape them somehow so I added some escape symbols. I could not find any info on how to do this in the docs, but this SO post gave me a hint about using double slashes, so I tried that.

command: |
    dagger -m github.com/levlaz/daggerverse/mastodon call toot \\
        --server="https://hachyderm.io" \\
        --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI" \\
        --client-secret=env:MASTODON_CLIENT_SECRET \\
        --access-token=env:MASTODON_ACCESS_TOKEN \\
        --msg="I just published a new version of a /#dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"    

This did not work, it still complained about not passing parameters. Then I tried single slashes. No luck. Then I tried to run the command locally and it worked just fine. I was completely lost and now we were in our 5th iteration of doing the force push rebase tag prayer.

What did work

After staring at my screen for a bit, I decided to try one last thing before just accepting that all of my toots were going to have \# in them, perhaps I could have started a new trend! Slashtag?

I decided to remove all the extra indentation on the subsequent parts of the command.

command: |
    dagger -m github.com/levlaz/daggerverse/mastodon call toot \
    --server="https://hachyderm.io" \
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI" \
    --client-secret=env:MASTODON_CLIENT_SECRET \
    --access-token=env:MASTODON_ACCESS_TOKEN \
    --msg="I just published a new version of a #dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse/releases/tag/$CIRCLE_TAG"    

This worked! And I still have no idea why. Its possible I am just stupid, but I have no idea what the combination of bash, yaml, and the CircleCI engine actually wants from me.

I’m dreaming of a world where no one has to deal with this yaml hell anymore. We’re halfway there, and if you follow me on mastodon you’ll be the first to hear about it, without the slashtag.