Lev Lazinskiy

Multiline Bash in CircleCI YAML: How I almost invented the slashtag

| circleci | dagger | yaml |

Warning this post contains a lot of YAML

We talk a lot about push and pray at Dagger but I spent the better part of the last few days saying every prayer I know trying to get my CircleCI config to work properly with multi-line bash commands inside of a single YAML block.

It was particularly painful because this was for a job that only runs on git tags and I was trying to publish a new release, so every time it didn’t work I had to remove the git tag, make the tweak to the yaml file, and then push the same tag back out.

I had to look it up, but the steps to remove a remote and local tag are:

git tag -d $TAG
git push origin --delete $TAG

Then do some work, commit it, rebase it, tag it, and force push it out again.

If this was for a project that actually mattered, I think it would have been much more painful because I would not have rebased and force pushed and instead I would have had at least 6 new patch versions of my program with the CI config file being the only thing that changes.

What I was trying to do

A few weeks ago I published a mastodon module that sends a toot out every time I release a new tagged version of a dagger module in my daggerverse repo. This takes advantage of CircleCI tag filtering to only execute this job when I have a git tag.

This function calls my dagger module with a bunch of parameters and I wanted to split them out into multiple lines so that its easier to read.

The first version that I tried did work, but it did not allow me to add the # symbol because bash (or yaml) was treating this as a comment and breaking. Since I was trying to add the #dagger tag to my toots this needed to be fixed. I was able to get it to work by escaping the tag with \#dagger. This kind of worked, but the escape symbol was not escaped, so my toots looked a bit weird.

works

command: dagger -m github.com/levlaz/daggerverse/mastodon call toot
    --server="https://hachyderm.io"
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
    --client-secret=env:MASTODON_CLIENT_SECRET
    --access-token=env:MASTODON_ACCESS_TOKEN
    --msg="I just published a new version of a Dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"

breaks because # is not escaped

command: dagger -m github.com/levlaz/daggerverse/mastodon call toot
    --server="https://hachyderm.io"
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
    --client-secret=env:MASTODON_CLIENT_SECRET
    --access-token=env:MASTODON_ACCESS_TOKEN
    --msg="I just published a new version of a #dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"

works but /# is not escaped

command: dagger -m github.com/levlaz/daggerverse/mastodon call toot
    --server="https://hachyderm.io"
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
    --client-secret=env:MASTODON_CLIENT_SECRET
    --access-token=env:MASTODON_ACCESS_TOKEN
    --msg="I just published a new version of a /#dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"

What didn’t work

Then I remembered something about the | symbol allowing to do more stuff so I tried that. This didn’t work because my module kept complaining that I was not passing any parameters.

command: |
    dagger -m github.com/levlaz/daggerverse/mastodon call toot
        --server="https://hachyderm.io"
        --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI"
        --client-secret=env:MASTODON_CLIENT_SECRET
        --access-token=env:MASTODON_ACCESS_TOKEN
        --msg="I just published a new version of a /#dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"    

Next, I remembered that maybe I needed to escape them somehow so I added some escape symbols. I could not find any info on how to do this in the docs, but this SO post gave me a hint about using double slashes, so I tried that.

command: |
    dagger -m github.com/levlaz/daggerverse/mastodon call toot \\
        --server="https://hachyderm.io" \\
        --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI" \\
        --client-secret=env:MASTODON_CLIENT_SECRET \\
        --access-token=env:MASTODON_ACCESS_TOKEN \\
        --msg="I just published a new version of a /#dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse"    

This did not work, it still complained about not passing parameters. Then I tried single slashes. No luck. Then I tried to run the command locally and it worked just fine. I was completely lost and now we were in our 5th iteration of doing the force push rebase tag prayer.

What did work

After staring at my screen for a bit, I decided to try one last thing before just accepting that all of my toots were going to have \# in them, perhaps I could have started a new trend! Slashtag?

I decided to remove all the extra indentation on the subsequent parts of the command.

command: |
    dagger -m github.com/levlaz/daggerverse/mastodon call toot \
    --server="https://hachyderm.io" \
    --client-id="-3UXCeKKYu3U1z2ZANnONS-Artc-WbMuMRKOuSvzuiI" \
    --client-secret=env:MASTODON_CLIENT_SECRET \
    --access-token=env:MASTODON_ACCESS_TOKEN \
    --msg="I just published a new version of a #dagger module, $CIRCLE_TAG, check it out! https://github.com/levlaz/daggerverse/releases/tag/$CIRCLE_TAG"    

This worked! And I still have no idea why. Its possible I am just stupid, but I have no idea what the combination of bash, yaml, and the CircleCI engine actually wants from me.

I’m dreaming of a world where no one has to deal with this yaml hell anymore. We’re halfway there, and if you follow me on mastodon you’ll be the first to hear about it, without the slashtag.

Articles from blogs I follow around the net

The Converse Experiment: Report One

How long does a pair of Converse last?

via Robb Knight • Posts • Atom Feed May 30, 2024

Pluralistic: The Pizzaburger Presidency (29 May 2024)

Today's links The Pizzaburger Presidency: Why the Biden administration talks so very little about the very small number of very good things it has done. Hey look at this: Delights to delectate. This day in history: 2004, 2009, 2014, 2019, 2023 Upcomin…

via Pluralistic: Daily links from Cory Doctorow May 29, 2024

Writing a Unix clone in about a month

I needed a bit of a break from “real work” recently, so I started a new programming project that was low-stakes and purely recreational. On April 21st, I set out to see how much of a Unix-like operating system for x86_64 targets that I could put together in …

via Drew DeVault's blog May 24, 2024

Generated by openring