Sea Monster of the Lost Dependency (a real story)

It was just another cloudy evening in the home office. Code written, recipes updated — all seemed ready to go. Yet something felt not right: a strange certainty that things were just off. Maybe it was the fact we still sat on a few-months-old Kirkstone release while fetching dependencies from the Internet sea. So just to calm my nerves I decided to make a clean build. Nothing is better than a fresh clean image waiting for you at dawn. Unaware of what was coming, I went to sleep.

In the morning there was a sea monster waiting for me, right in the middle of the console.

ERROR: efibootmgr-17-r0 do_fetch: Fetcher failure: Unable to find revision e067160ecef8208e1944002e5d50b275733211fb in branch master even from upstream
ERROR: efibootmgr-17-r0 do_fetch: Bitbake Fetcher Error: FetchError('Unable to fetch URL from any source.', 'git://github.com/rhinstaller/efibootmgr.git;protocol=https;branch=master')
ERROR: Logfile of failure stored in: /work/clean_run/nwbe-ems-yocto/build/tmp/work/corei7-64-poky-linux/efibootmgr/17-r0/temp/log.do_fetch.6077
ERROR: Task (/work/clean_run/nwbe-ems-yocto/build/../layers/poky/meta/recipes-bsp/efibootmgr/efibootmgr_17.bb:do_fetch) failed with exit code '1'

Not often does a tech sailor meet such a beast in his life.* It sat there looking at me with its failed-fetch eyes. Smiling and waiting for its food — my precious time.

A few hours later I knew where it had come from.

There was once an old repository. Stable and proven, like an island carved from rock. github.com/rhinstaller/efibootmgr. If you try to find it now, you will discover that it was quietly moved to github.com/rhboot/efibootmgr. Not really a big surprise — things do get renamed all the time.**

But that was not the source of the problem (it will be, the day someone decides to remove the redirection). The real troublemaker was the master branch. Or to be exact: the fact that it was gone.

Most of my wasted time went into trying to figure out where the branch had gone. My suspicion was that it was deleted long ago and had worked only because of some strange GitHub magic similar to this one — making it look like everything still worked in the old way, for as long as the spell held. Whatever it was — it was gone. And tech sea monsters wait exactly for such occasions. A fix eventually landed in Poky, but the world does not pause while every project catches up. Our project did not, and I had no intention of making that my first task of the morning.

What saved me was my laziness. Deleting old files takes too long, so there were always old builds lying somewhere on the filesystem. From there I could use the existing downloads and bypass the whole “can’t fetch it” drama. Image got baked, I could do my tests, and the next day I started by updating all the layers properly — the way it should have been done months ago.

Now comes the lesson:

The internet is a dangerous place. Repositories move. Branches vanish. Magic spells expire. Whatever lives outside your local build environment needs to be treated as if it could vanish at any time without any warning. So when the old tech sailor says you should keep a backup of the Internet — don’t laugh at him and just use BB_GENERATE_MIRROR_TARBALLS.

* mostly because we follow best practices when it comes to dependency management in Yocto
** old sailors might remember the Freescale-to-NXP storm that rolled through years ago


Leave a comment