My workflow as an embedded Linux developer

I’m rarely one to follow the beaten path when I find something new, and kernel development was no different. It usually takes me weeks or months of pain before I give up my home-grown workflow for something a bit more standard, nonetheless I’ve picked up a few tricks along the way…

On optimising my workflow

My motivation to customise and fix things usually arises when it’s juuust slightly too annoying - when I realise I’ve been relying on a huge shell oneliner which doesn’t exist outside of my .zsh_history, or when I find myself typing fastboot flash boot some/annoyingly/long/path/boot.img reboot one too many times. I’ll stop what I’m doing, open a new window and write a script to alleviate some common patterns.

aflash.sh

This exact occurance lead me to write this fastboot wrapper which lets me perform all the fastboot actions I perform regularly with a whole lot less typing. The above example would just be aflash.sh -b. I optimise this further by using aliases for common combinations of flags, e.g. aflashall is aliased to aflash.sh -t -s -v -b, the -t flag runs adb reboot bootloader. As a result I’ve gone from regularly picking up a device to hard-reboot into fastboot - or having to type out adb reboot bootloader manually, to just, having it happen for me. In the ~5 months I’ve had this script alone I have undoubtably saved a significant amount of time which would otherwise be spent holding buttons or typing.

How often DO you actually reboot your device?

Fortunately (or unfortunately) the OnePlus 6 bootloader spits out a “boot count” over UART while the bootloader is initialising, this was what I got recently:

This… is a little scary. If you’re interested, you can get a full op6 bootloader log here

I bought my OnePlus 6 brand new in the Summer of 2019, and I’ve been doing Linux development for it since around November that year, I got into mainline Linux and postmarketOS for it around April 2020.

Lets call it 27 months, at ~30 days a month this works out to something like 16 reboots a day! I’m pretty confident this value isn’t correct, although it does make one think.

When you consider the amount of times I’ve accidentally left it bootlooping overnight, hacked together some automated kernel bisect, or usually just made a dumb mistake somewhere requiring I reboot to re-flash, it doesn’t seem that unrealistic 😅.

When working on postmarketOS, there is another optimisation I’ve made which leads to a pretty big speedup in my workflow: flashing from userspace.

Over a year ago now, I implemented support for boot-deploy in postmarketOS, it was the final missing step to have kernel upgrades actually work on Android devices. Before then, when upgrading your kernel it would run through the exact same steps that pmbootstrap does; building an Android boot.img from the kernel, dtb and ramdisk for your device (as referenced in /etc/deviceinfo), but it would never get flashed! Instead requiring users do it manually.

Fixing this by having the hook also flash the new boot image lead to a new workflow for kernel development and is what I use to this day.

envkernel native

This is something I’ve wanted to talk about for a while, as I think this workflow is a huge productivity boost over the traditional reboot-flash-reboot methodology, please bare with me :)

pmbootstrap - the postmarketOS build tool - comes with this super nifty utility called envkernel.sh, it’s a script which you source to build kernels with zero setup required on your host machine. It works by using pmbootstrap to spin up a chroot with the correct dependencies to build a kernel for whatever architecture your currently configured device is. Then it overrides make with an alias to call make inside the chroot. The make alias does the following (slightly trimmed for brevity):

pmbootstrap -q chroot --user -- ARCH=arm64 \
CROSS_COMPILE=/usr/bin/aarch64-alpine-linux-musl- \
make -C /mnt/linux O=/mnt/linux/.output \
CC=aarch64-alpine-linux-musl-gcc HOSTCC=gcc

The magic that makes this happen is that envkernel.sh creates a bind-mount of your kernel source dir into the chroot!

As a method lowering the barrier of entry for new hackers to get kernels building and get their phones booting, this is a pretty wonderful and hugely underrated tool - I’m yet to see it mentioned outside of postmarketOS, or to see anything which offers anywhere near the same ease-of-use for newbies.

It gets better

With your kernel built, now you need to deploy it, all the necessary tooling to make this happen lives in postmarketOS, more specifically on the device rootfs. It can be run either on the host in a chroot (the usual method when building an image) or on the device (for end-users to receive updates). postmarketOS doesn’t deploy boot images directly for quite a few reasons, but that’s out of scope here.

So, we know we can easily build a boot image just by installing a new kernel package, we also know we can easily build a kernel… All that’s left is turning that kernel into a kernel package, and thanks to envkernel it’s as easy as this one-liner:

pmbootstrap build --envkernel linux-postmarketos-qcom-sdm845

Of course, replacing the package name with whatever your kernel package is. This does a bunch of unspeakable horrors with APKBUILDs to use the artifacts from your local kernel build and run the packaging steps for your kernel package.

For folks (like me) who don’t like dealing with some of the quirks envkernel brings with it, this MR I really REALLY need to finish goes one step further and lets you package kernels you compiled with regular make on your host

Bringing it all together

The vast majority of the time when I’m doing kernel development I’m not breaking the boot process entirely, usually I still get to a rootfs, allowing me to use this nifty one-liner to build and deploy a whole kernel and modules:

mm && \
pmbootstrap build --envkernel linux-postmarketos-qcom-sdm845 && \
pmbootstrap sideload linux-postmarketos-qcom-sdm845

The final step, pmbootstrap sideload, is a wrapper around ssh which pushes the new package and then installs it on-device with apk, the post-install hooks installed by the postmarketOS mkinitfs and boot-deploy packages build a new boot image and automatically flash it.

The end-result is that I save a whole extra reboot by never having to enter fastboot to flash unless I actually break the kernel.

If I do break it, I just run the exact same process that I would on-device on my PC instead with:

pmbootstrap chroot -r apk upgrade -a
pmbootstrap flasher flash_kernel && fastboot reboot

AOSP and VSCode

Now, onto the original inspiration for this article… As you can tell I got a liiitle sidetracked 😅.

Despite being an avid VSCode user, I generally don’t make a lot of changes to the stock configuration, a few modified keybinds and ~50 extensions for various languages, themes, features (I need to go through them and probably delete a lot of them). I’m not one for reading docs, usually I’ll learn just enough to be dangerous, and this is especially true with intellisense.

The comfortable default behaviour, super easy .clang-format integration, and compdb support make for a pretty enjoyable time, or so I thought until I started working on AOSP.

After almost a year, I’m now fairly comfortable with AOSP, I keep the whole thing open in a single vscode window and with 64GB of RAM it handles just fine. But this didn’t use to be the case…

The C/C++ extention for vscode includes some great intellisense, if your project can provide a compdb compile_commands.json file then you’re off to the races.

For the last 10 months I didn’t know AOSP could do this, I dealt with the intellisense not respecting it’s memory limit, using up 50GB+ of RAM and making me much more familiar with the OOM killer than I ever want to be.

Much to my surprise (and dismay) AOSP can actually do this \o/, it’s just a shame I found out too little too late as this literally rocked my world.

It’s seriously easy too… Here are the docs. With your environment set up, just run the following:

SOONG_GEN_COMPDB=1 m nothing

Then open the C/C++ Configuration (ui), go to advanced and set the compile_commands.json path to

out/soong/development/ide/compdb/compile_commands.json

Reset the intellisense database, restart VSCode, and enjoy near-instant intellisense for any C/C++ source file in all of AOSP.