All Control

I’ve recently been working on putting together a CI system for postmarketOS that will allow us to do proper automated integration testing. That is to say - when someone opens a merge request that modifies our initramfs (for example), we should be able to click a button and some minutes later know that this change doesn’t break any of our important usecases.

QEMU absolutely can (and will) get us most of the way there, but at some point we need to just run the same software that we’re running on end user devices. Furthermore, QEMU can’t tell us anything about changes in the kernel that might affect our devices, and manually testing during kernel upgrades, frankly, sucks.

Enter: the board farm

So we need a fancy board farm, this is one of those things where folks with the right technical background could build something over the course of a week. But for someone like me it’s full of trial and error and hidden complexity… It’s easy enough to do this with one device - just hack something together, but to be successul we need something reliable and adaptable, that we can adjust to fit our needs in the future, and the wide range of devices we support.

Unfortunately, that isn’t quite what I’m able to present in this blog post - it’s coming though. Instead what I do have is somewhere in the middle, let’s run through it and I’ll show you how to build your own, and how you can offer it up to be used by postmarketOS CI going forward.

I should also mention, the whole point in this setup is to be as cheap and easy as possible, fully distributed, and shareable. I want this to become a community resource that can benefit any and all Linux mobile projects, initially I hope to allow Mobian to use it, and branch out from there as we handle scaling up the infrastructure.

Enough rambling

I think that what I have currently will work for most phones from before ~2019, newer ones have crazy battery tech with batteries in series and might need more complex power circuitry. What you’re seeing here is the following:

I’m ashamed to admit how long it’s taken to come up with even this prototype, but hey it works, and pretty damn well too :D

In essence, the pico is running this great firmware which was written my Martijn Braam, he has also been working on building some job scheduling software which I’ll discuss in a bit.

The firmware exposes two serial ports to the controller, once is passed through from the phone, the other is used to interface with the pico to turn the power on and off as well as press buttons on the phone.

Most Qualcomm phones use 1.8v UART for the serial port, this isn’t a problem for the pico to receive reliably, but it isn’t so good for transmit, so to handle this I use a simple voltage divider of two 10k resistors from the pico Tx pin to ground.

The OnePlus 6 battery has a chip in it (specifically the BQ27541) which provides “fuel gauging”, as the built in capabilities of the PMIC can’t handle the absolutely cursed mechanism OnePlus use for their dash charging (hooking VBAT directly up to the USB port VBUS lol). So we probably shouldn’t remove it. Instead the trick is to take the kaptan tape off the top of the battery and cut off the actual lithium pack, leaving just the circuit board. Then the output of your voltage regulator board can be hooked up in place (make sure you check the polarity before cutting off the battery!).

Phew, ok so with that done, now onto the buttons and UART. The buttons probably aren’t necessary, but they might be useful for future testing! Most devices will always boot into fastboot when you erase the boot partition which keeps things wayyy simpler. If you’re lucky, your device has already been documented on the postmarketOS wiki either [here] or here, otherwise you’ll have to either track down schematics or use a multimeter and trial and error to find UART.

The software

For this initial setup, I’m using this tool currently maintained by some colleagues of mine at Linaro called CDBA. We use it internally for a lot of our testing and development, where getting the right hardware in the hands of the right people isn’t always possible.

CDBA has a long history that I won’t get into, it’s only recently started being given the love and care it deserves, so there’s still quite a lot of rough edges and lacking documentation. Nonetheless, it’s built from the ground up for this usecase and has a lot to offer. It currently only supports fastboot, sorry Pinephone users but until someone enables fastboot in U-Boot you guys are gonna have a tough time here.

CDBA works over SSH, which for now poses some issues with port forwarding, however we have some ideas to deal with this thanks to the scheduling software that Martijn has been working on. I’m glossing over the cdba setup here, it’s a little finnicky but check out the shell directory for some rough instructions. It works using some git magic and expects you to maintain a git repo with the CDBA config and the SSH public keys that should have access. It works using a new user account on your control PC for security.

devices:
  - board: sdm845-enchilada-1
    qcomlt_debug_board: /dev/serial/by-id/usb-postmarketOS_Test_Device_E6604430430-if00
    console: /dev/serial/by-id/usb-postmarketOS_Test_Device_E6604430430-if02
    name: "OnePlus 6"
    fastboot: 5725c671
    ppps_path: "3-3:1.0/3-3-port2"
    fastboot_set_active: a

This tells cdba to use the qcomlt_debug_board backend for board control, that opens the serial port specified and sends commands to it, the pico firmware implements this protocol. The console itself is the other serial port exposed by the pico. And fastboot_set_active: a means that before booting we should reset the boot retry counter to avoid the bootloader marking the current slot as unbootable.

The ppps_path options denotes the path to the USB hub, you need to have the uhubctl tool installed for this. You can determine the correct path by running uhubctl with no arguments and then looking at /sys/bus/usb/..., you’ll always need to append the :1.0.

GitLab CI

The bane of my existence… I’ll just summarise here, you need to generate a new SSH key pair and configure CDBA to allow the public key, put the private key in a protected CI variable.

In your CI, have a job to generate a boot.img for your device that runs some tests in the initramfs, and then run a script like this one which should set up the SSH key and then invoke CDBA (don’t forget to specify the timeouts!).

Congrats, you should did your first fully automated integration test on a real device!

We hope to have something like this available in postmarketOS in the not too distant future. Initially it will likely call CDBA directly, eventually we’ll need to scale up and manage more projects using the infra, as well as more devices and farms being added. To handle that we’ll switch over the Martijn’s awesome PTS system which can take over the complex parts and we’ll just submit jobs to it. I hope to be able to have it integrated locally with CDBA so that the switch-over can be extremely straightforward as it will just continue running CDBA internally. By using a central control it will also alleviate the need for portforwarding!

fin

Well there we have it, this is a very rough and ready infodump while I take a train ride. There has been so much going on behind the scenes here and I’m really excited for things to start becoming more visible over the coming months.

At the very least, I have most of the info out now. I want to duplicate everything here into the postmarketOS wiki as some of the pain points get resolved.