Kevin's musings

In october, I went on 2 trips and after that several things became very clear: First, covid is still out there and you can still get it (as I did) and that my current trusty phone that I had been using for the last 7 years finally needed to be replaced.

The old phone is a one plus 3t and it’s been a great phone. I currently have been running /e/ on it (with no google) and it’s been fine. Unfortunately, the years have now taken their toll on the battery. When traveling I had to put it on super battery save just to have any battery left after a day away from a charger. On super battery save it’s slow slow slow, bordering on unusable. The sad thing is, if I could replace the battery I could probibly be fine with this phone for more years. It’s true it doesn’t take super great pictures, only has 64GB space, and has a bunch of scratches now. But alas, the battery is non replaceable, so I decided I needed to do something.

Of course the first thing I looked at was just using one of the 2 pinephones or 1 pinephone pro I already have. Sadly, they just aren’t good enough for a daily driver for me. They are slow, the battery life is also bad (on the pro at least), and… the biggest problem: The camera is just not good at all. I end up taking a lot of pictures and I really need them to be viewable. Finally, despite lots of work, there’s still a bunch of things non upstreamed, so I would have to run a custom kernel and a bunch of other non upstreamed parts.

Next, there are now some phones you can buy with /e/ pre-installed. There’s the Murena one and Teracube 2e. The one has more storage space, but otherwise they have less ram than my oneplus 3t. 🙂 This was a tempting option, but /e/ hadn’t been impressing me of late. Updates were few and far between, I had to do a bunch of tinkering to get on the latest stream (based on android 11).

The fairphone3/4 are interesting, but don’t seem to be available in the US at all. Of course I am sure I could get one, but likely it wouldn’t work with US telco’s.

I wanted to get something that would have a nice long life of updates also.

Finally I ended up deciding on just getting a unlocked pixel 7 and going with grapheneos on it. I deliberately choose the 7 over the 7 pro because it’s slightly smaller (but still big) and from all reviews I read had a better battery life. The pixel 7 gets 5 years of updates, which is about as good as you can do these days.

Why grapheneos? Well, I did not want to go back to being tied to google if I could help it and /e/ doesn’t really support any very modern phones. The free phone os’es (postmarketos, mobian, etc) also don’t support pretty new hardware either. However, I figure after a few years there’s a good chance something like a pixel will be supported by more of those and I can choose to jump to one of them if I want. grapheneos is basically ASOP (the “upstream” android) with a bunch of security enhancements added to it. The install process was pretty painless, but I did hit one problem where I tried to install with the web installer using firefox, then switched to chrome and got an error. I finally figured out that I forgot to close the firefox tab out and it was keeping the webusb locked so the other browser couldn’t install. 😉

Install went fine after that and I installed f-droid and got my applications and data moved over to the new phone. The biggest headache was of course signal. I had been using it for sms/mms after my last /e/ re-install, but they are dropping support now, so I had to export my sms’s back out of it to get them moved over. The export function doesn’t tell you that it doesn’t handle duplicates and you should wipe your sms db first, so I ended up with 2x my sms messages. Finally got that transfered over and signal deleted. signal could have been a great app, but they seem determined to made decisions that will drive them into irrelevance now. It’s sad.

Anyhow, I hope the pixel 7 will last me a few more years until I can get a modern phone and put Fedora on it. 🙂

Astute observers will have noticed recently that Fedora rawhide composes stopped after 2022-04-13 and didn’t resume until 2022-04-19. A particularly odd bug was to blame. This is the story of that bug and my investigations of it.

A bit of background first on how nightly images are made currently. There’s a cron job that calls a script (called nightly.sh) in the pungi-fedora pagure.io git repo. It does a number of small associated things (like sending message bus messages of status, sending report emails, copying results around, etc) but the big thing it does is to call pungi. pungi in turn looks at it’s config (in that same pungi-fedora repo) and does all the heavy lifting of the compose, calling mostly out to koji to do things, but also doing some things locally. For images, pungi calls koji with some parameters (use this repo and name for kickstart file, use these compose repos for packages, etc). koji then calls different tools depending on what the image is. For livemedia the tool is livemedia-creator (in the lorax package), for qcow2/raw images it calls ImageFactory/oz to do the actual image build on a builder.

The compose on the 14th failed making a aarch64 Cloud-Base image. https://koji.fedoraproject.org/koji/taskinfo?taskID=85654099 with an odd traceback at the very very end of the build:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 17480: invalid start byte

So, my first go-to on these sorts of things is to look at what packages changed between the successfull compose on the 13th and the failed one on the 14th. That day of changes was pretty small, being a thursday in a week when everyone was focused on Fedora 36 final testing. The only thing that seemed like it could affect booting at all was a grub2 update. That update only added a ‘read’ module though, which normally I wouldn’t think could cause any issue, but perhaps it was a odd aarch64 toolchain issue? So, I tried some composes with grub2 untagged. No luck, still failed the same way.

Next I tried to get some more information about exactly what was not utf-8 that it was complaining about. I did this by doing some koji scratch image builds. But of course my scratch builds never seemed to fail, they all worked fine.

Of course then it was easter weekend, I got busy around the house and didn’t really dig back in to it until the following monday. This time I did some more scratch builds and finally managed to get it to fail correctly and with some added debug in oz on the builder to print out the data it was trying to convert to utf-8. At the very end of making the image (while it’s shutting down), it was getting a bunch of:

cpio: Malformed number <bunch of junk encoded weird stuff>

This of course was not something python wanted to convert to utf-8. It looked like a totally different encoding in there, so oz marked it a failure.

A sidebar here on how oz works. oz takes its inputs and tells libvirt to fire off a vm for it, running the installer with the kickstart provided, and serial console pointed to a socket. It then starts polling (by default every 10 seconds). If there’s disk or network activity on the vm, it gets data from the console and waits. If it hits the total timeout it kills the vm and returns a failure. If it doesnt see any disk or network activity for (default 300 seconds) it assumes the install has stalled/failed, and kills the vm and returns a failure. If the vm has shutdown, it assumes it’s successfull and returns. The output of the console is returned back to koji so you can see the log file in the koji task. However, as Adam Williamson pointed out: That last 10 second window never actually gets collected by oz for converting/adding to logs. If the install shuts down right after oz polls, the next time it will poll the vm is shutdown and it just returns.

So, what is this cpio output? Well, when you shut down a Fedora system, dracut calls “/usr/lib/dracut/dracut-initramfs-restore”. This script runs at the very end of the shutdown cycle and is supposed to copy out the initramfs to memory and then pivot your install to using that as / so it can unmount your real root directory cleanly. The script uses a wrapper called ‘skipcpio’ thats shipped with dracut. This wrapper skips past the first part of the initramfs, as this is just a small archive with the microcode to load that first thing when you boot. The next archive is the actual initramfs that does the heavy lifting. The script has no idea how you have compressed your initramfs (if you have), so it just blindly tries: not compressed? gzipped? bzipped? xz? lz4? lzop? zstd? The first try (not compressed) is the thing that results in the error spew. cpio bravely tries to unpack the zstd compressed initramfs and… has problems.

So, the flow is:

• nightly.sh calls pungi
• pungi calls koji to make Cloud-Base image
• Oz calls libvirt to do install, starts polling.
• Install boots, installs and finishes.
• On shutdown dracut-initramfs-restore runs and because it tries uncompressed first, spews cpio errors
• Oz polls again after the errors start, but before the vm is completely off.
• Oz tries to convert the log to utf-8 and blows up.

Upstream dracut as it turns out just merged a PR to change the order it tries these things in (moving zstd to the first one) because it resulted in several seconds slower shutdowns. So, I just added that to the rawhide dracut package and everything started composing again.

So, what caused this to start happening now? Well, queue back to oz not getting the last bit of console output. My theory is that this has been happening for a long time, it’s just since it was at the very end of the output, it always got dropped on the floor and oz never saw it. For some reason now we are shutting down faster, or slower and it’s getting caught in that last poll that does have data.

Always fun digging into these sorts of things. 🙂

Some of these may apply to all laptops and some may be frame.work specific, but I thought I would throw them out there to help folks out.

• Remove any of your modules that aren’t usb-c. At least the microsd and HDMI modules pull a small amount of power and prevent the laptop from going into the lowest power saving levels. Without doing this, I never get pc10 (the lowest power saving mode). framework is working on fixing this in a upcoming bios update.
• Turn off the camera and microphone with the switches at the top of the display. Again, these draw a small amount of power and keep things from going to lowest power settings.
• dynamic power profiles powersave mode is much better in my testing than tlp or tuned for saving power. It’s also easy to manage with workstation/gnome desktop. Others can use the powerprofilesctl command line tool.
• powertop –auto-tune (or simply ‘systemctl enable powertop’ and reboot) seems to work ok on the framework laptops, with one exception: If you disable bluetooth and then try and re-enable it, it will find no controller. You will need to do a cold boot (completely power off and back on) or ‘rmmod btusb; modprobe btusb enable_autosuspend=n’ to get it back.
• ‘echo powersupersave > /sys/module/pcie_aspm/parameters/policy’ provides some power savings and to me at least doesn’t seem to cause much in the way of slowdowns.
• Make sure you aren’t hitting this bug in firefox ( https://bugzilla.redhat.com/show_bug.cgi?id=1936901 ) which basically causes it to start speech-dispatcher and then never ever let it go away. This was causing about 3-5% cpu use here, but it also causes fans to start up and no power savings to happen.
• If you don’t mind using rpmfusion-nonfree items, installing intel-media-driver from there helps video playback nicely (depending on the video).
• bluetooth and wifi both use a fair bit of power, if you don’t need them, do disable them for more savings.

Hope that helps someone out there get more battery life… 🙂