Archive for fedora

410 results.

Ansible and Fedora/EPEL packaging status

by on 2021/04/12 at 4:35 pm
Posted In: ansible, fedora, linux

Just thought I would post a current status on ansible packaging in Fedora/EPEL

ansible 2.9.x (aka, “ansible classic”) continues to be available in EPEL7/EPEL8 and all supported Fedora releases. Odds are most people are just still using this. It does still get security and some small bugfixes, but no big changes or fixes.

ansible 3.x (aka, “ansible-base 2.10.x + community collections”). I had packaged ansible-base in rawhide/f34, but due to the naming changing and lack of time, I have dropped it. ansible-base is retired now in Fedora and likely never will land there.

ansible 4.x (aka, “ansible-core 2.11 + community collections”). I have renamed ansible-base to ansible-core in rawhide. Unfortunately, a dep was added on python-packaging, so there’s 6 or so packages to finish packaging up and getting reviewed. The collections are a bit all over the place as people have been submitting them and getting them in. You can find the ansible collections via ‘dnf list ansible-collection\*’. After I get ansible-core in shape, I am going to look at packaging up at least the rest of the collections for 4.x. At that point we could look at dropping ansible-classic (or moving it to ‘ansible-classic’ and shipping ansible-core + community collections as ‘ansible’ 4.x. Note that collections work with both ansible-classic and ansible 4.x.

ansible-core 2.12 (I don’t know if this will be in ansible 5.x, but I guess so?) will REQUIRE python-3.8 or larger. So, EPEL7 will probibly never move to this. EPEL8 may, but it might be tricky to use the python3.8 module for this.

So, progress is being made, all be it slowly. 🙂 I’ll post again soon hopefully to announce that ansible-core is usable/testable in rawhide.

Comments Off on Ansible and Fedora/EPEL packaging status

A critique of google chat

by on 2021/04/14 at 11:58 am
Posted In: fedora, linux

I’m not usually one to say bad things about… well, anything, but more and more people I interact with lately have been using google chat at their preferred communication medium and I have been asked a few times why I dislike it, so I thought I would do a blog post on it and can then just point people here.

First a few pros before any cons:

  • It works fine in firefox as far as I have seen
  • It’s included in “google workspace’, so if you have that you have google chat too.
  • It looks ok. Links, emojis, reactions, links, videos all look/work fine.
  • It’s easy to start a chat with multiple people or groups
  • All your chat history is there on restart (but see below)
  • It’s encrypted (but no idea if google can read all the history or not)

And now all the actual current cons:

  • It’s a browser only application. If I don’t remember to reload that tab after rebooting, it’s just not there and I get no notices. Needless to say: no 3rd party applications have a chance to do better than the web interface.
  • It’s thread model is utter junk. There are ‘threads’ of conversation, but they are all in the same linear timeline, so its really really hard to tell or notice if someone added something to an old thread without knowing you need to scroll back up to it.
  • It’s horribly “unreliable”. By this I mean I could have the app loaded in my browser and _still_ not see notifications because: google is randomly asking me to login again, google chat decides to put <— and 100 more lines here –> collapsed section, google chat randomly decides to put “and 10 more messages” button at the bottom, Someone could have started a new thread and the one you care about has any of the above happen to it and you never know.
  • Someone you are messaging could just not use google chat or be busy or away. Google chat will helpfully mail them about your message… but 12 hours later. You can also say you are ‘busy’ but only for 8 hours at a time. If you are gone for a week, too bad.
  • Surprisingly search is kind of bad. It finds things you search for, but there’s no way to go to the ‘next’ match or see how many or navigate at all in it.
  • Related to search, there’s no way to have logs locally to use unix tools on like grep, etc.
  • There’s a indicator of how many messages are unread in a room/chat, but no indications if thats just general or if someone mentioned your name.
  • No good keyboard navigation, you have to click around to change channels, threads, conversations, etc.
  • Unless you have been invited, it’s virtual impossible to find rooms. There’s a ‘browse rooms’ dialog, but it only lets you search in the room name, not description or who is in what room or anything.
  • Depending on how things are configured, you may not be able to chat with everyone, only people in your company or workspace.
  • You can adjust notifications per room somewhat, but it’s not as flexible as any IRC client. You can notify on everything, or on just mentions of your name, or mentions of your name and every new thread or nothing. No way to look for keywords or anything.

And of course there’s potential future cons:

  • google changes chat products more than a bakery changes it’s day old bread. Is this iteration of google chat going to stick around? Who knows? Could be a completely rebranded and different one next week. ;(

So, all in all, I will use google chat if I have to, but it’s going to be my least favorite chat method and I wish everyone would switch to something else. Matrix perhaps?

Comments Off on A critique of google chat

Nest with Fedora: 2021

by on 2021/08/11 at 4:43 pm
Posted In: fedora, flock

Last week was the 2021 edition of Nest with Fedora. Nest is the all virtual version of Flock, the annual Fedora community conference. While sad we couldn’t all get together in person again this year, the virtual version was fine. The platform again used was ‘hopin’ and it did a reasonable job overall. I did have some problems sharing my presentation for my talk, but otherwise I think everything went smoothly. A few of the talks I went to and some thoughts on them:

  • State of Fedora keynote: Nothing earth shattering here, but good that things all seem to be going up and to the right. 🙂
  • Fedora Council Town Hall: Some good questions, next time it might be good to go round robin as I think there were a few council members that didn’t talk and some that talked a lot.
  • Meet CPE: Good overview of how we do things and whats going on from my teammates.
  • 6 tips working with communties: I think these were all things I knew, but I think it was a good talk for those that don’t know or haven’t thought about how open source communities work.
  • My Pinephone talk: I had technical problems to start with, but after that it went smoothly enough. I was worried that I would have too much time to fill, but I just had enough with some questions. (More on this talk below)
  • Exploring our bugs: Lots of interesting charts and graphs. I am still not 100% sure what they might mean. I wish Fedora had a data analysts SIG or something that tried to ask questions and answer them with data. You could spend a LOT of time in this. 🙂
  • CentOS and Alma: Nice talk about how things were going in both projects and how they collaborate.
  • fbrnch a year on: I knew about fbrnch, but I haven’t used it. This talk made me put playing with it on my list for sure. Lots of things it can do for you/simplify workflows.
  • But it’s all ‘upstream’, why doesn’t it work on fedora? great talk from Peter about different levels of upstream and how you need to integrate things in fedora, not just assume they will all work together when updated.
  • Hassle-free Throw-away VM’s with Testcloud: I missed the start of this talk, so I really want to watch the recording later and get the basic info. I was a bit lost lacking that info. 🙂
  • Fedora Trivia Pub Quiz: Fun as always. I never win.
  • Fedora Project Leaders of Legend and Lore: This was _AWESOME_ to see (almost all) the ex-fpl’s in one screen and answering questions. I had to leave about 1/2 way through this talk, so I am looking forward to the recording to watch the rest of the fun.
  • Giving and Receiving Feedback: This was a great talk, I hadn’t really though about feedback from all the angles, and changing the dynamics so you are on the same side trying to reach a goal is brilliant. I hope to use this and to see if we can make it pervasive in out communities. Also ended up reading thought a bunch of related blog posts and twitter threads… great stuff!
  • Building Initrd Images from RPMs: A plan to make initramfs from rpms instead of using dracut. I’m ok with this plan.
  • Making Your Life Easier with the Packager Dashboard: The packager dashboard is great, I hope this gets more folks using it. I have been using it myself for a while and it’s really quite nice for package maintainance.
  • Cross Collaboration Panel: This was interesting, but might have been more interesting from a ‘each person tell your collaboration story’ instead of just trying to answer questions from the audience. Still, cross distro collaboration is a great thing and we should all strive to do it when we can.
  • Closing Remarks: Most signups and attendees ever. It seems clear even if we do in person next year, we should either _also_ do a virtual conference or some kind of hybrid. It’s just too handy to not have to travel. 🙂

My talk on the pinephone seemed to go well, but it’s hard to tell. 🙂 I tried to put forth the idea that because smart phones are slowing down (people want longer support, don’t replace their phones every year anymore, no new superfeatures, % sales lower year over year) this is a great time to move a open source community into the phone space. It’s very similar to the pc market in a lot of ways. I went over the pinephone hardware (will never be super fast, but at ~150$ it’s reachable to hack on from everyone) and hopefully someday may be a daily driver for some. We really need upstreaming of some connectivity so we can start making fedora images: wifi/bt/modem/usb-c, etc. Hopefully that will happen soon. Userspace is coming along fast with Phosh/plasma/other desktops and applications.

Overall a nice time, many thanks to all the organizers again this year!

Comments Off on Nest with Fedora: 2021

Frame.work laptop: The hyperdetailed Fedora review

by on 2021/08/29 at 1:39 pm
Posted In: fedora, linux

Frame.work is a new company that appeared eariler this year. They announced that they were going to make a laptop focusing on repair-ability, with replaceable interface modules. I didn’t really need another laptop, but I was very interested in the idea and wanted to support them, so I preordered one as soon as preorders started up. That laptop arrived here a few weeks ago, and I used it as my main laptop last few weeks.

Framework offered several already assembled models, as well as a “DYI” edition where you could get the base laptop and as many or as few add on parts as you liked. I of course went for the DIY edition.

The packaging was the typical these days recycled cardboard and paper, but all very nicely packed and professional looking. In the box was the body of the laptop itself (the screen/keyboard/motherboard) and then small packages with the parts: memory, wireless card, nvme storage stick, and modules. Framework wanted to ship the DYI edition completely disassembled so you had to do a lot more assembly, but it turns out there’s a bunch of silly rules about laptops vs laptop parts and if they shipped the unassembed version they would run afoul of customs and shipping and taxes rules. There’s no instructions or manual in the box aside from a small “user manual” that just describes the features of the latop (where the switches and ports are) and has a bunch of regulatory information. It also comes with a torx tool/screwdriver/spludger, which is a nice touch.

After unpacking and looking at everything I poked around and found https://guides.frame.work/Guide/Framework+Laptop+DIY+Edition+Quick+Start+Guide/57 that describes how to assemble things. The hardest part was getting the wireless card antenna connected and running the right way, everything else was a complete dream.The captive screws for the cover was a super nice touch. I managed to get everything installed and assembled in about 10min (and that was with some kittens wandering around distracting me 🙂

Some general pictures and observations on hardware:

  • In general as many have noted the laptop reminds heavily of the older macbook pro 13″ models. Might be the aluminum case, or the keyboard layout or just everything. Thats not a bad thing, just an observation. The laptop seems solid and well constructed. Some reviews said the hinge was somewhat flimsy, but the one I have doesn’t show that at all. It seems quite solid and workable.
  • frame.work laptop
  • The power/fingerprint reader button in the upper right has a light/glow around it when the laptop is on. Currently there isn’t any way to adjust this, but apparently its hooked into the EC so it might be possible to get firmware or linux able to adjust/disable it. It doesn’t bother me, but some folks don’t like it in dark rooms.
  • You can charge from any of the expansion card ports if you have a usb-c card in there. On both the right and left side in between the expansion card bays is a charging light. So, if you plug power into a bay on the right, the light on the right will light up to indicate charging.
  • The expansion cards are just usb-c to whatever. You could just connect directly to the usb-c connector in the bay if you wanted to for some reason. The usb-c expansion cards are just basically passthroughs. frame.work has released 3d models for the cards and is even funding people’s production of card ideas. See https://github.com/FrameworkComputer/ExpansionCards for all the details. Someone has even already come out with a ‘storage’ card (bascially lets you store things like microsd cards or 3 m&m’s or something). See https://community.frame.work/t/the-snack-drawer-has-arrived/5757 framework is working on a ethernet card.
  • There is a headphone jack… for people who like that sort of thing. I doubt I will ever use it.
  • You can tilt the screen all way back, 180degrees (or a bit more). I don’t often need that, but the dell xps 13 I have only goes about 120degrees and it’s sometimes anoying.
  • The keyboard backlight has 4 settings, which you can step through with function space. low, med, high and off.
  • The hinge is kind of square and has a gap behind it when opening. Until I got used to it I found it to sometimes pinch my legs when opening it on my lap. This seemed to only happen the first few times I opened it so not really a big deal.
  • The camera and microphone have disable switches (showing red when disabled).
  • There’s a light sensor next to the microphone on/off switch.
  • The firmware is a “InsydeH2O” (which I think is a 3rd party that ibm also uses on some of their laptops). Framework has expressed interest in getting coreboot working. There’s a complete firmware guide at https://community.frame.work/t/bios-guide/4178 They also are going to release the EC part of their firmware
  • Framework has a ‘testing’ account on fwupd and plans to release firmware updates there.
  • You can boot from: nvme, usb stick, microsd in the microsd expansion, and any of the storage expansions.
  • There is a small button between the expansion card bays on both sides on the bottom of the laptop. That button is a release then you can extract a expansion card. I’ve found this to be somewhat difficult, which is ok as I wouldn’t want cards sliding too easily out, but it could be a tad easier to extract them.
  • The screen is a ‘2k’ screen. 3:2 aspect with 2256×1504 pixels. It’s a nice bright screen, but I do miss the 4k screen on the dell xps 13 I was using (more on that below)

I choose the middle of the 3 cpu options available. There was a i5, a i7 midrange and a i7 top end. The top end i7 was quite a bit more, so the middle option seemed good to me. For storage I went with a midrange option too, a 512GB WD nvme. Finally for memory, I saw that the laptop supported 64GB, and since you can never have too much memory thats what I went with (2 32GB dimms).

There’s lots of things framework plans to do moving forward, but I am sure all of those depend on how well the laptop sells. They did sell out of their first batch thought, so thats a good sign. A few things that they have in the works:

  • As mentioned above, they are working on a gigabit ethernet expansion card. I will definitely get one of those.
  • Also related to expansion cards, they are planning a ‘marketplace’ for people to produce and sell whatever crazy cards they make. Also in this marketplace will be replacement parts for the laptop and things like different colored display bezels, different keyboards (they are planning a transparent one ,which could be fun).
  • They have gotten tons of requests for AMD cpus or arm cpus or something else cpus. So, hopefully they will be able to ship new replacement motherboards someday with these. I would jump on a aarch64 mainboard in a second. 🙂
  • They have also gotten requests for different display options. I might be interested in a 4k display.
  • Finally there are folks wanting a 15″ form factor. I have no idea why anyone would want that, but I suppose they might release one.

Moving on now to software. The DIY edition doesn’t come with any OS, you can choose or use your own.This is of course exactly what I wanted. A quick download and boot on the Fedora Rawhide workstation image on usb stick and a install and Fedora was up and running nicely. Since I used rawhide I avoided two small gotchas for stable release users: You need a newer kernel to support wireless/bluetooth (use a Fedora respin) and a newer fprint package to get the fingerprint reader working.

Some software impressions:

  • Aside the wireless needing newer kernel and fingerprint reader needing newer fprintd, Fedora workstation just works on the laptop.
  • The screen is nice and bright. I do wish there was a 4k option, but that of course would affect performance and battery life (see below on that)
  • Everything is really fast. The Xe graphics + fast cpu + enough memory to cache everything you do makes this laptop a joy to use.
  • Switching between this laptop and my dell xps 13 9300 make me realize how little state I actually store locally. Newsflash/rss feeds are all in my tinytinyrss server, hexchat messages are all in znc on my main server, mail is synced from main server to both laptops with mbsync, and as long as a sync happens when I switch everything is on the other laptop, element, discord all store state elsewhere. Firefox does have local state, but I can share links easily so that was not a big problem. Git repos? I don’t tend to have things I work on for a long time in git repos, so as long as I finish and push a pr or commit I can just pull on the other laptop and be in sync.
  • Fingerprint reader works great (apply updates if you are on f34). I’m not sure I will want to keep using it, especially when traveling, but it’s kinda nice here at home.
  • kvm / virt works fine. Installing a Fedora 35 vm took just a few minutes and had no problems.
  • The default suspend level is s2idle, which uses quite a lot of power while sleeping. You can switch to ‘deep’ sleep mode by doing a quick grubby call to update kernel command arguments and rebooting:
grubby --update-kernel=ALL --args="mem_sleep_default=deep"

Now, lets talk about battery life. There’s been some reports that battery life is really poor, and I can understand why people think so. In the early days of laptops, the laptop consumed about the same amount of power no matter what it was doing. This made is easy to show how much battery life you had left and estimates were pretty accurate. Fast forward to todays modern laptops and… it’s utterly different. You have a ton of adjustments you can make: The brightness of the screen can reduce power use, what you are currently doing on the laptop matters much more, and the cpu has a million things it can do to save power or not. Sadly, it’s not even as clear as that however. Consider you are on your laptop on battery power. You need to finish a spreadsheet. So, you want the cpu to run as slowly as possible, but… when you go to recalculate a bunch of things, the cpu going slow and taking 5minutes to do something it could do in 15seconds on higher power use actually net drops you power. It would be much better to be able to burst up and do something quickly and use that power for 15seconds than it would be to not and use 5minutes of lower power. You also have to take into account fans and heat and GPU and… it’s complicated. So, all this complexity and knobs you can adjust result in seeing accurate estimates of how much battery life you have left, but only if you are doing _exactly_ what you are doing right now. There’s a ton of linux tools related to power management now: thermald, powertop, tlp, tuned, and the newest entry: power-profiles-daemon

On the framework laptop on battery, I can see anything from 2hours (in google meet video call with bluetooth headset on, camera on, performance setting) to 14hours (low backlight, camera off, openvpn off, keyboard backlight lowest, powertop auto-tuned, powersave dynamic profile). Powertop is quite useful for tracking down things that are using power. Using it:

  • The usb camera draws a ton of power, even when not in use (this might be because it’s not auto suspending correctly?) But you can fortunately just use the switch next to it to disable it when you want to save some power.
  • Folks on the framework community site are seeing a similar thing with the HDMI expansion card.
  • I was also seeing similar with the microsd card expansion card.
  • bluetooth also draws some even if you aren’t connected to any bluetooth devices (sub 1w)
  • power profiles are by far the best method to get more battery life. Moving from performance mode to power-save adds about 3 hours to the estimates.
  • If you do ‘powertop –auto-tune’, beware that btusb doesn’t handle auto suspend right and if you reboot it will load, but not work at all. You then need to rmmod btusb and ‘modprobe btusb enable_autosuspend=n’ to get it working again. It won’t work even after reboots until you reload it without autosuspend. That one confused me for quite a while.
  • For some reason my work vpn (openvpn) uses a fair bit of power. Wireguard which I use for home seems much more energy efficent here.

In the end battery life is going to vary depending on what you want to do and how long you want it to take. I was planning on seeing how well I could do on battery during a ‘normal’ day, but I didn’t get to it. I suspect many people never run their laptops on battery (or just when moving from place to place) or only use battery much when traveling (remember when we used to do that?). I admit to being kind of in that boat myself… the only times I am on battery is when I go outside on the deck or am needing to do something in the car while someone else drives. The battery life on my xps 13 9300 is much worse than the frame.work laptop. It makes some sense however, as it’s a older generation cpu, a 4k screen (sucks much more power) and a smaller battery.

So, speaking of the two laptops, how do they compare? Well, on the pro xps 13 9300 side:

  • smaller / lighter
  • no bezel around display
  • display is 4k
  • display is a touchscreen. I admit to almost never using the touchscreen. The only exception being miro boards… somehow touchscreen helps navigating those.

And cons of the xps 13 9300:

  • battery life not good
  • only 2 usb-c ports, thats it. (well, microsd also I guess if that counts for anything)
  • graphics are kind of sluggish
  • no fingerprint reader (it has one, but doesn’t work under linux)

Pros for the frame.work:

  • Better battery life
  • fast fast fast. Both cpu wise and graphics wise.
  • Screen is lower resolution, but I think it’s brighter/better colors
  • camera is higher res (1080p vs 720p)
  • Repairability/Replacability is much nicer. I spent an entire day replacing the keyboard in my dell.

And Cons for the frame.work:

  • Lower resolution screen, and 3:2 is weird after using 16:10
  • larger
  • bigger bezel around display

So, which laptop will I use day to day? I’m not fully sure yet. 🙂 I might keep switching or see how I feel after longer.

Overall the framework laptop is a pretty darn nice laptop. I really hope frame.work is around for many years and we get to see neat things like new cards, new motherboards, new screens and so on. A aarch64 motherboard would be fantastic!

WordPress openid is somewhat fragile, so if you want to ask me anything about this review, feel free to ping me on mastodon: @nirik@fosstodon.org or drop me an email: kevin@scrye.com

References / more reading:

Comments Off on Frame.work laptop: The hyperdetailed Fedora review

Some quick framework laptop power saving tips

by on 2021/10/26 at 1:58 pm
Posted In: fedora, linux

Some of these may apply to all laptops and some may be frame.work specific, but I thought I would throw them out there to help folks out.

  • Remove any of your modules that aren’t usb-c. At least the microsd and HDMI modules pull a small amount of power and prevent the laptop from going into the lowest power saving levels. Without doing this, I never get pc10 (the lowest power saving mode). framework is working on fixing this in a upcoming bios update.
  • Turn off the camera and microphone with the switches at the top of the display. Again, these draw a small amount of power and keep things from going to lowest power settings.
  • dynamic power profiles powersave mode is much better in my testing than tlp or tuned for saving power. It’s also easy to manage with workstation/gnome desktop. Others can use the powerprofilesctl command line tool.
  • powertop –auto-tune (or simply ‘systemctl enable powertop’ and reboot) seems to work ok on the framework laptops, with one exception: If you disable bluetooth and then try and re-enable it, it will find no controller. You will need to do a cold boot (completely power off and back on) or ‘rmmod btusb; modprobe btusb enable_autosuspend=n’ to get it back.
  • ‘echo powersupersave > /sys/module/pcie_aspm/parameters/policy’ provides some power savings and to me at least doesn’t seem to cause much in the way of slowdowns.
  • Make sure you aren’t hitting this bug in firefox ( https://bugzilla.redhat.com/show_bug.cgi?id=1936901 ) which basically causes it to start speech-dispatcher and then never ever let it go away. This was causing about 3-5% cpu use here, but it also causes fans to start up and no power savings to happen.
  • If you don’t mind using rpmfusion-nonfree items, installing intel-media-driver from there helps video playback nicely (depending on the video).
  • bluetooth and wifi both use a fair bit of power, if you don’t need them, do disable them for more savings.

Hope that helps someone out there get more battery life… 🙂

Comments Off on Some quick framework laptop power saving tips

Pinephone pro

by on 2022/03/17 at 3:46 pm
Posted In: fedora, linux

Once again it’s been a while since I’ve posted, but I am getting a bit caught up with life and am going to try and resume some more regular blogging.

Lets start with a quick post about the Pinephone pro. I had gotten not one, but two of the orig pinephones. Nice fun to play with, but so many non upstream patches to get everything working made it pretty much impossible that it would ever be able to have a Fedora spin. Also, it was really quite slow at pretty much everything. ;( Enter the pinephone pro that was announced late last year.

I signed up for the very first batch of pinephone pro’s and managed to get one! First, for some reason, the shipper (DHL international) decided to use USPS for the last leg of the shipping to my house. This is not ok, because I don’t actually have USPS service here. 🙁 I did finally manage to get a hold of the local postoffice to go in and claim my package from them.

The pro is very similar to the older pinephone in size, appearance and also accessories like batteries, usb dock and such. It’s a LOT faster than the old pinephone. Still probibly not up to a top tier phone, but it seems pretty similar in performance to my daily driver phone (a OnePlus 3t running /e/ os). Also, much more support is already upstream or more easily upstreamable.

The pinephone pro has a SPI (apparently all generations, although I don’t have a cite, this is just what I have heard). A SPI is a small flash area you can place a boot loader. There’s a fork of the uboot loader called ‘towboot’ that many folks are flashing there. towboot has nice support for a phone device. It lights up the led based on whats happening (before the screen), it lets you use volume and select to decide if you want to boot the eMMC (internal flash) or microsd card, etc. I finally broke down and ran the towboot installer. Worked fine and it’s nice to have a bit more control over the boot process. A group of folks are pushing pine64 to just ship with towboot on SPI in later batches, we will see how that goes.

Thanks to javierm, we have a Fedora kernel and uboot with pinephone pro patches on top in copr now, along with osbuild templates to use that kernel/uboot to make a raw image for the phone. There’s more patches to test and then all of them to upstream, but progress is being made.

Next steps for the mobility sig are to start getting a change together, update the phosh comps group, and get kickstarts together (in case we can land before fedora uses osbuild). Progress is slower than anyone would like, but I hope later this year we will have a pinephone pro image (or two)!

Comments Off on Pinephone pro

Another tale of a rawhide compose bug

by on 2022/04/21 at 2:45 pm
Posted In: fedora, linux

Astute observers will have noticed recently that Fedora rawhide composes stopped after 2022-04-13 and didn’t resume until 2022-04-19. A particularly odd bug was to blame. This is the story of that bug and my investigations of it.

A bit of background first on how nightly images are made currently. There’s a cron job that calls a script (called nightly.sh) in the pungi-fedora pagure.io git repo. It does a number of small associated things (like sending message bus messages of status, sending report emails, copying results around, etc) but the big thing it does is to call pungi. pungi in turn looks at it’s config (in that same pungi-fedora repo) and does all the heavy lifting of the compose, calling mostly out to koji to do things, but also doing some things locally. For images, pungi calls koji with some parameters (use this repo and name for kickstart file, use these compose repos for packages, etc). koji then calls different tools depending on what the image is. For livemedia the tool is livemedia-creator (in the lorax package), for qcow2/raw images it calls ImageFactory/oz to do the actual image build on a builder.

The compose on the 14th failed making a aarch64 Cloud-Base image. https://koji.fedoraproject.org/koji/taskinfo?taskID=85654099 with an odd traceback at the very very end of the build:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 17480: invalid start byte

So, my first go-to on these sorts of things is to look at what packages changed between the successfull compose on the 13th and the failed one on the 14th. That day of changes was pretty small, being a thursday in a week when everyone was focused on Fedora 36 final testing. The only thing that seemed like it could affect booting at all was a grub2 update. That update only added a ‘read’ module though, which normally I wouldn’t think could cause any issue, but perhaps it was a odd aarch64 toolchain issue? So, I tried some composes with grub2 untagged. No luck, still failed the same way.

Next I tried to get some more information about exactly what was not utf-8 that it was complaining about. I did this by doing some koji scratch image builds. But of course my scratch builds never seemed to fail, they all worked fine.

Of course then it was easter weekend, I got busy around the house and didn’t really dig back in to it until the following monday. This time I did some more scratch builds and finally managed to get it to fail correctly and with some added debug in oz on the builder to print out the data it was trying to convert to utf-8. At the very end of making the image (while it’s shutting down), it was getting a bunch of:

cpio: Malformed number <bunch of junk encoded weird stuff>

This of course was not something python wanted to convert to utf-8. It looked like a totally different encoding in there, so oz marked it a failure.

A sidebar here on how oz works. oz takes its inputs and tells libvirt to fire off a vm for it, running the installer with the kickstart provided, and serial console pointed to a socket. It then starts polling (by default every 10 seconds). If there’s disk or network activity on the vm, it gets data from the console and waits. If it hits the total timeout it kills the vm and returns a failure. If it doesnt see any disk or network activity for (default 300 seconds) it assumes the install has stalled/failed, and kills the vm and returns a failure. If the vm has shutdown, it assumes it’s successfull and returns. The output of the console is returned back to koji so you can see the log file in the koji task. However, as Adam Williamson pointed out: That last 10 second window never actually gets collected by oz for converting/adding to logs. If the install shuts down right after oz polls, the next time it will poll the vm is shutdown and it just returns.

So, what is this cpio output? Well, when you shut down a Fedora system, dracut calls “/usr/lib/dracut/dracut-initramfs-restore”. This script runs at the very end of the shutdown cycle and is supposed to copy out the initramfs to memory and then pivot your install to using that as / so it can unmount your real root directory cleanly. The script uses a wrapper called ‘skipcpio’ thats shipped with dracut. This wrapper skips past the first part of the initramfs, as this is just a small archive with the microcode to load that first thing when you boot. The next archive is the actual initramfs that does the heavy lifting. The script has no idea how you have compressed your initramfs (if you have), so it just blindly tries: not compressed? gzipped? bzipped? xz? lz4? lzop? zstd? The first try (not compressed) is the thing that results in the error spew. cpio bravely tries to unpack the zstd compressed initramfs and… has problems.

So, the flow is:

  • nightly.sh calls pungi
  • pungi calls koji to make Cloud-Base image
  • Oz calls libvirt to do install, starts polling.
  • Install boots, installs and finishes.
  • On shutdown dracut-initramfs-restore runs and because it tries uncompressed first, spews cpio errors
  • Oz polls again after the errors start, but before the vm is completely off.
  • Oz tries to convert the log to utf-8 and blows up.

Upstream dracut as it turns out just merged a PR to change the order it tries these things in (moving zstd to the first one) because it resulted in several seconds slower shutdowns. So, I just added that to the rawhide dracut package and everything started composing again.

So, what caused this to start happening now? Well, queue back to oz not getting the last bit of console output. My theory is that this has been happening for a long time, it’s just since it was at the very end of the output, it always got dropped on the floor and oz never saw it. For some reason now we are shutting down faster, or slower and it’s getting caught in that last poll that does have data.

Always fun digging into these sorts of things. 🙂

Comments Off on Another tale of a rawhide compose bug

Onlykey DUO

by on 2022/05/08 at 1:47 pm
Posted In: fedora, linux

Last year, I backed the onlykey DUO on kickstarter: https://www.kickstarter.com/projects/timsteiner/onlykey-duo-portable-protection-for-all-of-your-devices It seemed like a interesting device and I like that it’s fully opensource, unlike modern yubikeys.

The device finally arrived last month, and I’ve had a chance to play around with it some. Sadly, I don’t think it’s going to replace my yubikey anytime soon.

On the good side: The device itself is nicely constructed. It has a multicolored led on it that indicates which profile is in use (There are 4: green, blue, yellow, purple). It’s got 2 buttons on the end, so you can press one or the other or both at the same time and long or short presses for different slots. That means each profile has 6 ‘slots’ for a total of 24 in all 4 profiles. You can set a pin to lock the key which you have to enter before using it, along with a ‘self destruct’ pin that will wipe all configuration when entered.

On the bad side however, there’s a fair bit. The software to manage the onlykey is provided as either a ubuntu .deb or a snap. I tried to get the snap working with no luck at all, and ended up unpacking the deb to get things working. I looked into making a Fedora package but it’s a node app and has a pile of deps.

Next, I tried to enroll a otp for our Fedora account system, but found that the TOTP secret wouldn’t work. Further investigation showed that the onlykey NEO only supports sha1 for TOTP secrets and our account system uses SHA512. ;( There’s a old closed ticket about this on the onlykey firmware repo: https://github.com/trustcrypto/OnlyKey-Firmware/issues/101

There’s also no way to generate a ssh private key on the device (like you can using the opensc support on a yubikey). You can generate ecdsa sk openssh keys, which is great, but not too useful to me yet as RHEL7 and RHEL8 don’t support them.

So, at this point I would not recommend these devices unless you don’t need to interact with the Fedora account system or want to use the device with a Fedora linux install.

Comments Off on Onlykey DUO

New phone, who dis?

by on 2022/11/04 at 2:13 pm
Posted In: fedora, linux

In october, I went on 2 trips and after that several things became very clear: First, covid is still out there and you can still get it (as I did) and that my current trusty phone that I had been using for the last 7 years finally needed to be replaced.

The old phone is a one plus 3t and it’s been a great phone. I currently have been running /e/ on it (with no google) and it’s been fine. Unfortunately, the years have now taken their toll on the battery. When traveling I had to put it on super battery save just to have any battery left after a day away from a charger. On super battery save it’s slow slow slow, bordering on unusable. The sad thing is, if I could replace the battery I could probibly be fine with this phone for more years. It’s true it doesn’t take super great pictures, only has 64GB space, and has a bunch of scratches now. But alas, the battery is non replaceable, so I decided I needed to do something.

Of course the first thing I looked at was just using one of the 2 pinephones or 1 pinephone pro I already have. Sadly, they just aren’t good enough for a daily driver for me. They are slow, the battery life is also bad (on the pro at least), and… the biggest problem: The camera is just not good at all. I end up taking a lot of pictures and I really need them to be viewable. Finally, despite lots of work, there’s still a bunch of things non upstreamed, so I would have to run a custom kernel and a bunch of other non upstreamed parts.

Next, there are now some phones you can buy with /e/ pre-installed. There’s the Murena one and Teracube 2e. The one has more storage space, but otherwise they have less ram than my oneplus 3t. 🙂 This was a tempting option, but /e/ hadn’t been impressing me of late. Updates were few and far between, I had to do a bunch of tinkering to get on the latest stream (based on android 11).

The fairphone3/4 are interesting, but don’t seem to be available in the US at all. Of course I am sure I could get one, but likely it wouldn’t work with US telco’s.

I wanted to get something that would have a nice long life of updates also.

Finally I ended up deciding on just getting a unlocked pixel 7 and going with grapheneos on it. I deliberately choose the 7 over the 7 pro because it’s slightly smaller (but still big) and from all reviews I read had a better battery life. The pixel 7 gets 5 years of updates, which is about as good as you can do these days.

Why grapheneos? Well, I did not want to go back to being tied to google if I could help it and /e/ doesn’t really support any very modern phones. The free phone os’es (postmarketos, mobian, etc) also don’t support pretty new hardware either. However, I figure after a few years there’s a good chance something like a pixel will be supported by more of those and I can choose to jump to one of them if I want. grapheneos is basically ASOP (the “upstream” android) with a bunch of security enhancements added to it. The install process was pretty painless, but I did hit one problem where I tried to install with the web installer using firefox, then switched to chrome and got an error. I finally figured out that I forgot to close the firefox tab out and it was keeping the webusb locked so the other browser couldn’t install. 😉

Install went fine after that and I installed f-droid and got my applications and data moved over to the new phone. The biggest headache was of course signal. I had been using it for sms/mms after my last /e/ re-install, but they are dropping support now, so I had to export my sms’s back out of it to get them moved over. The export function doesn’t tell you that it doesn’t handle duplicates and you should wipe your sms db first, so I ended up with 2x my sms messages. Finally got that transfered over and signal deleted. signal could have been a great app, but they seem determined to made decisions that will drive them into irrelevance now. It’s sad.

Anyhow, I hope the pixel 7 will last me a few more years until I can get a modern phone and put Fedora on it. 🙂

Comments Off on New phone, who dis?

Some thoughts on a new home server

by on 2022/12/13 at 1:10 pm
Posted In: fedora, linux

I’ve been spending some of my time off in the last few days pondering replacing my old reliable home server with something new and shiny. I figured this might be a good time to write up some thoughts around this.

So, the first question that I am sure leaps to mind for people is: Home server? why on earth do you want one of those! Move it to “The Cloud”! Of course doing so would indeed have a number of advantages:

  • Better bandwith
  • No need to hassle with hardware, someone else would do that
  • Less noise and power usage at home
  • Depending on how deep in the clouds you go: less hassle running services

On the other hand it has real disadvantages to me:

  • No “real life” home setup to test/try/figure things out.
  • Never really 100% sure who has/owns/can do things with your data
  • Ability to mess with hardware, which can be kind of fun.
  • I have a small list of close friends who I provide services to. It’s fun to keep in touch with them that way and have something I can do for them.
  • Ability to mess with running a bunch of services, which can be kind of fun.
  • Paying a cloud provider recurring fees for something I could just buy once and not pay for over and over again seems like it could be a win, depending on the fees.

Someday I might give up and move things, but it’s not come fully to that yet. Email has been slowly getting more difficult to run on a non gigantic domain, but I’ve managed to overcome so far, so I will keep going until that becomes completely untenable. I really like having my data close by and knowing that I can go fix some problem when it happens. It’s also been a while, but I want to look at spinning up a home OpenShift instance so I can dig into it more and learn more about the low level parts of it. Might need to use OKD or k3s or something instead of OpenShift, but should let me find out more about how ks8 works.

All that background said, lets look at my current home server. It’s a Dell PowerEdge C1100/CS24-TY. I got it from https://deepdiscountservers.com long long long ago, along with another identical server. You can really get pretty great stuff there. It’s basically all the old compute that cloud companies have aged out. So, they are usually older, but tons of memory and disk and cpu. These ones I got have 72GB memory, 24 cpu threads, and 4 3.5″ hot swap drive bays in the front. The second one I got I used for a long time as a test machine, but it has a slightly too old cpu to do power management, so it’s really really loud. The main server does do some power management, but it’s pretty loud too. In my current house I have a closet for computer stuff, but even with the door closed I am near enough to it that I can hear the server running. Of course I can also usually hear the fridge in the kitchen running too. The drives I currently have are 3TB 7200 rpm hitachi’s. Which have also been quite reliable. The server has a pci card in it for some more network ports. It serves as my main firewall / virthost / storage server.

So, why replace it? Well, it was made in the fall of 2012. Yes, thats 10 years old now. Thats ages in computer hardware. It’s slow. The cpu is pretty slow and the storage is super slow. It’s running the 7200rpm spinning disks on a 3GB/sec sata bus (They can do 6GB/s). Taking backups or moving a bunch of things or running a postgresql vacuum just takes ages. It’s also loud. Not earthshatteringly so, and like I mentioned our fridge is also kinda loud, but there’s a lot of times when the fridge compressor is off and I can hear the server distinctly. Finally, it’s fun to look at things and then install and assembe them. Computer geeks gotta geek. Also, this is perhaps a chance for me to play with some things I haven’t yet, like perhaps moving over to a AMD cpu instead of intel or raid on nvme, etc

So, my first thought was to just get another rackmount from deepdiscountservers, which would work fine, but it would be intel based, basically just a newer version of what I have now with more memory and cpus. The cpus would be intel and while newer servers are likely to do throttling better, I don’t think the noise would be all that much lower. rack mount servers are just not designed to be quiet.

Next, I poked around on the net and ran accross silentpc.com, which has some interesting computers on offer. I focused in on the “Powerhouse Ryzen PC” box. It’s a tower case, which is not ideal, but I’m sure I can fit it in somewhere. It’s a Ryzen cpu, a power supply that can power completely off if things are idle, super quiet fans, etc. It’s got enough room so I can move my existing 4 drives over to it (and add in a 5th that I have I was keeping for a spare). Only 2 NVME slots available, but… that takes me into an aside I had:

Most motherboards these days I have seen have just a few NVME slots on them. However, they make PCIe cards that have NVME slots (one, two, or four). The four slot NVME’s are interesting. You need to have a motherboard that supports “pci bifurication” on the slot you are putting it in. If you don’t, you can only see one drive and thats it. If you do, the motherboard takes the x16 slot and carves it into 4 x4 slots and you see all the drives.

From what I have been able to gather the Powerhouse Ryzen PC has a motherboard that has 1 pcie slot that can do bifurcation (but I asked them in email to make sure). If so, then I can get it with 2 NVME’s and raid1 them for now, move the 3.5″ drives over with most of my data, and then down the road I can get a PCIe 4 NVME card and stick 4 NVME’s in there and raid 6 them with the 2 on the MB and then perhaps retire the spinning drives. 🙂 Sadly, their web interface seems to only offer nvidia cards (which I really don’t want), but I asked them in email and they can indeed do other cards. So, waiting to hear back, but I think this might work out nicely for a new box. If it does, I’m also thinking about moving the existing 1U boxes out to the garage and see if I can set them up with a wake on lan or the like so I can use them if I need to test something.

Looking forward to tinkering with it (or looking more if this one doesn’t pan out).

└ Tags: , , ,
Comments Off on Some thoughts on a new home server