On generators and physics

by on 2022/12/30 at 1:20 pm
Posted In: fedora

On tuesday here we had a massive wind storm come through. 25-35mph winds with gusts up to 75mph or more, along with heavy rains and near freezing temps. It’s a pretty crazy thing to experience in the forest. 250-300ft douglas fir trees swaying 30 or 40 degrees in the winds, branches coming off and flying by. A large gust came through and broke 3 of our large trees off and pulled another one out by it’s roots. Very sad to see these big trees go.

But to bring things back to a tech focus, also on tuesday due to the storm, we lost power for about 16 hours. We had gotten a generator to handle just this situation and had run it happily for a spring outage earlier in the year. We were running on 20lb propane tanks (since they are so easy to get). In the spring we were getting about 5-6 hours of use out of a tank.

So, I fired up the generator and we got about 3 hours out of a partial tank, where I switched to our last full tank and headed into town to stock up. I exchanged the empty tank and got 3 more. That last full one was out when I got home, so I swapped another full one in and… it stopped after an hour. At first we were wondering if we had some heavy load on the generator we didn’t realize, but on checking that wasn’t the case. We looked for leaks in propane, but wait… the tank is all covered in frost and all frozen up. Here we realized the real problem.

Propane in tanks is in a liquid state, under pressure. When you open the valve, propane comes out and changes into a gas to power your whatever. This state change requires energy. Usually this is just pulled from the tank shell and everything is fine. However, when it’s cold, the tank gets colder and colder and can’t supply that energy anymore and with no gas flowing anymore, the regulator cuts off the generator.

So, at first we tried putting the tank in the garage instead of outside, wrapping it in blankets, etc. But that didn’t really help too much. So, then we got a tub and filled it with water and put the tank in there. That helped quite a lot, we managed to get about 4 hours out of a tank then. At the end the tank had a 1 inch or so ‘wrapper’ of ice, but it did get the tank lasting longer.

So, whats the solution here? Well, I think getting just one larger tank (say a 100lb one) would help a lot as it will have a ton more surface area. Not as easy you move around, but oh well. Also, they make apparently powered heating blankets for the smaller tanks. I’ll probibly pick up one of those in the short term. So, lesson learned: temperature has a lot to do with how well your propane generator will work.

└ Tags: , , ,
Comments Off on On generators and physics

Some thoughts on a new home server

by on 2022/12/13 at 1:10 pm
Posted In: fedora, linux

I’ve been spending some of my time off in the last few days pondering replacing my old reliable home server with something new and shiny. I figured this might be a good time to write up some thoughts around this.

So, the first question that I am sure leaps to mind for people is: Home server? why on earth do you want one of those! Move it to “The Cloud”! Of course doing so would indeed have a number of advantages:

  • Better bandwith
  • No need to hassle with hardware, someone else would do that
  • Less noise and power usage at home
  • Depending on how deep in the clouds you go: less hassle running services

On the other hand it has real disadvantages to me:

  • No “real life” home setup to test/try/figure things out.
  • Never really 100% sure who has/owns/can do things with your data
  • Ability to mess with hardware, which can be kind of fun.
  • I have a small list of close friends who I provide services to. It’s fun to keep in touch with them that way and have something I can do for them.
  • Ability to mess with running a bunch of services, which can be kind of fun.
  • Paying a cloud provider recurring fees for something I could just buy once and not pay for over and over again seems like it could be a win, depending on the fees.

Someday I might give up and move things, but it’s not come fully to that yet. Email has been slowly getting more difficult to run on a non gigantic domain, but I’ve managed to overcome so far, so I will keep going until that becomes completely untenable. I really like having my data close by and knowing that I can go fix some problem when it happens. It’s also been a while, but I want to look at spinning up a home OpenShift instance so I can dig into it more and learn more about the low level parts of it. Might need to use OKD or k3s or something instead of OpenShift, but should let me find out more about how ks8 works.

All that background said, lets look at my current home server. It’s a Dell PowerEdge C1100/CS24-TY. I got it from https://deepdiscountservers.com long long long ago, along with another identical server. You can really get pretty great stuff there. It’s basically all the old compute that cloud companies have aged out. So, they are usually older, but tons of memory and disk and cpu. These ones I got have 72GB memory, 24 cpu threads, and 4 3.5″ hot swap drive bays in the front. The second one I got I used for a long time as a test machine, but it has a slightly too old cpu to do power management, so it’s really really loud. The main server does do some power management, but it’s pretty loud too. In my current house I have a closet for computer stuff, but even with the door closed I am near enough to it that I can hear the server running. Of course I can also usually hear the fridge in the kitchen running too. The drives I currently have are 3TB 7200 rpm hitachi’s. Which have also been quite reliable. The server has a pci card in it for some more network ports. It serves as my main firewall / virthost / storage server.

So, why replace it? Well, it was made in the fall of 2012. Yes, thats 10 years old now. Thats ages in computer hardware. It’s slow. The cpu is pretty slow and the storage is super slow. It’s running the 7200rpm spinning disks on a 3GB/sec sata bus (They can do 6GB/s). Taking backups or moving a bunch of things or running a postgresql vacuum just takes ages. It’s also loud. Not earthshatteringly so, and like I mentioned our fridge is also kinda loud, but there’s a lot of times when the fridge compressor is off and I can hear the server distinctly. Finally, it’s fun to look at things and then install and assembe them. Computer geeks gotta geek. Also, this is perhaps a chance for me to play with some things I haven’t yet, like perhaps moving over to a AMD cpu instead of intel or raid on nvme, etc

So, my first thought was to just get another rackmount from deepdiscountservers, which would work fine, but it would be intel based, basically just a newer version of what I have now with more memory and cpus. The cpus would be intel and while newer servers are likely to do throttling better, I don’t think the noise would be all that much lower. rack mount servers are just not designed to be quiet.

Next, I poked around on the net and ran accross silentpc.com, which has some interesting computers on offer. I focused in on the “Powerhouse Ryzen PC” box. It’s a tower case, which is not ideal, but I’m sure I can fit it in somewhere. It’s a Ryzen cpu, a power supply that can power completely off if things are idle, super quiet fans, etc. It’s got enough room so I can move my existing 4 drives over to it (and add in a 5th that I have I was keeping for a spare). Only 2 NVME slots available, but… that takes me into an aside I had:

Most motherboards these days I have seen have just a few NVME slots on them. However, they make PCIe cards that have NVME slots (one, two, or four). The four slot NVME’s are interesting. You need to have a motherboard that supports “pci bifurication” on the slot you are putting it in. If you don’t, you can only see one drive and thats it. If you do, the motherboard takes the x16 slot and carves it into 4 x4 slots and you see all the drives.

From what I have been able to gather the Powerhouse Ryzen PC has a motherboard that has 1 pcie slot that can do bifurcation (but I asked them in email to make sure). If so, then I can get it with 2 NVME’s and raid1 them for now, move the 3.5″ drives over with most of my data, and then down the road I can get a PCIe 4 NVME card and stick 4 NVME’s in there and raid 6 them with the 2 on the MB and then perhaps retire the spinning drives. 🙂 Sadly, their web interface seems to only offer nvidia cards (which I really don’t want), but I asked them in email and they can indeed do other cards. So, waiting to hear back, but I think this might work out nicely for a new box. If it does, I’m also thinking about moving the existing 1U boxes out to the garage and see if I can set them up with a wake on lan or the like so I can use them if I need to test something.

Looking forward to tinkering with it (or looking more if this one doesn’t pan out).

└ Tags: , , ,
Comments Off on Some thoughts on a new home server

New phone, who dis?

by on 2022/11/04 at 2:13 pm
Posted In: fedora, linux

In october, I went on 2 trips and after that several things became very clear: First, covid is still out there and you can still get it (as I did) and that my current trusty phone that I had been using for the last 7 years finally needed to be replaced.

The old phone is a one plus 3t and it’s been a great phone. I currently have been running /e/ on it (with no google) and it’s been fine. Unfortunately, the years have now taken their toll on the battery. When traveling I had to put it on super battery save just to have any battery left after a day away from a charger. On super battery save it’s slow slow slow, bordering on unusable. The sad thing is, if I could replace the battery I could probibly be fine with this phone for more years. It’s true it doesn’t take super great pictures, only has 64GB space, and has a bunch of scratches now. But alas, the battery is non replaceable, so I decided I needed to do something.

Of course the first thing I looked at was just using one of the 2 pinephones or 1 pinephone pro I already have. Sadly, they just aren’t good enough for a daily driver for me. They are slow, the battery life is also bad (on the pro at least), and… the biggest problem: The camera is just not good at all. I end up taking a lot of pictures and I really need them to be viewable. Finally, despite lots of work, there’s still a bunch of things non upstreamed, so I would have to run a custom kernel and a bunch of other non upstreamed parts.

Next, there are now some phones you can buy with /e/ pre-installed. There’s the Murena one and Teracube 2e. The one has more storage space, but otherwise they have less ram than my oneplus 3t. 🙂 This was a tempting option, but /e/ hadn’t been impressing me of late. Updates were few and far between, I had to do a bunch of tinkering to get on the latest stream (based on android 11).

The fairphone3/4 are interesting, but don’t seem to be available in the US at all. Of course I am sure I could get one, but likely it wouldn’t work with US telco’s.

I wanted to get something that would have a nice long life of updates also.

Finally I ended up deciding on just getting a unlocked pixel 7 and going with grapheneos on it. I deliberately choose the 7 over the 7 pro because it’s slightly smaller (but still big) and from all reviews I read had a better battery life. The pixel 7 gets 5 years of updates, which is about as good as you can do these days.

Why grapheneos? Well, I did not want to go back to being tied to google if I could help it and /e/ doesn’t really support any very modern phones. The free phone os’es (postmarketos, mobian, etc) also don’t support pretty new hardware either. However, I figure after a few years there’s a good chance something like a pixel will be supported by more of those and I can choose to jump to one of them if I want. grapheneos is basically ASOP (the “upstream” android) with a bunch of security enhancements added to it. The install process was pretty painless, but I did hit one problem where I tried to install with the web installer using firefox, then switched to chrome and got an error. I finally figured out that I forgot to close the firefox tab out and it was keeping the webusb locked so the other browser couldn’t install. 😉

Install went fine after that and I installed f-droid and got my applications and data moved over to the new phone. The biggest headache was of course signal. I had been using it for sms/mms after my last /e/ re-install, but they are dropping support now, so I had to export my sms’s back out of it to get them moved over. The export function doesn’t tell you that it doesn’t handle duplicates and you should wipe your sms db first, so I ended up with 2x my sms messages. Finally got that transfered over and signal deleted. signal could have been a great app, but they seem determined to made decisions that will drive them into irrelevance now. It’s sad.

Anyhow, I hope the pixel 7 will last me a few more years until I can get a modern phone and put Fedora on it. 🙂

Comments Off on New phone, who dis?

Onlykey DUO

by on 2022/05/08 at 1:47 pm
Posted In: fedora, linux

Last year, I backed the onlykey DUO on kickstarter: https://www.kickstarter.com/projects/timsteiner/onlykey-duo-portable-protection-for-all-of-your-devices It seemed like a interesting device and I like that it’s fully opensource, unlike modern yubikeys.

The device finally arrived last month, and I’ve had a chance to play around with it some. Sadly, I don’t think it’s going to replace my yubikey anytime soon.

On the good side: The device itself is nicely constructed. It has a multicolored led on it that indicates which profile is in use (There are 4: green, blue, yellow, purple). It’s got 2 buttons on the end, so you can press one or the other or both at the same time and long or short presses for different slots. That means each profile has 6 ‘slots’ for a total of 24 in all 4 profiles. You can set a pin to lock the key which you have to enter before using it, along with a ‘self destruct’ pin that will wipe all configuration when entered.

On the bad side however, there’s a fair bit. The software to manage the onlykey is provided as either a ubuntu .deb or a snap. I tried to get the snap working with no luck at all, and ended up unpacking the deb to get things working. I looked into making a Fedora package but it’s a node app and has a pile of deps.

Next, I tried to enroll a otp for our Fedora account system, but found that the TOTP secret wouldn’t work. Further investigation showed that the onlykey NEO only supports sha1 for TOTP secrets and our account system uses SHA512. ;( There’s a old closed ticket about this on the onlykey firmware repo: https://github.com/trustcrypto/OnlyKey-Firmware/issues/101

There’s also no way to generate a ssh private key on the device (like you can using the opensc support on a yubikey). You can generate ecdsa sk openssh keys, which is great, but not too useful to me yet as RHEL7 and RHEL8 don’t support them.

So, at this point I would not recommend these devices unless you don’t need to interact with the Fedora account system or want to use the device with a Fedora linux install.

Comments Off on Onlykey DUO

Another tale of a rawhide compose bug

by on 2022/04/21 at 2:45 pm
Posted In: fedora, linux

Astute observers will have noticed recently that Fedora rawhide composes stopped after 2022-04-13 and didn’t resume until 2022-04-19. A particularly odd bug was to blame. This is the story of that bug and my investigations of it.

A bit of background first on how nightly images are made currently. There’s a cron job that calls a script (called nightly.sh) in the pungi-fedora pagure.io git repo. It does a number of small associated things (like sending message bus messages of status, sending report emails, copying results around, etc) but the big thing it does is to call pungi. pungi in turn looks at it’s config (in that same pungi-fedora repo) and does all the heavy lifting of the compose, calling mostly out to koji to do things, but also doing some things locally. For images, pungi calls koji with some parameters (use this repo and name for kickstart file, use these compose repos for packages, etc). koji then calls different tools depending on what the image is. For livemedia the tool is livemedia-creator (in the lorax package), for qcow2/raw images it calls ImageFactory/oz to do the actual image build on a builder.

The compose on the 14th failed making a aarch64 Cloud-Base image. https://koji.fedoraproject.org/koji/taskinfo?taskID=85654099 with an odd traceback at the very very end of the build:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 17480: invalid start byte

So, my first go-to on these sorts of things is to look at what packages changed between the successfull compose on the 13th and the failed one on the 14th. That day of changes was pretty small, being a thursday in a week when everyone was focused on Fedora 36 final testing. The only thing that seemed like it could affect booting at all was a grub2 update. That update only added a ‘read’ module though, which normally I wouldn’t think could cause any issue, but perhaps it was a odd aarch64 toolchain issue? So, I tried some composes with grub2 untagged. No luck, still failed the same way.

Next I tried to get some more information about exactly what was not utf-8 that it was complaining about. I did this by doing some koji scratch image builds. But of course my scratch builds never seemed to fail, they all worked fine.

Of course then it was easter weekend, I got busy around the house and didn’t really dig back in to it until the following monday. This time I did some more scratch builds and finally managed to get it to fail correctly and with some added debug in oz on the builder to print out the data it was trying to convert to utf-8. At the very end of making the image (while it’s shutting down), it was getting a bunch of:

cpio: Malformed number <bunch of junk encoded weird stuff>

This of course was not something python wanted to convert to utf-8. It looked like a totally different encoding in there, so oz marked it a failure.

A sidebar here on how oz works. oz takes its inputs and tells libvirt to fire off a vm for it, running the installer with the kickstart provided, and serial console pointed to a socket. It then starts polling (by default every 10 seconds). If there’s disk or network activity on the vm, it gets data from the console and waits. If it hits the total timeout it kills the vm and returns a failure. If it doesnt see any disk or network activity for (default 300 seconds) it assumes the install has stalled/failed, and kills the vm and returns a failure. If the vm has shutdown, it assumes it’s successfull and returns. The output of the console is returned back to koji so you can see the log file in the koji task. However, as Adam Williamson pointed out: That last 10 second window never actually gets collected by oz for converting/adding to logs. If the install shuts down right after oz polls, the next time it will poll the vm is shutdown and it just returns.

So, what is this cpio output? Well, when you shut down a Fedora system, dracut calls “/usr/lib/dracut/dracut-initramfs-restore”. This script runs at the very end of the shutdown cycle and is supposed to copy out the initramfs to memory and then pivot your install to using that as / so it can unmount your real root directory cleanly. The script uses a wrapper called ‘skipcpio’ thats shipped with dracut. This wrapper skips past the first part of the initramfs, as this is just a small archive with the microcode to load that first thing when you boot. The next archive is the actual initramfs that does the heavy lifting. The script has no idea how you have compressed your initramfs (if you have), so it just blindly tries: not compressed? gzipped? bzipped? xz? lz4? lzop? zstd? The first try (not compressed) is the thing that results in the error spew. cpio bravely tries to unpack the zstd compressed initramfs and… has problems.

So, the flow is:

  • nightly.sh calls pungi
  • pungi calls koji to make Cloud-Base image
  • Oz calls libvirt to do install, starts polling.
  • Install boots, installs and finishes.
  • On shutdown dracut-initramfs-restore runs and because it tries uncompressed first, spews cpio errors
  • Oz polls again after the errors start, but before the vm is completely off.
  • Oz tries to convert the log to utf-8 and blows up.

Upstream dracut as it turns out just merged a PR to change the order it tries these things in (moving zstd to the first one) because it resulted in several seconds slower shutdowns. So, I just added that to the rawhide dracut package and everything started composing again.

So, what caused this to start happening now? Well, queue back to oz not getting the last bit of console output. My theory is that this has been happening for a long time, it’s just since it was at the very end of the output, it always got dropped on the floor and oz never saw it. For some reason now we are shutting down faster, or slower and it’s getting caught in that last poll that does have data.

Always fun digging into these sorts of things. 🙂

Comments Off on Another tale of a rawhide compose bug