Day -2 (monday 2025-06-02)

My travels started on Monday the 2nd. I got up early, grabbed a cup of coffee and checked in on things before heading for the airport. It's a great thing I did as I managed to block some scrapers that were causing the python mass rebuild to go very very slowly. Pesky scrapers. Then 2 hour drive to the airport in Portland. Some traffic, but no real delays. Then a quick bite to eat and my first flight: PDX to AMS. It's about a 9 hour flight, which is pretty crazy, but I definitely like it better than more smaller hops. There's less to go wrong.

As a side note, I cannot stress to others enough how much noise canceling headphones really make plane flights more bearable. They cut off the nasty drone of the engines/wind and make it vastly less painfull. I wore my headphones with noise canceling on all the time, even when I wasn't listening to anything.

On these long flights I can't really sleep, so I like to read ebooks and catch up on podcasts. This flight I listened to some great song exploder episodes ( "our house" and "everybody wants to rule the world") and some radiolab and others.

Then arrival in AMS. I only had a 1.5 hour layover, which it turned out was just perfectly timed. I managed to get through the customs line and to the new gate just before they started boarding for Pague. They did make me check my bag here because the flight was so full, but that was fine.

Day -1 (tuesday 2025-06-03)

Got into Prague, got my bag and got a uber to the hotel. The hotel checkin was not until 3pm according to their website but they managed to already have my room ready, so I was able to check in and decompress.

I then met Carl and Noel at a resturant nearby for a nice lunch. We chatted on all kinds of topics.

Back to the hotel to relax a bit. I was determined to try and stay up and go to sleep later so my schedule would shift, but then I decided to just lay down for a few minutes and bam! I did wake up around 8pm local time and went back to bed, and I did wake up early, but that was after getting a lot of sleep.

Day 0 (wed 2025-06-04)

I met up with tons of folks at breakfast at the hotel and we went over to the venue. The hotel I was staying at was about 1.5 blocks from the venue hotel, it worked out just fine.

Some of us went to a outside air food court place for lunch. It was nice. I had some tacos and we had some more conversations at the lunch table.

The afternoon I went to a meetup between fesco and council members that were present. I think there was some productive discussion there. There were a lot of suggestions on how fesco and the council could communicate more and how the council could communicate better with the communty and what sort of workflows might make sense. I think it mostly resulted in some council action items, but also on the fesco side more communication to the council on difficult topics.

After that I was off to the sponsor dinner. This is a dinner with folks from the groups/companies that sponsored flock along with fesco and council members. This time it was at a lovely resturant that was a 600+ year old underground wine cellar! https://www.napekle.cz/en/ The food was great and the conversations were also... great!

Back to the hotel and in bed around midnight.

Day 1 (thursday 2025-06-05)

Flock begins!

We had opening remarks and handoff of the Fedora Project Leader baton (wand? septere? curling boom?). I was hit by how long I have been around this community. I met Jef back in the fedora.us/fedora-extras days, almost 20 years ago now, and Matthew showing a picture of his kids when he first became FPL and now and how much they had grown. We are all getting older and we really need to bring in new folks to carry the fedora foundations forward.

Then, right after that was the 'meet your FESCo' panel. We didn't have all FESCo members present, but a good many of them. We did a quick introduction and then took questions from the audience. I'm glad Peter was there and asked about the provenpackager stuff from eariler this year. I hope answers were helpfull. There were questions all over the place, new arches, ai, workflows, provenpackagers, etc. Do view the video if you are interested.

Next I had planned to go to the forgejo talk, but then... as with so many times in this flock, I got in discussions in the 'hallway track' with people and wasn't able to make it there in time. :( I hope to catch the recording (along with many other talks).

Then lunch at the venue, but this time I signed up for the mentor/metee lunch matching. There was only 3 of us at the Infra/Releng table, but we had a great time I thought. I was happy to share what I could, and Justin and Hristo had a lot of good perspectives. I hope I helped them out, and I know they gave me things to think about. Overall it was very nice. It might have been better with more people, but I'd definitely sign up for this again at another flock.

After lunch I spent a lot of time in the hallway track talking to folks. One super great surprise was that Toshio was there! I got to chat with him a bunch over flock and it was so nice to see him again. Later the next day another person I hadn't seen in a while appeared too: Vipul! I hadn't been too much in touch with him since he moved to a new job at the UN, but it was super great to see him as well (even though I did not recognize him at first with his new glasses!).

It would likely be too hard to list all the people I talked to about all the things I talked to them about but some themes became clear:

• There was a lot of talk about AI. Yes, the usual funny stores about how AI got it wrong or was used in a stupid way, but also a lot of 'how can we leverage this stuff in a resonable way and how can we make AI more in fitting with our foundations'.

• A lot of discussion about community building, communication and bringing in new contributors.

I did mange to get to Greg's talk on "Build it and they will come" and other myths. Some great stuff there around community building. I particularly liked some of the examples, which showed things we do wrong all the time. Things to think about and leverage.

Then, thursday night we had a CLE dinner for all the members of my team in Red Hat. It turned out to be at the same place we went for lunch on tuesday, but thats fine. It was good. Some good converations there where I got to chat with Julia and Patrik (although it was kind of loud there, so it was hard hearing anything).

After that some of us rushed off to the Candy Swap. Always a fun event. This time it wasn't at the venue, but instead at a 'games cafe'. They had a full bar there and a bunch of games. It was kind of small, but it worked out ok for our crowd. After the candy swap I got to chat with Stephen about books and movies. We always catch up this way and share nice books we have read. We were joined by Peter and Troy too, so I have a list of books to look up now.

Day 2 (Friday 2025-06-06)

Friday came too early after too little sleep.

I went to the 'what about a better dist-git' talk. Some nice information there, but I think I knew much of it before. There were some good questions starting, but I got pulled out to the hallway track, so I will need to look back at those.

Some more hallway discussions and then off to the "One year in: Microsoft as a Fedora Contributor" talk by Bex. It was great to see the parts of the project that microsoft folks are contributing to, I don't think many people realize it, so it was great to get some visibility to their efforts. I'm personally very happy that Jeremy has been helping out with our signing infra and cloud uploads. Thats really important stuff that we didn't have anyone to drive forward until he was able to do so.

I really planned to go to the Fedora Server talk next, but then again I got into discussions in the hallway until I had missed it. :(

After lunch I went to some lightning talks. This was a last minute thing as the speaker in that slot was not able to be there, but wow... fedora contributors are always ready with talks a the drop of a fedora. I really liked Emma's presentation about design. It's not something developers think about or realize, but we should! Lots of other great ones too!

I went to Greg's discourse tips and tricks. Learned a few things, but I would definitely recommend people who aren't up on discourse to watch the recording. It will help you out getting started!

Then more hallway and the Fedora Council AMA. There were some good questions here and some good discussions about various council related topics. I probibly need to watch the recording even thought I was there because I was tired after a long day.

There was there some short closing remarks (even though there would be workshops the following day) from Justin. he thanked all the folks who helped make flock happen and gave info on the evening event and workshops the next day. There was one thing missing however: We should have all thanked Justin! I know flock is a massive undertaking, and I am sure there were issues I have no idea about, but from my side flock was a great success! Thank you Justin!

The evening event was a boat ride with dinner. We did this same event last time flock was in pague. It was fun then and again now. I had Troy and Evan and Jens at my dinner table and we had a bunch of great discussion about shirts, travel, home assistant, people we all knew and more. Then after dinner everyone mingled on the top deck until they kicked us off the boat around 10pm.

A group of us then went to a beer place near the hotel for a few more beers. Some more talk there about... lots of things. I managed to get back and in bed around midnight (thats the theme).

Day 3 (Saturday 2025-06-07)

The last day was workshops and breakouts. I went to the start of the initial setup hackfest, but I knew much of the information that they were going over, so I allowed myself to be pulled into the hallway again.

After lunch, I went to the distgit implementation in forgejo talk. There was some good discussion about workflows and how things could work. We did get a bit off topic I think with talking about provenpackagers, but I guess it's all related. I'm really looking forward to us using forgejo here.

I did go to Aurélien's improve fedora infra applications, but again I kind of knew the background here, so I got pulled off into some other things:

• I had a (I hope) nice talk with Peter about server setup and deliverables. I do owe him some more docs/info on one part of it I could not happen to recall, but hopefully this gives him info he needs to work on the server deliverables some more.

• A talk with a few folks about plans for handling the matrix spam issues. We came up with some proposed short term and longer term plans. Look for plans asking for feedback in the coming weeks. We really need to get things workable there.

• A nice talk with the person who actually started the opensuse foundation. He was there looking to see if it would be useful to start a fedora one. I don't know the answer, but It sounded very interesting.

• Got to catch up on ARM stuff with Peter (another one). Always great to talk to him and hopefully we can find some ways forward for the x1e / snapdragon laptops sooner rather than later.

• The new FPL, Jef. I was in several conversations with him. He seemed to be keeping afloat with all the stuff going on, which I thought was amazing. I'm sure he will be a great FPL.

• Some good discussions with Greg. He's on my team at Red Hat and working with myself and Fabian on infra, so we were able to have a few higher bandwith discussions that should help us in the coming weeks.

• Got to catch up a bit with Fabian on a bunch of topics.

• Had a few nice discussions with Brendan (my grand boss).

After things closed out a bunch of us went to a Dim Sum place nearby for dinner. More books discussion, along with travels and interesting places.

I went back to the hotel and crashed before 9am, which was good, because my flight to AMS was at 6am the next day.

Day 4 (Sunday 2025-06-08)

Travel back home. Got a cab with Brendan at 4am, got to the airport, through security and onto my first flight in time at 6am. Then, in AMS, walking accross the airport, grabbed a quick breakfast and got to the terminal in time to get on my AMS to PDX flight. On the way back my usual podcast and books didn't work because I was so sleepy. It was hard to pay attention. So, instead I watched a few movies: the new marvel captain america one and the second dune movie. Both were ok, but nothing super amazing. My flight from AMS left at 10am, and arrived in PDX at 11am the same day, but it was definitely not a 1 hour flight.

I was worried about customs coming back to the US, but it turned out that they just asked me if I had any food, I said nope, they said ok.

Then the 2 hour drive home. I was pretty sleepy at this point, but I got some cafene and was able to make it home fine finally. There was a lot of stop and go traffic this time, which was anoying, so the drive took an extra hour or so.

Health and diet

I'm going to digress here about heath, diet and conferences, so if that doesn't interest you, feel free to skip it.

I gained weight on this trip, and thats unfortunately pretty usual. I think there's several reasons for this. If you are traveling you may not have much choice of food, or might not know when next you will get food. Breakfast is often included in hotels, and it's almost always a 'all you can eat' buffet type thing. I can eat a lot.

But also, conferences always seem to put food in front of you, and I am pretty bad about just eating food if it's there. I don't want it to go to waste, and it's something I do as a background.

Of course the simple answer is to just have more willpower and eat smaller amounts, but it's not simple to do that sometimes. I don't know if there's much that could be done from a conference point of view. I guess less food with coffee/tea/water breaks? Or moving away from buffets?

Anyhow, something to think about.

Matrix spam

The horrific matrix spam ramped up before flock and measures were put into place that blocked it. Some of those measures are pretty heavy handed, but we really did not want to have to handle this junk at flock. As I mentioned above we did some up with some plans, and I hope we can make things still safe but more open soon.

Some links

The fedora youtube channel should have talk recordings soon: https://www.youtube.com/@fedora

The list of talks and slides, etc: https://cfp.fedoraproject.org/flock-to-fedora-2025/schedule/

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114655707058309116

Datacenter move

The switch week is still scheduled for the week of June 30th. We made some progress this last week on installs. Got everything setup to install a bunch of servers. I installed a few and kept building out services. I was mostly focusing on getting things setup so I could install openshift clusters in both prod and staging. That will let us move applications. I also setup to do rhel10 installs and installed a test virthost. There's still a few things missing from epel10 that we need: nagios clients, collectd (thats on me) and zabbix clients, otherwise the changes were reasonably minor. I might try and use rhel10 for a few things, but I don't want to spend a lot of time on it as we don't have much time.

Flock

Flock is next week! If you are looking for me, I will be traveling basically all monday and tuesday, then in prague from tuesday to very early sunday morning, when I travel back home.

If you are going to flock and want to chat, please feel free to catch me and/or drop me a note to try and meet you. Happy to talk!

If you aren't going to flock, I'm hoping everything is pretty quiet infrastructure wise. I will try and check in on any major issues, but do try and file tickets on things instead of posting to mailing lists or matrix.

I'd also like to remind everyone going to flock that we try and not actually decide anything there. It's for discussion and learning and putting a human face on your fellow contributors. Make plans, propose things definitely, just make sure after flock you use our usual channels to discuss and actually decide things. Deciscions shouldn't be made offline where those not present can't provide input.

I'm likely to do blog posts about flock days, but may be delayed until after the event. There's likely not going to be a regular saturday post next week from me.

Arm laptop

So, I successfully used this Lenovo slim7x all week, so I guess I am going to try and use it for my flock travel. Hopefully it will all work out. :)

Issues I have run into in no particular order:

• There are a bunch of various people working on various things, and all of that work touches the devicetree file. This makes it a nightmare to try and have a dtb with working bluetooth, ec, webcam, sound, suspend, etc. I really hope a bunch of this stuff lands upstream soon. For now I just Have a kernel with bluetooth and ec working and am ignoring sound and webcam.

• s2idle sleep "works", but I don't trust it. I suspended the other day when I was running some errands, and when I got home, the laptop had come on and was super super hot (it was under a jacket to make it less a theft target). So, I might just shutdown most of the time traveling. There's a patch to fix deep sleep, but see above.

• I did wake up one day and it had rebooted, no idea why...

• Otherwise everything is working fine and it's pretty nice and zippy.

• Battery life is... ok. 7-8 hours. It's not hitting the lowest power states yet, but that will do I think for my needs for now.

So, hopefully it will all work. :)

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114603298176306720

Datacenter Move

Due to delays in getting network to new servers and various logistics, We are going to be moving the switcharoo week to the week of June 30th. It was set for June 16th, but thats just too close timing wise, so we are moving it out two weeks. Look for a community blog post and devel-announce post next week on this. I realize that that means that friday is July 4th (a holiday in the US), but we hope to do the bulk of switching things on monday and tuesday of that week, and leave only fixing things for wed and thursday.

We did finally get network for the new servers last week. Many thanks to all the networking folks who worked hard to get things up and running. With some network I was able to start bootstrapping infrastructure up. We now have a bastion host, a dhcp/tftp host and a dns server all up and managed via our existing ansible control host like all the rest of our hosts.

Friday was a recharge day at Red Hat, and monday is the US Memorial day holiday, but I should be back at deploying things on tuesday. Hopefully next week I will get a initial proxy setup and can then look at doing openshift cluster installs.

Flock

The week after next is flock! It came up so fast. I do plan on being there (I get into prague late morning on the 3rd). Hope to see many folks there, happy to talk about most anything. I'm really looking forward to the good energy that comes from being around so many awesome open source folks!

Of course that means I may well not be online as much as normal (when traveling, in talks, etc), so Please plan accordingly if you need my help with something.

Laptop

So, I got this lenovo slim7x snapdragon X laptop quite a long time ago, and finally I decided I should see if I can use it day to day, and if so, use it for the flock trip, so I don't have to bring my frame.work laptop.

So, I hacked up a aarch64 rawhide live with a dtb for it and was able to do a encrypted install and then upgrade the kernel. I did have to downgrade linux-firmware for the ath12k firmware bug, but thats fine.

So far it's looking tenable (I am typing this blog post on it now). I did have to add another kernel patch to get bluetooth working, but it seems to be fine with the patch. The OLED screen on this thing is wonderfull. Battery life seems ok, although it's hard to tell without a 'real life' test.

Known things not working: camera (there's patches, but it's really early so I will wait for them), sound (there's also patches, but it has the same issue the mac laptops had with there being no safeguards so you can easily destroy your speakers if you adjust too loud).

Amusing things: no discord flatpak available (the one on flathub is x86_64 only), but the web version works fine. (Although amusingly it tells you to install the app (which doesn't exist).

Also, no chrome, but there is chromium, which should be fine for sites that firefox doesn't work with.

I'll see if I can get through the weekend and upcoming week and decide what laptop I will take traveling.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114564927864167658

Data Center Move

We have pretty much gotten all the new servers setup firmware wise. We have applied all the updates that happened since they were shipped, configured things as best we could for now. A few notable configuration changes we made:

• Enabled lldp on the machines that support it. This allows networking folks to see information about which nics are on which ports, etc. Just a bunch more handy info for us and them.

• Disabled 'hot spare' on power supply configuration. Wouldn't we want a 'hot spare'? well, no as it turns out if you enable that it means that all the servers only use the first power supply, keeping the second one idle. This means that in a rack, ALL the servers pull power from one side, which makes things very imbalanced. Instead disabling this has the server use both supplies and balance, and in the event of a failure, it just switches to the one thats still working. So, you want to be able to run everything from one side, but you definitely don't want to do so all the time.

I installed a few servers manually (see last weeks benchmarking entry), and this week I got local network setup as it should be on one: 2 25G nics bonded with 802.3ad, and a bridge on top for guests. Should be super zippy for anything local, and has the great advantage that networking folks can upgrade/reboot switches without us noticing any outages.

I also did a bunch of work on dns configuration. In order to make things easier on both us and networking folks, I asked them to just setup the new datacenter nets with a translation of existing datacenter configuration. That means we have the same number of vlans for the same purposes. Machines will be at the same last octet in both places. So for example our iad bastion server is internally at 10.3.163.31 in IAD2, and will be at 10.16.163.31 in RDU3. This also means we have a great starting point for network acls and such.

We are now somewhat in a holding pattern, waiting on external network for the servers themselves. Since we have gotten behind where we were hoping to be at this point, we very likely will be moving the actual datacenter switcharoo week out. Should know more next week if we have networking setup by then or not.

As soon as network is available, I will be bootstrapping up things in the new datacenter. Thats starting with a bastion host (to allow our existing ansible control host in our current datacenter to provision things there in the new one), then a dhcp/tftp server, then dns, then an ipa replica, then the rest of the servers, etc. After that is far enough along, we will be installing openshift clusters, getting our new signing infra working, and openqa machines and start migrating things that aren't heavily tieed to our current datacenter.

Things are gonna be busy the next month or so.

Bot blocking

A while back, we added some apache rules to block some bots that were providing a user agent, but were ignoring robots.txt, or were trying to crawl things we didn't want them to crawl or made no sense to be indexed. Last week I was looking at some AI scrapers (which don't pass a user agent saying they are a bot at all) and noticed that our block for 'normal' bots wasn't working. It turns out we had the right expression, but it only does a string match if you put the expression in "s. :(

So, I fixed that and I think it's helped reduce load over a bunch of things that shouldn't have been getting crawled in the first place.

The AI bots are still around, but mostly mitigated via various blocking of networks or specific things they decide they really really want. They are like a dog with a bone on some projects/areas... I am pretty sure they are re-crawling things they already crawled, they also seem particularly interested in forks or mirrors of things they have already crawled (even when those forks/mirrors have 0 other changes from the upstream). Here's hoping the market for these goes bust and they all go out of business.

F40 EOL and upgrades

Fedora 40 went end of life on tuesday of this last week. It's served long and well. Fond farewell to it.

We had a very few Fedora 40 instances left. The wiki was using F40. We upgraded staging and got all the issues sorted out and should be moving production to f42 next week. Bodhi was using f40 for some things (and f41 for others). There was a new upstream release with some minor rolled up changes. I upgraded staging yesterday and today, and will be rolling production very soon.

comments? additions? reactions?

As always, comment on mastodon: https://scrye.com/blogs/nirik//posts/2025/05/17/second-week-of-may-2025-fedora-infra-bits/

Datacenter Move

Spent a fair bit of time this week configuring and looking at the new servers we have in our new datacenter. We only have management access to them, but I still (somewhat painfully) installed a few with RHEL9 to do some testing and benchmarking.

One question I was asked a while back was around our use of linux software raid over hardware raid. Historically, there were a few reasons we choose mdadm raid over hardware raid:

• It's possble/easy to move disks to a different machine in the event of a controller failure and recover data. Or replace a failed controller with a new one and have things transparently work. With hardware raid you need to have the same exact controller and same firmware version.

• Reporting/tools are all open source for mdadm. You can tell when a drive fails, you can easily re-add one, reshape, etc. With hardware raid you are using some binary only vendor tool, all of them different.

• In the distant past being able to offload to a seperate cpu was nice, but anymore servers have a vastly faster/better cpu, so software raid should actually perform better than hardware raid (barring different settings).

So, I installed one with mdadm raid another with a hardware raid and did some fio benchmarking. The software raid won overall. Hardware was actually somewhat faster on writes, but the software raid murdered it in reads. Turns out the cache settings defaults here were write-through for software and write-back for hardware, so the difference in writes seemed explainable to that.

We will hopfully finish configuring firmware on all the machines early next week, then the next milestone should be network on them so we can start bootstrapping up the services there.

Builders with >32bit inodes again

We had a few builders hit the 'larger than 32 bit inode' problem again. Basically btrfs starts allocating inode numbers when installed and builders go through a lot of them by making and deleting and making a bunch of files during builds. When that hits > 4GB, i686 builds start to fail because they cannot get a inode. I reinstalled those builders and hopefully we will be ok for a while more again. I really am looking forward to i686 builds completely going away.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114484593787412504

Datacenter Move

Actual progress to report this week! Managed to get access to the mgmt on all our new hardware in the new datacenter. Most everything is configured right in dhcp config now (aarch64 and power10's need still some tweaking there).

This next week will be updating firmware, tweaking firmware config, setting up access, etc on all those interfaces. I want to try and do some testing on various raid configs for storage and standardize the firmware configs. We are going to need to learn how to configure the lpars on the power10 machines next week as well.

Then, the following week hopefully we will have at least some normal network for those hosts and can start doing installs on them.

The week after that I hope to start moving some 'early' things: possibly openqa and coreos and some of our more isolated openshift applications. That will continue the week after that, then it's time for flock, some more moving and then finally the big 'switcharoo' week on the 16th.

Also some work on moving some of our soon to be older power9 hardware into a place where it can be added to copr for more/better/faster copr builders.

OpenShift cluster upgrades

Our openshift clusters (prod and stg) were upgraded from 4.17 to 4.18. OpenShift upgrades are really pretty nice. There was not much in the way of issues (although a staging compute node got stuck on boot and had to be power cycled).

One interesting thing with this upgrade was that support for cgroups v1 was listed as going away in 4.19. It's not been the default in a while, but our clusters were installed so long ago that they were still using it as a default.

I like that the upgrade is basically to edit one map and change a 1 to a 2 and then openshift reboots nodes and it's done. Very slick. I've still not done the prod cluster, but likely next week.

Proxy upgrades

There's been some instablity with our proxies in particular in EU and APAC. We are going to be over the coming weeks rolling out newer/bigger/faster instances which should hopefully reduce or eliminate problems folks have sometimes been seeing.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114445144640282791

Datacenter Move

Still don't have access to our new hardware, but I'm hoping early next week I will. I did find out a good deal more about networking there and setup our dhcp server already with all the mac addresses and ip's for the management interfaces. As soon as that comes up they should just get the right addresses and be ready to work on.

Next week then would be spent setting firmware the way we want it, testing a few install paramaters to make sure how we want to install the hosts, then move on to installing all the machines.

Then on to bootstrapping things up (we need a dns server, a tftp server, etc) and then installing openshift clusters and virthosts.

So, we are still on track for the move in June as long as the management access comes in next week as planned.

nftables in production

We rolled out our switch from iptables to nftables in production on thursday. Big shout out to James Antill for all the scripting work and getting things so they could migrate without downtime.

The switch did take a bit longer than we would have liked, and there were a few small hiccups, but overall it went pretty well.

There are still some few openqa worker machines we are going to migrate next week, but otherwise we are all switched.

Staging koji synced

To allow for some testing, I did a sync of our production koji data over to the staging instance. This takes a long long time because it loads the prod db in, vacuums it, then modifies it for staging.

There was a bit of breakage at the end (I needed to change some sequences) but otherwise it went fine and now staging has all the same tags/etc as production does.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114405223201008788

Fedora 42 out! Get it now!

Fedora 42 was released on tuesday. The "early" milestone even. There was a last minute bug found ( see: https://discussion.fedoraproject.org/t/merely-booting-fedora-42-live-media-adds-a-fedora-entry-to-the-uefi-boot-menu/148774 ) Basically booting almost any Fedora 42 live media on a UEFI install results in it adding the "Fedora" Live media to your boot manager list. This is just booting, not installing or doing anything else. On the face of it this is really not good. We don't want live media to affect installs without installing or choosing to do so. However, in this case the added entry is pretty harmless. It will result in the live media booting again after install if you leave it attached, and if not, almost all UEFI firmware will just see that the live media isn't attached and ignore that entry.

In the end we decided not to try and stop the release at the last minute for this, and I think it was the right call. It's not great, but it's not all that harmfull either.

Datacenter Move news

Networking got delayed and the new date we hope to be able to start setting things up is this coming friday. Sure hope that pans out as our window to setup everything for the move is shrinking.

There was some more planning ongoing, but will be great to actually start digging in and getting things all setup.

AI Scraper news

The scrapers seem to have moved on from pagure.io. It's been basically unloaded for the last week or more. Sadly, they seem to have discovered koji now. I had to block a few endpoints on the web frontend to stop them. Unfortunately there was a short outage of the hub caused by this, and there were 2 builds that were corrupted as a result. Pretty aggravating.

Nftables

Worked with James to roll out our iptables->nftables switch to production. All the builders are now using nftables. Hopefully we will roll out more next week.

Thats it for this week, catch everyone next week!

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114371722035321670

Datacenter Move

I wrote up a community blog post draft with updates for the community. Hopefully it will be up early next week and I will also send a devel-announce list post and discussion thread.

We had a bit of a snafu around network cards. The new aarch64 boxes we got we missed getting 10G nics, so we are working to aquire those soon. The plan in the new datacenter is to have everything on dual 10G nics connected to different switches, so networking folks can update them without causing us any outages.

Some new power10 machines have arrived. I'm hopeful we might be able to switch to them as part of the move. We will know more about them once we are able to get in and start configuring them.

Next week I am hoping to get out of band management access to our new hardware in the new datacenter. This should allow us to start configuing firmware and storage and possibly do initial installs to start bootstraping things up.

Exciting times. I Hope we have enough time to get everything lined up before the june switcharoo date. :)

Fun with databases

We have been having a few applications crash/loop and others behave somewhat sluggishly of late. I finally took a good look at our main postgres database server (hereafter called db01). It's always been somewhat busy, as it has a number of things using it, but once I looked at i/o: yikes. (htop's i/o tab or iotop are very handy for this sort of thing). It showed that a mailman process was using vast amounts of i/o and basically causing the machine to be at 100% all the time. A while back I set db01 to log slow queries. So, looking at that log showed that what it was doing was searching the mailman.bounceevents table for all entries were 'processed' was 'f'. That table is 50GB. It has bounce events back 5 or 6 years at least. Searching around I found a 7 year old bug filed by my co-worker Aurélien: https://gitlab.com/mailman/mailman/-/issues/343

That was fixed! bounces are processed. However, nothing ever cleans up this table at least currently. So, I proposed we just truncate the table. However, others made a good case that the less invasive change (we are in freeze after all) would just be to add a index.

So, I did some testing in staging and then made the change in production. The queries went from: ~300 seconds to pretty much 0. i/o was now still high but around the 20-30% range most of the time.

It's amazing what indexes will do.

Fedora 42 go for next week!

Amazingly, we made a first rc for fedora 42 and... it was GO! I think we have done this once before in all of fedora history, but it's sure pretty rare. So, look for the new release out tuesday.

I am a bit sad in that there's a bug/issue around the Xfce spin and initial setup not working. Xfce isn't a blocking deliverable, so we just have to work around it. https://bugzilla.redhat.com/show_bug.cgi?id=2358688 I am not sure whats going on with it, but you can probibly avoid it by making sure to create a user/setup root in the installer.

I upgraded my machines here at home and... nothing at all broke. I didn't even have anything to look at.

comments? additions? reactions?

As always, comment on mastodon: posts/2025/04/12/early-mid-april-infra-bits-2025.rst