Data Center Move

We have pretty much gotten all the new servers setup firmware wise. We have applied all the updates that happened since they were shipped, configured things as best we could for now. A few notable configuration changes we made:

• Enabled lldp on the machines that support it. This allows networking folks to see information about which nics are on which ports, etc. Just a bunch more handy info for us and them.

• Disabled 'hot spare' on power supply configuration. Wouldn't we want a 'hot spare'? well, no as it turns out if you enable that it means that all the servers only use the first power supply, keeping the second one idle. This means that in a rack, ALL the servers pull power from one side, which makes things very imbalanced. Instead disabling this has the server use both supplies and balance, and in the event of a failure, it just switches to the one thats still working. So, you want to be able to run everything from one side, but you definitely don't want to do so all the time.

I installed a few servers manually (see last weeks benchmarking entry), and this week I got local network setup as it should be on one: 2 25G nics bonded with 802.3ad, and a bridge on top for guests. Should be super zippy for anything local, and has the great advantage that networking folks can upgrade/reboot switches without us noticing any outages.

I also did a bunch of work on dns configuration. In order to make things easier on both us and networking folks, I asked them to just setup the new datacenter nets with a translation of existing datacenter configuration. That means we have the same number of vlans for the same purposes. Machines will be at the same last octet in both places. So for example our iad bastion server is internally at 10.3.163.31 in IAD2, and will be at 10.16.163.31 in RDU3. This also means we have a great starting point for network acls and such.

We are now somewhat in a holding pattern, waiting on external network for the servers themselves. Since we have gotten behind where we were hoping to be at this point, we very likely will be moving the actual datacenter switcharoo week out. Should know more next week if we have networking setup by then or not.

As soon as network is available, I will be bootstrapping up things in the new datacenter. Thats starting with a bastion host (to allow our existing ansible control host in our current datacenter to provision things there in the new one), then a dhcp/tftp server, then dns, then an ipa replica, then the rest of the servers, etc. After that is far enough along, we will be installing openshift clusters, getting our new signing infra working, and openqa machines and start migrating things that aren't heavily tieed to our current datacenter.

Things are gonna be busy the next month or so.

Bot blocking

A while back, we added some apache rules to block some bots that were providing a user agent, but were ignoring robots.txt, or were trying to crawl things we didn't want them to crawl or made no sense to be indexed. Last week I was looking at some AI scrapers (which don't pass a user agent saying they are a bot at all) and noticed that our block for 'normal' bots wasn't working. It turns out we had the right expression, but it only does a string match if you put the expression in "s. :(

So, I fixed that and I think it's helped reduce load over a bunch of things that shouldn't have been getting crawled in the first place.

The AI bots are still around, but mostly mitigated via various blocking of networks or specific things they decide they really really want. They are like a dog with a bone on some projects/areas... I am pretty sure they are re-crawling things they already crawled, they also seem particularly interested in forks or mirrors of things they have already crawled (even when those forks/mirrors have 0 other changes from the upstream). Here's hoping the market for these goes bust and they all go out of business.

F40 EOL and upgrades

Fedora 40 went end of life on tuesday of this last week. It's served long and well. Fond farewell to it.

We had a very few Fedora 40 instances left. The wiki was using F40. We upgraded staging and got all the issues sorted out and should be moving production to f42 next week. Bodhi was using f40 for some things (and f41 for others). There was a new upstream release with some minor rolled up changes. I upgraded staging yesterday and today, and will be rolling production very soon.

comments? additions? reactions?

As always, comment on mastodon: https://scrye.com/blogs/nirik//posts/2025/05/17/second-week-of-may-2025-fedora-infra-bits/