As many of you may have noticed, we have had some issues the last few days in Fedora Infrastructure, in particular with metalinks (the files used by dnf/yum to pull down repository data).

First, I’d like to let you know that the mirrorlists/metalink issues should all be fixed now, so please do ‘dnf clean all’ or ‘yum clean all’ and retry your operations and let us know (via a Fedora Infrastructure ticket or #fedora-admin on irc) if you still see any problems. If you changed your repo files while this issue was happening, I would strongly urge you to change it back to the metalink. It has a number of great advantages over a specific mirror or mirrorlist.

We don’t yet have a detailed root cause write up, but should have something next week. The problems seem to have been a number of issues, starting with a network problem that seems to have caused some issue with a virtualhost server that had our main vpn hub and postgresql server on it, then on to a possible mirrormanager bug where it removed the wrong alternative repomd data, breaking metalinks. Most of the metalinks were fixed last night, but we wanted to make sure and have everything back to normal before returning to normal status.

We did create some valuable tools out of this unfortunate outage, including scripting to update specific repomd.xml’s and not wait for an entire crawl of the tree (which can take many many hours), scripting to check all metalinks against the master mirror and yell when they are not completely in sync, and more.

I’d like to thank everyone for being patient while we work on this issue and especially thank all the hard working folks on the Fedora Engineering team that worked on this around the clock! I’m proud to work with such great folks.