Now that we have had bodhi2 in production use for a while, I thought I would talk some about it’s backend and how it manages updates pushes.
First, for a bit of background, lets mention how bodhi1 did things:
- release engineers would gather a list of pending updates and look over them.
- release engineers would sign those updates
- release engineers would tell bodhi1 to push them. Typically in freezes (alpha, beta, final) the branched updates-testing ones were pushed, then all the stable releases (both testing and stable). Outside of freezes, all branches were pushed at the same time (both testing and stable).
- bodhi1 would start going through the list one by one and mashing the packages into repos.
- At the end when all branches were done it would go through them one by one and gather updateinfo and other data it needed to inject.
- It would then create rpm-ostree/atomic updates for each branch that has one one by one.
- Then it would wait for all of them to be synced to the master mirrors.
- Then it would go through them one by one and update bugs and note things in the bodhi updates.
As you might imagine this could take a very long time. It wasn’t unusual to see this be 12-18 hours or more for the full set.
Bodhi2 instead of being single threaded is very much mult-threaded:
- release engineers gather a list of pending updates and look them over.
- release engineers sign updates
- release engineers tell bodhi2 to push them.
- bodhi2 looks at all the pending updates: Are any of them security? It will do all those branches that have security updates first.
- Threads are fired off for each branch that is to be done (first all security holding ones, then all the rest).
- In each of those threads, subthreads are fired off to: mash updates (using createrepo_c instead of createrepo that bodhi1 used), gather updateinfo, create digests, ie, anything that can be done and wait to be collected together at the end is done.
- When one branch is done, atomic trees are updated, it’s synced out and all work is completed on it (updating bugs, etc).
So, where bodhi1 took 12-18 (or sometimes more!) hours to push all Fedora branches, bodhi2 is taking around 6 hours to push them all. The slowest of the branches is of course fedora-21 stable updates (It takes about 4.5 hours to complete), which makes sense as it’s been out for a very long time now and has a very large pile of updates in it.
Since things are multi threaded in bodhi2, failures and issues are better as well. If something broke somewhere in bodhi1 pushes, everything ground to a halt until it was fixed and resumed. In bodhi2 if there’s an issue, only that particular branch will fail and can be resumed anytime. Everything else completes.
Moving forward there’s even more improvements we hope to roll out: Right now bodhi handles the mashing of updates on it’s own instance, and can be somewhat limited by it’s I/O bandwith, so we would like to move that out into koji and have koji do the mashes and create the updates trees. That way it can be farmed out to a bunch of seperate builders. There’s some createrepo_c changes hopefully we can get working to add multithreading to delta rpm creation, which may speed things up too. Finally we may tweak the threading to do all the security and non security branches at once (memory and I/O permitting).