debugging koji build failures
From time to time builds fail in koji (The Fedora build system), and it’s good to know how to figure out where to look for the reason. Koji has a central hub that manages jobs and a bunch of builders that actually do the builds. When someone initiates a build you are talking to the hub and either uploading a src.rpm (for a scratch build) or telling it to use a particular git hash/repo for an official Fedora build. For official builds, koji will first generate a job to build the src.rpm from git and the packages lookaside cache with the source. This job will run on some builder thats ready and has capacity for it. Once the src.rpm is generated (or if you are providing it for a scratch build), the hub will generate build tasks for all the arches that are set in the target tag you are building to. Each one of those will go out to a builder of the right arch type that is enabled and has capacity, etc. If any of these fail, the entire build fails.
Once you have gotten notice of a failure, the first thing to do is go to the link for the build in the koji web interface and then click on the task link to show the build tasks. There you will see one (or more) links in red that failed. Don’t click on any of them yet though, instead click on the “show result” link. That will tell you exactly which task failure failed the build. When one failure happens koji cancels the other builds, but sometimes they fail at nearly the same time and you are looking at the wrong place for the real failure. Sometimes all the links are green and the reason for the failure is only visible in the “show result” area (for example, if you have a archfull package with noarch subpackages and they are different on different arches, thats a failure, or if you don’t have permissions to build the thing you are building, thats a tagging failure).
Next click on that task that you saw in the “show result” link (if you haven’t already seen the error as above), and again click on the “show result” link. You should see something like:
"BuildError: error building package (arch i686), mock exited with status 1; see build.log for more information"
Or it could refer you to the root.log. Go to that log and look toward the end for the error. If the error is in the root.log it means there was a problem setting up the mock chroot. That could be that there is a package you BuildRequire that in not installable, or some package in the build root is completely broken.
If the issue is in the build.log its the actual output from the rpmbuild, so this would be where to look at your general compiler issues, etc.
If you ask someone to help you out and look at the error(s) you are getting, it’s important to give them the top level link to the task instead of just a link to the root.log or build.log. If you only have those links it’s impossible to back up and look at the higher levels, but if you have the top level task you can look at the entire thing.
Aside from normal rpm package builds, koji also builds a bunch of other things like livemedia and other images. Finding issues there is very similar. Always check the “show results” area for which logs to look at for the actual failure.