In case you didn’t hear it all the other places you would normally hear it: Fedora 28 is out. Get it! https://getfedora.org/
I updated my servers here at home last friday, as usual 0 problems at all with the upgrade. Upgrades really are painless these days.
Last week I had the pleasure of attending the 2018 Infrastructure Hackfest in Fredricksberg, VA. It was a very productive week and very nice to meet up face to face with a lot of folks I work with mostly over IRC and email.
Travel went pretty well for me (direct flights, 4-5 hours each way) and the hotel worked out nicely. I liked that the hotel had a big table (with power!) in the corner of the lobby for us to use in evenings for more hacking. Our day workspace was a classroom at a nearby grad college. Aside from some firewall issues monday morning (They were blocking everything but 80/443) it worked pretty well too. Lots of tables we could move around, and whiteboards/projector.
Monday we went over current package maintainer workflows and looked for improvements we could land. Pagure 4.0 (out very soon now) should let us fix the silly ‘get a pagure token for this and put it in a config file’ workflow. We also got bodhi to have a ‘waive’ button to waive test results from the web ui, along with showing what exact tests were missing or failing (so we can get rid of a hacky shell script on the wiki for this purpose). After workflows, we started on package maintainer docs rework. We wanted to move them out of the wiki and clean them up and reorg them. First we tried to identify all wiki pages that are related here and make sure we got all the popular ones, then we brainstormed some personas (new to packaging, upstream maintainer wanting to just package their project, someone who doesn’t have time to follow process and just wants the steps to do an update, someone who wants to know how it all works, etc), then we wrote up a bunch of things these people would want to do and tried to organize new docs for it. It’s not really yet in a state to get wide feedback, but hopefully soon it will be.
Tuesday we talked about reassigning some apps that were maintained by someone who moved to other work, then retired a few apps that have been limping along that we no longer want to spend time on: darkserver (which actually has been broken the last year or so), and summershum (which got checksums of source file uploads), which we never quite figured a use for. Next up was rawhide gating. We went over the proposal send to the devel list and got more detailed about what exact things we would need to do for it. It turns out that just adding some things to the update object we could handle the case of side tags and the work won’t be as big as we thought it would.
Wed was setting up a AWX instance. This is the upstream version of ansible tower. We hoped that using it would give us some good advantages. Unfortunately we hit a number of issues. If we tried to install with a external db the install failed, so we had to move back to a db in container next to the rest, then we got pretty far on configuring authentication and hit a bug preventing SAML from working, then finally we hit an issue where we couldn’t get our private repo mounted in the right container to allow us to have access to our secrets. There will be more discussion on this in the coming weeks and we can decide if the effort is worth it or if we should just keep on going the way we had been. As a side note we looked more at the loopabull setup we have. It has only one demo job in it, but it seems ready to add more into as soon as we would like. Look for us to use that more soon too.
Thursday was more work on rawhide gating and some discussion around openshift. Patrick took the projector and he and Randy got bodhi’s web frontend completely moved over to our openshift (but prod is still not pointing at these new instances until we are ready). We also did some more work on release-monitoring.org (anitya) and got it much further along. There were a number of tweaks and improvements to our openshift playbooks too.
Friday we just met at the hotel as everyone was traveling away, so we just worked on finishing some things up.
A few things that got done also over multiple days or between other issues:
All in all, a very pproductive week of hacking on things. Many thanks to Paul Frields (stickster) for organizing everything and the Council/OSAS/Fedora Engineering for sponsoring things.
It’s continued to be a bit of a rocky trail of late for rawhide.
The last 6 composes have failed due to:
I am looking forward to next week when we hope to get things setup for some gating in rawhide. I know it couldn’t handle all these issues, but it’s a start and we can add things as we know how to detect them in advance.
As many of you know, we had a long string of difficulties with getting rawhide to compose at all recently. This was mostly a combination of poor timing for things landing, people landing things at the last minute for Fedora 28, and various bugs.
Last week, we hit something that seemed to be pretty special: An actual finished compose. To explain, pungi (the tool that does the composing in Fedora land) has several states that a compose can be in: It can be “STARTED” which means that it started and is composing, or died in such a way it couldn’t properly note that to the compose. It can be “DOOMED” which means that one (or more) required deliverable items failed to compose. In the past this might still have meant part of the compose was shipped out anyhow, but not these days. That prevents the compose from going out at all, and releng folks have to look and make sure the issue that caused the problem is fixed. A compose can be “FINISHED_INCOMPLETE”: This means all the required deliverables are there, but optional/non blocking ones may or may not be. This is the status most of our composes have gotten in the past. Finally, a compose can be ‘FINISHED’ which means all parts of it successfully composed. We had our first “FINISHED” rawhide compose last week. So, break out the harmonica and start dancing right? well, not so fast. There was a bug in pungi where it would have some failed optional deliverables, but still report FINISHED. So, the FINISHED we had wasn’t really a true FINISHED (some ppc images were still failing). Finally late last week we got in a workaround for those last few images, and I am happy to report today that the FINISHED we got was a true, full, real, everything composed FINISHED.
I think some of the credit for this goes to the simple compose tracker that Dusty Mabe setup. This allowed us to track down and fix things that in the past we would just have written off that we don’t have time to fix. Thanks for the issue tracking Dusty! There was of course lots of hard work from many others as well to get to this point: Thanks Mohan, Patrick, Adam, Dennis, Sinny and others I am sure I am forgetting.
Congrats on a FINISHED compose, and lets see how long we can keep it that way.
Greetings everyone! Lots of news from the rawhide trail this time:
Astute observers will note that from 2018-02-04 until 2018-02-17 there were no (success-full) rawhide composes. This was due to a long line of issues, problems, new packages, changes and just bad luck, including, but not limited to:
In order to help out all the folks working on fixing things we are trying out a ticket system to track failed composes. See the proof of concept version at: https://pagure.io/dusty/failed-composes/issues I think this already is helping coordinate and communicate breakage. Hopefully we will get it polished up and made “official” soon.
Of course some of the above compose issues were due to the mass rebuild, which happened a few weeks ago. Things seemed to take longer than in the past, I think due to src.fedoraproject.org not committing as quickly as it had in the past. Hopefully we can fix that.
Also, today we are at another milestone: Fedora 28 is branching off rawhide. Rawhide will continue on it’s trail to Fedora 29. Do make sure you know what path you want to follow and that your machine(s) are taking that path. If you do nothing you should end up on Fedora 28 branched and follow it to release.
Gnome was briefly broken in rawhide, but quickly fixed by a new gnome-session build. Thanks gnome maintainers for quickly fixing things up!
The kernel has moved on to 4.16 pre releases in rawhide. Early (before rc1) releases here had no networking with the ath10k card I have in my laptop (it would just never see any wireless networks). After rc1 however things are mostly back to normal. The ath10k is a bit flaky, but mostly works.
Finally, on a personal note, one of my dogs has had major surgery and a long slow road to recovery. I’ve started a gofundme to help defray his massive medical bills. If you would like to donate, pass along or just look at some cute pictures of him, you can go here: https://www.gofundme.com/medical-bills-for-nash
Loading ...