Rawhide notes from the trail, a Milestone. No? Yes!

by on 2018/03/26 at 5:22 pm
Posted In: fedora, linux, rawhide

As many of you know, we had a long string of difficulties with getting rawhide to compose at all recently. This was mostly a combination of poor timing for things landing, people landing things at the last minute for Fedora 28, and various bugs.

Last week, we hit something that seemed to be pretty special: An actual finished compose. To explain, pungi (the tool that does the composing in Fedora land) has several states that a compose can be in: It can be “STARTED” which means that it started and is composing, or died in such a way it couldn’t properly note that to the compose. It can be “DOOMED” which means that one (or more) required deliverable items failed to compose. In the past this might still have meant part of the compose was shipped out anyhow, but not these days. That prevents the compose from going out at all, and releng folks have to look and make sure the issue that caused the problem is fixed. A compose can be “FINISHED_INCOMPLETE”: This means all the required deliverables are there, but optional/non blocking ones may or may not be. This is the status most of our composes have gotten in the past. Finally, a compose can be ‘FINISHED’ which means all parts of it successfully composed. We had our first “FINISHED” rawhide compose last week. So, break out the harmonica and start dancing right? well, not so fast. There was a bug in pungi where it would have some failed optional deliverables, but still report FINISHED. So, the FINISHED we had wasn’t really a true FINISHED (some ppc images were still failing). Finally late last week we got in a workaround for those last few images, and I am happy to report today that the FINISHED we got was a true, full, real, everything composed FINISHED.

I think some of the credit for this goes to the simple compose tracker that Dusty Mabe setup. This allowed us to track down and fix things that in the past we would just have written off that we don’t have time to fix. Thanks for the issue tracking Dusty! There was of course lots of hard work from many others as well to get to this point: Thanks Mohan, Patrick, Adam, Dennis, Sinny and others I am sure I am forgetting.

Congrats on a FINISHED compose, and lets see how long we can keep it that way.

Comments Off on Rawhide notes from the trail, a Milestone. No? Yes!

Rawhide notes from the trail, the cold hard winter trail…

by on 2018/02/20 at 11:23 am
Posted In: fedora, linux, rawhide

Greetings everyone! Lots of news from the rawhide trail this time:

Astute observers will note that from 2018-02-04 until 2018-02-17 there were no (success-full) rawhide composes. This was due to a long line of issues, problems, new packages, changes and just bad luck, including, but not limited to:

  • New pypkickstart that broke livemedia-composer
  • New pykickstart that broke appliance-creator
  • New libevent that bumped SO, requiring everything using it to be rebuilt but…
  • GCC8 issues with chromium (and also qt5-qtwebengine) that prevented them from being rebuit for the libevent bump.
  • Ongoing issue with armv7 composers getting stuck in mkfs.ext4 on composes, requiring manual intervention to continue the compose.
  • General broken deps after the mass rebuild that broke some (release blocking) live composes.
  • system-config-printer pulling in a PackageKit subpackge, which broke the Xfce live due to it excluding PackageKit*
  • rng-tools, which was broken in the past and untagged, but the mass rebuild rebuilt it again due to the maintainer not cleaning it up yet.
  • Probably other things I am forgetting.

In order to help out all the folks working on fixing things we are trying out a ticket system to track failed composes. See the proof of concept version at: https://pagure.io/dusty/failed-composes/issues I think this already is helping coordinate and communicate breakage. Hopefully we will get it polished up and made “official” soon.

Of course some of the above compose issues were due to the mass rebuild, which happened a few weeks ago. Things seemed to take longer than in the past, I think due to src.fedoraproject.org not committing as quickly as it had in the past. Hopefully we can fix that.

Also, today we are at another milestone: Fedora 28 is branching off rawhide. Rawhide will continue on it’s trail to Fedora 29. Do make sure you know what path you want to follow and that your machine(s) are taking that path. If you do nothing you should end up on Fedora 28 branched and follow it to release.

Gnome was briefly broken in rawhide, but quickly fixed by a new gnome-session build. Thanks gnome maintainers for quickly fixing things up!

The kernel has moved on to 4.16 pre releases in rawhide. Early (before rc1) releases here had no networking with the ath10k card I have in my laptop (it would just never see any wireless networks). After rc1 however things are mostly back to normal. The ath10k is a bit flaky, but mostly works.

Finally, on a personal note, one of my dogs has had major surgery and a long slow road to recovery. I’ve started a gofundme to help defray his massive medical bills. If you would like to donate, pass along or just look at some cute pictures of him, you can go here:  https://www.gofundme.com/medical-bills-for-nash

Comments Off on Rawhide notes from the trail, the cold hard winter trail…

Fedora Engineering meetings and devconf.cz

by on 2018/01/31 at 10:25 am
Posted In: fedora, linux, travel

Last week (and part of this), I traveled out to lovely Brno, CZ for some Fedora Engineering meetings and then devconf.cz. Overall it was a great trip!

On the way out (last sunday) I drove up to the portland arirport (about an hour), then plane from PDX to AMS (amsterdam) (about 10 hours), where I met up with Patrick and we both were on a flight from there to PRG (prague)(about 2hours). Then it was a train from Prague to Brno (about 2.5hours) and a deep sleep at the hotel.

Since many of the folks in Fedora Engineering were going to be coming out to devconf.cz, we decided to do some meeting up and discussing things to work on this coming year and to plan out a hackfest we are hoping to do in a few months. Look for more details on all of that, but in a nutshell:

  • We planned out things we could work on a a hackfest: https://fedoraproject.org/wiki/Infrastructure_Hackathon_2018 which looks to be in early april.
  • We talked over ownership of some apps that are needing love and/or having team members move to other things.
  • We talked about plans for account system and where we would like to be.
  • We talked about gating packages in rawhide. We came up with somewhat of a plan, but needs some work/discussion for sure, look for more details soon.
  • We talked about some apps we want to just drop or move to another service, etc.

Look for more news on all of those before we do anything about any of them.

Next was devconf.cz. I was told how big and busy it was, but seeing it all was different. I went to a number of talks, some were pretty good, others just things I already knew. The short talks (23min) were kind of difficult, because it’s really hard to give someone much information in just 23min. You do a intro, discuss something and then it’s over. I was going to give a talk about using Fedora Modular server 27 in production, but since that didn’t ever happen to be released I gave my talk slot over to Stephen Gallagher to talk about the new modularity approach. The hallway track was very nice at devconf.cz… lots of Fedora contributors around to meet again and talk with. Had a bunch of production discssions with too many people to count. It will be interesting to see how the devconf.us works out, it it’s as busy and vibrant.

Then all too soon it was time to pack back up and head home. I managed to get in tuesday afternoon, but promptly crashed for 14 hours. All in all a great trip!

Comments Off on Fedora Engineering meetings and devconf.cz

Rawhide notes from the trail, the holiday edition

by on 2017/12/25 at 2:44 pm
Posted In: fedora, linux, rawhide

Happy holidays everyone! A few notes for those riding the rawhide trail or just joining those of us who are.

  • NetworkManager has finally dropped libnm-glib (deprecated 3 years ago now). There’s proposed gnome-shell changes being worked on to switch to libnm, but they have not yet landed. This means if you connect to VPN’s via gnome-shell, things aren’t going to work. As a workaround for now use nmcli: mcli c up ‘your vpn name here’ –ask. More info at https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4YJIWJXDMND2VL7KGG2C6UNE7RJMHJEI/ and https://bugzilla.gnome.org/show_bug.cgi?id=789811
  • better SATA power management is being enabled. See https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/T2STBDPXKZ7DHC7GS6VLM3ESYI6RHVDM/ There is a small chance of disk corruption, so this is a great time to check your backups. If you have a laptop with a SATA SSD you should expect some nice power savings from the change.
  • Additionally bluetooth power saving mode has been enabled a while back. If you find your bluetooth no longer working you can boot with btusb.enable_autosuspend=n. (My bluetooth here on my yoga 920 needs to revert this change currently. Tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1514836 )
  • I reinstalled the armv7 compose builders a few days ago with Fedora 27 and since then they have done great for rawhide composes. With Fedora 26 they were sometimes hanging, causing rawhide composes to hang until manually rebooted. If they keep working well we can hopefully close https://bugzilla.redhat.com/show_bug.cgi?id=1504264 soon, but I would like to see it work for a few more days.

Here’s to a happy rawhide holiday for everyone!

Comments Off on Rawhide notes from the trail, the holiday edition

Rawhide notes from the trail, the early December issue

by on 2017/12/11 at 12:37 pm
Posted In: fedora, linux, rawhide

Once again, things have been real busy and I haven’t kept up with sending my rawhide notes, but I will try and do better!

Astute rawhide trail travelers would have noted that there was a distinct lack of updates to rawhide recently. There were composes up to December 4th, and then nothing until late yesterday (December 11th). Here’s a list of the reasons all those composes failed:

* As part of a big infrastructure move (Moving every server in our main datacenter to a new area) we also applied updates all around. The December 5th rawhide failed because the new koji package had a bug with python3 and ‘runroot’ plugin we use for koji. This has since been fixed upstream and a fixed version applied to infrastructure machines.

* Another few runs failed because we were moving the signing infrastructure and didn’t have everything back up and working. Recent changes to the rawhide compose require signing because it signs ostrees commits that are made as part of the compose (unlikely packages where they are signed before the compose).

* A firefox update came out that did not build on armv7. This was ok until a new hunspell update came out, a bunch of things built against it and the existing firefox had a broken dependency on the old version. Ideally we could have untagged the hunspell update and all the things that had already built against it, but that was a pile of packages to look for. In addition, the firefox update was a security one (of course). So, I filed a bug on it, excluded armv7 (just until the compile is fixed) and did a new build. This was only part of the rawhide fix however, as the Xfce armv7 image is release blocking and it no longer had a firefox. So, I dropped firefox from it for now (until the compile is fixed).

* While dealing with the firefox thing, I merged some more kickstart PR’s folks had submitted. Sadly, one of them had a syntax error in KDE (which is release blocking), causing another compose to fail.

So, we are back on track for now. Hopefully firefox will get fixed soon as we can revert hacks. I’ll keep watching things over the holidays.

Comments Off on Rawhide notes from the trail, the early December issue