Flock2017: day 0 (and -1)

by on 2017/08/30 at 9:38 am
Posted In: fedora, flock, linux

It’s almost time for flock 2017. This year it’s in Cape Cod. Also this year I am heading there not from Denver, but from Salem, OR.

My best option for flights this year was a red-eye from Portland to Boston, then the bus out to Hyannis. It was a long, but uneventfull journey: 10:45pm out of Portland, 7am arrival at Boston, and 10am arrival at the Hyannis bus station.

I then started the walk (about a mile) to the hotel, but turns out I ran into some Fedorans I know, and they had a car, so I had a nice ride to the hotel.

The hotel is pretty large, but it’s pretty weird to get to/from our rooms as you have to go up and down stairs and around. It’s definitely past it’s glory days, but still seems reasonable to me.

After making the hotel, some hacking in the lobby and visiting and then lunch and dinner at some local main street places (burger and beer and then italian for dinner). After the long day, I was asleep before 10pm. 🙂

Tomorrow: Day 1!

Comments Off on Flock2017: day 0 (and -1)

Public Service Announcement: Fedora 26 and delta rpms

by on 2017/08/26 at 11:44 am
Posted In: fedora, linux

Just a quick public service announcement:

Delta rpms (The much smaller files that contain just the changes from one package version to another) are currently not working for Fedora 26 (all other releases should be working fine).

They actually were not fully working before, but the problem wasn’t detected fully until a few weeks after the release. The issue is that we now push out a bunch of alternative arch builds in the same updates as before and we place those in another location on the master mirrors. This means that mash and bodhi (The things that make the updates repos/updates) need to know where to look for the older package rpms in order to make delta rpms against the new ones, but currently we just pass it one location for everything.

We hope to fix this soon, but if you don’t see delta rpms in F26 (yet) this is why.

EDITED: As of 2016-09-01 we have F26 delta rpms fixed. Everything should be back to normal. Thanks for your patience.

Comments Off on Public Service Announcement: Fedora 26 and delta rpms

Rawhide notes from the trail, the long road edition (2017-08-24)

by on 2017/08/25 at 11:38 am
Posted In: fedora, linux, rawhide

Hey look, it’s been exactly 30 days since my last rawhide post. What a ride it’s been this last month:

The mass rebuild was delayed a bit due to tooling issues, then releng availablity, but finally happened around the beginning of the month. Then, sadly, another one was called for due to a binutils issue on ppc64. Luckily this was just all the archfull packages (the noarch ones were unaffected and could be reused from the first rebuild).

There have been a bunch of issues plaguing composes, including, but not limited to:

  • rdma-core replacing a number of other packages and needing lorax to adjust to it pulling perl into the base images.
  • rdma-core not building for armv7 and lorax needing to not try and install it on that platform.
  • Storage backend issues. We are now thinking it’s some interaction between Linux kernel and NFSv4.1. But the slowness has made composes and new build repos very slow sometimes.
  • New rpm that switched to sha256 headers breaking signing because old rpm didn’t correectly read the new headers (in an update this has been fixed now).
  • New rpm needing a bunch of things that linked to it rebuilt.
  • After the mass rebuild, all f28 needed signing with the new f28 key. (took a few days, but done).

Due to all that we have had many fewer actual finished composes than we normally had in the past (now that pungi will fail a compose if any required part fails). I’m not sure if thats making things better or worse, but hopefully they will get better. It does mean when we do get a compose there’s a better chance of it being usable, but also when we don’t theres no new repos for people to use day to day.

And tons of new things landing:

  • Bunch of changes to debuginfo. Now there are split debuginfo per subpackage and sources are all in a debugsource package.
  • Kerberos KCM is setup by default.
  • Tons of new versions of things: rpm, glibc, gcc, kernel, etc.

F27 branched off rawhide to go it’s way to release, and f24 went end of life (We thank it for it’s long and valient service).

Next week is flock (Fedora’s big yearly conference). I look forward to seeing my Fedora family and working on tons and tons of things. (This years theme is more ‘do’ than ‘talk’). Hope to see many of you there!

Comments Off on Rawhide notes from the trail, the long road edition (2017-08-24)

rawhide notes from the trail, the 2017-07-24 edition

by on 2017/07/24 at 4:10 pm
Posted In: fedora, linux, rawhide

Greetings! Once again it’s been a long spell since one of these posts, so lets jump right in and rope that calf.

Rawhide has been marching along to the next branching point, when Fedora 27 will branch off for it’s release. There’s a mass rebuild that should be happening very very soon now, there was a bit of delay while tools were all sorted out. Look for a very very big rawhide update soon once that mass rebuild finishes.

All the alternative arches are now in the same koji. We finally got s390x added in and going. Initally it was only 5 builders with 4GB ram, so things got stopped up from time to time, as all it would take was 5 long builds to show up and everything else waited for them. A few weeks ago we increased things to 15 builders and 8GB of ram each so hopefully that will keep up with builds as we go. We will see what the mass rebuild looks like with all the alternative arches included this time. I suspect it’s going to take longer than before, but not sure how much longer.

Thanks to Patrick you may also notice that uploading sources to our lookaside cache is no longer uploading them twice. Instead it’s using the kerberos cache to authenticate on the first go so it just needs to upload once.

We have missed some composes in the last few months (once for quite a while), and this is due to the compose process now failing the compose if all the release blocking deliverables are not there. On one hand thats nice because it means we always have those if the compose finishes, on the other it’s not so nice because if it fails we need to go fix whatever breakage is there.

Look for a new note soon, and ride safe out there!

Comments Off on rawhide notes from the trail, the 2017-07-24 edition

CI and infrastructure hackfest 2017

by on 2017/05/15 at 5:40 pm
Posted In: fedora, linux

I was hoping to write a bit of a summary for each day of the CI and Infrastructure hackfest last week, but there just wasn’t enough time to sit down and write up blog posts. Each day we started bright and early gathering in the hotel lobby at 8am, got to the Red Hat tower, got some breakfast and got to work until 5 or 6 and then we went and got dinner and back to the hotel to do it all over the next day. We definitely had some excellent discussions and got a lot of work done.

First a few thank yous: Red Hat provided us the use of the fittingly named “Fedora” room on the 9th floor to work in. It was a perfect size for our group. I think once we had to scare up an extra chair, but usually we fit just fine. OSAS (Red Hats Open Source And Standards group) funded travel and lodging for folks. Paul Frields (stickster) handled logistics and tried to keep us all scheduled, on track and in general cat herded in the right direction.

We had 3 BIG goals for this hackfest and a bunch of little ones. I think we did great on all counts. First on larger goals:

  • Monday we got a detailed dump of information about our authentication systems, past, present and future. Both from a perspective of sysadmins wanting to manage and fix things and application developers wanting their app to authenticate right. The high level overview is that we have FAS2 currently as the source of all our authentication knowledge. It syncs (one way) to freeipa (a pair of replicated masters). This sync can sometimes fail, so we now know more about what to do in those cases from the sysadmin side. Then we have ipsilon that manages openid, openid-connect and (used to) handle persona. We got some detailed workflows for each of these cases. Moving forward we want to get apps using OpenID-Connect. Down the road we talked about replacing fas with a very thin layer community access api app. Not sure thats going to happen, but might be an interesting way to go.
  • We wanted to harness the CentOS CI pipeline for testing a subset of optin packages from Fedora. I wasn’t directly working in this area, but I know by the end of the week we had CentOS-CI sending fedmsgs in our staging env and had setup a ci instance near it with resultsdb and taskotron to manage tests. I think there’s some more hooking up of things to go, but overall it should be pretty cool.
  • Finally we wanted to look into and setup our own OpenShift instance. We had some very nice discussions with Red Hat OpenShift folks who manage and deploy things bigger than we can imagine. They gave us some very helpful information. Then we talked out initial policy questions and so forth. By friday we had a OpenShift cluster up and running in our staging env. We still need to get some official certs, sort out the ansible managing applications setup end, and figure out what we want to do for persistent storage, but otherwise we made a vast amount of progress. You can find out questions and answers at https://fedoraproject.org/wiki/Infrastructure/OpenShift

On smaller items:

  • We met with the Red Hat Storage folks who manage our backend storage. There’s some nice improvements coming along later this year if we can adjust things to take advantage of it. Mostly that means splitting up our gigantic koji volume into a ‘archive’ and ‘active’ volumes and splitting our ftp volume into a ‘active’ and ‘archive’ volumes. Then we can hopefully move the active volumes over to flash storage, which should be a great deal faster. All still up in the air, but promising.
  • We met with the folks who manage our main datacenter, and it’s looking possible that we will be moving all our stuff to a new area a short distance away. If this comes to pass, it would be later in the summer and there will likely be some downtime. On the plus side we would get a chance to organize machines better, get new racks and better power and all around be in a better place moving forward.
  • We had a nice discussion around making bodhi faster. I’ve been over that many a time, but we did come up with a few new ideas, like generating drpms at update submission time instead of at mashing time (but that would need createrepo_c changes). Or perhaps triggering pushes only on critpath or security updates and other leaf node packages would just wait.
  • There were a number of discussions around factory 2.0, modularity, branching, koji namespaces, etc.
  • We had several discussions on postgres BDR (Bi directional replication). I’ve moved a number of our apps to it in staging and hoped to roll it out in production, but there were some concerns. In the end we decided to look at deploying it and then deciding on a app by app basis which ones were ready to move over. In the end we hope to have everything using it, but some apps need time to make sure they follow all of BDR’s rules and do the right thing. Additionally some apps may want to use BDR, but in sync mode to avoid possible bad data on a node crash. koji upstream needs to support things before we can move koji, but some of our smaller apps may be able to move soon. We also decided we wanted a script that could detect problems by looking at an applications schema. Pingou already went and had this done by the end of the week.
  • Tim Flink (Fedora QA) and I had a nice discussion about scaling the existing QA setup. We agreed that it might make good sense to see if we could migrate this into our cloud, provided we can get it up on a modern version we could actually support. That is likely to happen in the next few months as we have some new hardware for it and are retiring some of the old hardware, so it’s a good time to force an new setup.
  • We went to the ansible offices and had beer and pizza and watched a baseball game. 🙂

All in all a super productive week. Look for lots of the above to come up in meetings, tickets, mailing lists and so forth as we share plans we made, and make sure everyone is on board with them or if we need to adjust anything.

Comments Off on CI and infrastructure hackfest 2017