CPE meetings and devconf2019

by on 2019/02/04 at 11:57 am
Posted In: fedora, linux

I recently went to Brno, CZ for CPE (Community Platform Engineering) meetings and then devconfcz 2019 and thought I would share my take on both of them.

Travel to and from Brno is always a long one for me. I’m currently based in Oregon, US, so my journey is:

  • Drive to portland airport (PDX)
  • Flight from Portland to Amsterdam (AMS) (a 9-11hour flight)
  • Flight to Prague (PRG) (usually a 1-2 hour flight)
  • Bus to train station (30-40min)
  • Train to Brno (2-3 hours)

And then the same in reverse on the way back, with all the associated timezone issues. 🙂 I am very happy about the direct amsterdam flight, so I don’t have to change planes in london or frankfurt or something.

A short word about the CPE team. We are a team in Red Hat that works on Fedora and CentOS (formerly Fedora Engineering). We have some application developer folks who write and fix our custom applications (bodhi, pagure, release-monitoring, etc) as well as a number of Operations folks who keep the Fedora and CentOS infrastructures running smoothly.

We spent the week of Jan 21st meeting up and discussing plans for the year as well as ways we could be more responsive to the community and better handle our (large) workflow.

  • 2019-01-21: Brian Stinson went over the CentOS CI setup we have and we identified projects that we care about that didn’t have any CI and worked on fixing them up. We got a bunch more projects with (all be it simple) tests running.
  • 2019-01-22: We talked about ways to be more efficent with out workload. We determined to try and have a ops person paired with a dev person on deployments to avoid delays. We talked about doing more pair work. We talked about changing our status reports. Then we wrote up all the planned work we know of in the coming year, prioritized it, gave it owners to write up. We should have this info up on the wiki before too long (or somewhere).
  • 2019-01-23: We talked about rawhide gating and changed our plan to be simpler than it had been. We went over the fedmsg to fedora-messaging changeover. We moved some apps to openshift and fedora-messaging. More to come.
  • 2019-01-24: We had some meetings with some internal Red Hat teams on how we could help each other by doing things first in Fedora and how best to do that. We worked some more on priorities and upcoming tasks.

Then it was time for devconfcz. Always a great conference. Tons of talks to see and tons of people to talk to in the hallway track. A few of the talks I really wanted to go to I got to too late and they were already full, but I did see some interesting ones.

There was a lot of discussion about EPEL8 in the hallway track, but luckily we had a number of the people who knew how modularity works there to quash plans that wouldn’t work and to propose ones that would. At this point the plan is to make a EPEL8beta that is just the “ursine” packages and test that out while working on modular EPEL8. For modular EPEL8 we are going to look at something that takes the modular RHEL repos and splits them out into one repo per module. Then we can hopefully get mbs to use these external modules when it needs them as build requirements and we can also decide what modules we want in the ‘ursine’ buildroot. This is all handwavy and subject to change, but it is a plan. 🙂

Smooge and I gave our EPEL talk and I think it went pretty well. There were a lot of folks there at any rate and we used up the time no problem.

As always after a chance to meet up with my co-workers and see tons of interesting talks I’m really looking forward to the next few months. Lots and lots of work to do, but we will get it done!

└ Tags: 
1 Comment

Rawhide notes from the trail, mid december 2018

by on 2018/12/17 at 11:59 am
Posted In: fedora, linux, rawhide

Just a few notes from the trail this week:

  • zchunk repodata should be in place as of todays compose. Feedback on how much repodata you need to download now or any other issues with it would be good to get fixed up before we branch f30 off of rawhide. Ideally people will be download a LOT less repodata now.
  • AdamW setup the openqa reports that go to the devel and test list to also note what tests would be gating and if we were gating what would we have hit. This is grep prep work for the gating landing, so we can fix those tests/issues and start with a GO.
  • Not directly rawhide, but related: bugzilla was updated finally to bugzilla5. Overall things went fine, but there’s a few issues: bodhi is having trouble updating bugs sometimes, and things that use libreport (anaconda and abrt) are no longer just sending 1 email on new bugs, but an email for every attachment. These issues are being worked on.

I hope everyone has a relaxing holiday season.

Comments Off on Rawhide notes from the trail, mid december 2018

OpenShift in Fedora Infrastructure

by on 2018/12/09 at 5:50 pm
Posted In: fedora, linux

I thought I would write up a quick post to fill folks in on what our OpenShift setup is in Fedora Infrastructure, what we are doing with it now, and what we hope to do with it in coming years.

For those that are not aware, OpenShift is the Red Hat version of OKD, which is a open source, container application platform. That is, it’s a way to deploy and manage application containers. Each of your applications can use a known framework to define how they are built, managed and run. It’s pretty awesome. If you need to move your applicaiton somewhere else, you can just export and import it into another OpenShift/OKD and away you go. Recent versions also include monitoring and logging frameworks too. There is also a very rich permissions model, so you can basically give as much control to a particular application as you like. This means the developer(s) of the applications can also deploy/debug/manage their application without needing any ops folks around for that.

Right now in Fedora Infrastructure we are running two separate OpenShift instances:One in our staging env and one in production. You may note that OpenShift changes the idea of needing a staging env, since you can run a separate staging instance or just test one container of a new version before using it for all of production, however, our main use for the staging OpenShift is not staging applications so much as having another OpenShift cluster to upgrade and test changes in.

In our production instance we have a number of applications already: bodhi (the web part of it, there is still a seperate backend for updates pushes), fpdc, greenwave, release-monitoring, the silverblue web site, and waiverdb. There’s more in staging that are working on getting ready for production.

One of the goals we had from the sysadmin side of things was the ability to be easily able to completely re-install the cluster and all applications, so we have made some different setup choices that others might. First, in order to deploy the cluster we have in our ansible playbooks one that creates and provisions a ‘control’ host. On this control host we pull a exact version of the openshift-ansible git repository and run ansible from the control host with a inventory we generate and the specific openshift-ansible repo. This allows us to provision a cluster exactly the same everytime. One the cluster is setup, we have setup our ansible repo to have the needed definitions for every application and it can provision them all with a few playbook runs. Of course this means no containers with persistent storage in them (or very few using NFS), but so far thats fine. Most of our applications store their state in a database and we just run that outside of the cluster.

Short term moving forward we plan to move as many applications as we can/make sense to OpenShift, as it’s a much easier way to manage and deploy things. We also intend to set things up so our prod cluster can run staging containers (and talk to all the right things, etc). Also we hope to run a development instance in our new private cloud. This instance we hope to open more widely to contributors for their developing applications or proof of concepts. We would like to get some persistent storage setup for our clusters, but it’s unclear right now what that would be.

Longer term we hope to run other clusters in other locations so we can move applications around as makes sense and also for disaster recovery.

I’d have to say that dealing with OpenShift has been very nice, there have been issues, but they are all logical and easy to track down, and the way things are setup just makes sense. Looking forward to 4.0!

Comments Off on OpenShift in Fedora Infrastructure

Rawhide notes from the trail, the late November issue

by on 2018/12/01 at 9:23 pm
Posted In: fedora, flock, rawhide

Greetings everyone! Lets take a look at notable things from the rawhide trail in the last week:

  • We had 2 DOOMED composes and 5 FINISHED_INCOMPLETE
  • The DOOMED ones failed because of broken deps in gnome-contacts (making the Workstation live media fail to compose). This was actually fixed very quickly (just needed a rebuild), but for some reason got stuck in the signing queue, so it still wasn’t fixed when we thought it was.
  • I just updated pungi on rawhide-composer to 4.31.1, which should make composes a little faster. See this post by pungi developer Lsedlar
  • Some folks have been having problems with the dbus->dbus-broker change. Do make sure that dbus-broker is enabled to start on boot if you run into strange boot issues.
  • There’s a good deal of high level talk about pushing the f31 release out next year to allow for more tooling and possibly longer lifecycles on the devel list. Do read and contribute if you have thoughts on the matter. I’d definitely like to see us improve things, but so far most of the discussion has been very high level and handywavy. Can’t wait to get into details.

Otherwise its been a pretty typical week, between 2018-11-23 and 2018-11-30:

Added packages: 19
Removed packages: 12
Modified packages: 1293

Comments Off on Rawhide notes from the trail, the late November issue

Rawhide notes from the trail: The thanksgiving edition

by on 2018/11/23 at 4:17 pm
Posted In: fedora, linux, rawhide

Sheesh, it’s been a while since I blogged anything again. I’m going to try and do at least 1-2 posts a week moving forward.

So, whats been happening in the rawhide world? Lets look back on this last month:

  • 2018-11-23 (today): dbus-broker switched to being the default dbus (A f29 change that didn’t make it and is planned for f30 now). Unfortunately, this broke anaconda as it launches it’s own bus and didn’t have the right package installed to do that anymore. This meant that everything that well… installed did not work. Adamw quickly submitted a PR and new fixed build and another rawhide is running now.
  • 2018-11-22: dnf maintainers were updating the dnf stack of things and happened to be in the middle of that when rawhide compose started, this meant that everything had broken buildroots and failed to compose. As a reminder, rawhide composes start at 05:15 UTC every day. Please avoid breaking things right before that.
  • 2018-11-21: finished (incomplete)
  • 2018-11-20: finished (incomplete)
  • 2018-11-19: finished (incomplete)
  • 2018-11-18: finished (incomplete)
  • 2018-11-17: finished (incomplete)
  • 2018-11-16 and 2018-11-15: The 14th compose got stuck, so the 15th never happened.
  • 2018-11-14: A change in an ostree file meant that pungi didn’t properly emit a fedmsg so the ostree could be signed. So, it got stuck there waiting. A fix was attempted, but (my mistake) not applied correctly (I didn’t update pungi on the composer after I built a patched one).
  • 2018-11-13: failed due to libreoffice deps being broken and the workstation live thus not building.
  • 2018-11-12: finished (incomplete)
  • 2018-11-11: finished (incomplete)
  • 2018-11:10: finished (incomplete)
  • 2018-11-09: finished (incomplete)
  • 2018-11-08: failed due to pungi-gather segfaults. This has proven very difficult to track down. In case it was memory related (and because we had more memory handy) I increased the rawhide composers memory by a ton. It seems to avoid the issue now at least since we haven’t seen this since this day.
  • 2018-11-07: failed due to pungi-gather segfaults.
  • 2018-11-06: failed due to pungi-gather segfaults.
  • 2018-11-05: finished (incomplete)
  • 2018-11-04: failed due to broken kde deps
  • 2018-11-03: failed due to two issues: 1) aarch64 had a issue with networking that caused the aarch64 cloud image to fail and 2) there was a weird issue with python3 and anaconda.
  • 2018-11-02: failed (see 11-03)
  • 2018-11:01: failed (see 11-03)

Most of the more anoying failures would have been blocked if we had gating in place (no broken deps would have crept in and if we tried a base image with anaconda/python/etc we would have found the breakage there eariler). I sure hope we can get that in place someday. 🙂

Comments Off on Rawhide notes from the trail: The thanksgiving edition