Attention epel6 and epel7 ppc64 users

by on 2019/05/23 at 12:55 pm
Posted In: fedora, linux

If you are a epel6 or epel7 user on the ppc64 platform, I have some sad news for you. If you aren’t feel free to read on for a tale of eol architectures.

ppc64 (the big endian version of power) was shipped with RHEL6 and RHEL7 and Fedora until Fedora 28. It’s been replaced by the ppc64le (little endian) version in Fedora and RHEL8.

The fedora build system (koji) runs Fedora on all it’s builders. We moved to doing this because we often needed new features in rpm or lower level packages like that. For example when rpm switched to xz compression, we needed a rpm package that was new enough to do/understand that on all the builders, so we either had to backport this support to the RHEL version or just switch to Fedora. We found it more supportable to just switch to Fedora.

However, since Fedora stopped supporting ppc64 in Fedora 29, all our current ppc64 builders are Fedora 28 (the last release with support for ppc64). Fedora 28 is about to go end of life and we had to decide what to do about epel6/epel7 since they still support ppc64.

epel6 ppc64 users may be sticking with that platform because their hardware doesnt support ppc64le. epel7 users could move to ppc64le, but they might be keeping ppc64 instances to remain compatible or not wanting to change. ppc64 users are a very low number (around 100 checkins to our mirroring system per day, next to 1.5million for x86_64). Additionally, I would expect few of those ppc64 installs are new deployments or even directly on the internet.

We could make RHEL7 builders just for epel6/7 ppc64. However, our ansible playbooks for builder deployment have counted on them being Fedora for a long time now, so it would be a pretty big retooling effort. Additionally, those builders would sit around mostly idle, taking up resources we could use for other ones.

So, in the end, I think we are going to look at retiring ppc64 in epel6/7 next week when Fedora 28 goes end of life. The old packages would still be available in koji, but the repos would disappear. If there’s some case we haven’t thought of here, do bring it up to the epel steering comittee: https://pagure.io/epel/issue/57 or on the epel-devel list.

Comments Off on Attention epel6 and epel7 ppc64 users

A preliminary review of /e/

by on 2019/03/22 at 7:10 pm
Posted In: fedora, linux

I’ve been running LineageOS on my phone for a while now (and cyanogenmod before that) and been reasonably happy overall. Still even LineageOS is pretty intertwined with the google ecosystem and worries me, especially given that google is first and foremost an ad company.

I happened to run accross mention of /e/ somewhere and since LineageOS did a jump from being based on ASOP15 to ASOP16 which required a new install anyhow, I decided to check it out.

As you may have gathered from the above, /e/ is a phone OS and platform, forked off from LineageOS14.1. It’s located at https://e.foundation based in france (a non profit) headed by Gaël Duval, who Linux folks may know from his Mandrake/Mandriva days. The foundation has a lot of noble goals, starting with “/e/’s first mission is to provide everyone knowledge and good practices around personal data and privacy.” They also have a slogan “Your data is your data!”

I downloaded and installed a 0.5 version here. Since I already had my phone unlocked and TWRP recovery setup, I just backed up my existing LineageOS install (to my laptop), wiped the phone and installed /e/. The install was painless and since (of course) there’s no google connections wanted, I didn’t even have to download a gapps bundle. The install worked just fine and I was off and exploring:

The good:

  • Most everything worked fine. Basically if it worked in LineageOS 14.1, it works here (phone, wifi, bluetooth, etc)
  • Many of the apps I use with my phone seem fine: freeotp, signal, twidere, tiny tiny rss reader, revolution irc are all the same apps I am used to using and are install-able from f-droid just fine.
  • There is of course no google maps anymore, but this was a great chance to try out OsmAnd, which has come a very long way. It’s completely usable except for one thing: The voice navigation uses TTS voices and it sounds like a bad copy of Stephen Hawking is talking to you. Otherwise it’s great!
  • My normal ebook reader app is available: fbreader, but I decided to look around as it’s getting a bit long in the tooth. I settled so far on KOReader, which was orig a kobo app, but works pretty nicely on this OS as well.
  • For podcasts I had been using dogcatcher, but now I am trying out AntennaPod.
  • The security level of the image I got was March 2019, so they are keeping up with at least the “android” security updates now.

The meh:

  • The fdroid app isn’t pre-installed, but it’s easy to install it. They plan to have their own store for apps that will just show additional information over the play store, etc.
  • There is ‘fennec’ in f-droid. You can’t seem to install firefox as all download links lead to the play store.
  • I had been using google photos to store backups/easy web access versions of pictures and movies I took, but of course now I just need to look into alternatives. Perhaps syncthing.

The bad:

  • A few apps I was using are of course non free and not available in f-droid: tello, vizio smartcast, various horrible IOT smart things apps, my credit unions silly app, etc. tello works fine if you can find a apk not on the play store. vizio smartcast seems to fail asking for location services (which should work, but oh well).
  • Untappd doesn’t seem to have a .apk easily available, so I guess twitter will be spared my been drinking adventures. 🙂
  • Some infosec folks looked closely and there was still some traffic to google: https://infosec-handbook.eu/blog/e-foundation-first-look/#e-foundation but they had a very reasonable reply I thought (not trying to reject or ignore anything): https://hackernoon.com/leaving-apple-google-how-is-e-actually-google-free-1ba24e29efb9

The install is all setup with MicroG. “A free-as-in-freedom re-implementation of Google’s proprietary Android user space apps and libraries.” It does a pretty good job pretending to be google for apps that need some google bits.

In addition to the OS, /e/ folks have a server side setup as well. I didn’t play with it too much as I am waiting for their promised containerized versions of the server side so I can run them myself. These provide replacements for google drive, notes, address book, mail, etc.

The name /e/ is a bit strange to try and pronounce, or search for. Turns out they had another name at first, but someone else was using it and took exception. There is some mention that they are going to rename things before the magic 1.0 comes.

All in all I think I am going to keep using /e/ for now. Keeping up on security and the ability to make me look at open source alternatives to the various apps I use seems pretty nice to me. I do hope it catches on and more folks start to use it.

Comments Off on A preliminary review of /e/

CPE meetings and devconf2019

by on 2019/02/04 at 11:57 am
Posted In: fedora, linux

I recently went to Brno, CZ for CPE (Community Platform Engineering) meetings and then devconfcz 2019 and thought I would share my take on both of them.

Travel to and from Brno is always a long one for me. I’m currently based in Oregon, US, so my journey is:

  • Drive to portland airport (PDX)
  • Flight from Portland to Amsterdam (AMS) (a 9-11hour flight)
  • Flight to Prague (PRG) (usually a 1-2 hour flight)
  • Bus to train station (30-40min)
  • Train to Brno (2-3 hours)

And then the same in reverse on the way back, with all the associated timezone issues. 🙂 I am very happy about the direct amsterdam flight, so I don’t have to change planes in london or frankfurt or something.

A short word about the CPE team. We are a team in Red Hat that works on Fedora and CentOS (formerly Fedora Engineering). We have some application developer folks who write and fix our custom applications (bodhi, pagure, release-monitoring, etc) as well as a number of Operations folks who keep the Fedora and CentOS infrastructures running smoothly.

We spent the week of Jan 21st meeting up and discussing plans for the year as well as ways we could be more responsive to the community and better handle our (large) workflow.

  • 2019-01-21: Brian Stinson went over the CentOS CI setup we have and we identified projects that we care about that didn’t have any CI and worked on fixing them up. We got a bunch more projects with (all be it simple) tests running.
  • 2019-01-22: We talked about ways to be more efficent with out workload. We determined to try and have a ops person paired with a dev person on deployments to avoid delays. We talked about doing more pair work. We talked about changing our status reports. Then we wrote up all the planned work we know of in the coming year, prioritized it, gave it owners to write up. We should have this info up on the wiki before too long (or somewhere).
  • 2019-01-23: We talked about rawhide gating and changed our plan to be simpler than it had been. We went over the fedmsg to fedora-messaging changeover. We moved some apps to openshift and fedora-messaging. More to come.
  • 2019-01-24: We had some meetings with some internal Red Hat teams on how we could help each other by doing things first in Fedora and how best to do that. We worked some more on priorities and upcoming tasks.

Then it was time for devconfcz. Always a great conference. Tons of talks to see and tons of people to talk to in the hallway track. A few of the talks I really wanted to go to I got to too late and they were already full, but I did see some interesting ones.

There was a lot of discussion about EPEL8 in the hallway track, but luckily we had a number of the people who knew how modularity works there to quash plans that wouldn’t work and to propose ones that would. At this point the plan is to make a EPEL8beta that is just the “ursine” packages and test that out while working on modular EPEL8. For modular EPEL8 we are going to look at something that takes the modular RHEL repos and splits them out into one repo per module. Then we can hopefully get mbs to use these external modules when it needs them as build requirements and we can also decide what modules we want in the ‘ursine’ buildroot. This is all handwavy and subject to change, but it is a plan. 🙂

Smooge and I gave our EPEL talk and I think it went pretty well. There were a lot of folks there at any rate and we used up the time no problem.

As always after a chance to meet up with my co-workers and see tons of interesting talks I’m really looking forward to the next few months. Lots and lots of work to do, but we will get it done!

└ Tags: 
1 Comment

Rawhide notes from the trail, mid december 2018

by on 2018/12/17 at 11:59 am
Posted In: fedora, linux, rawhide

Just a few notes from the trail this week:

  • zchunk repodata should be in place as of todays compose. Feedback on how much repodata you need to download now or any other issues with it would be good to get fixed up before we branch f30 off of rawhide. Ideally people will be download a LOT less repodata now.
  • AdamW setup the openqa reports that go to the devel and test list to also note what tests would be gating and if we were gating what would we have hit. This is grep prep work for the gating landing, so we can fix those tests/issues and start with a GO.
  • Not directly rawhide, but related: bugzilla was updated finally to bugzilla5. Overall things went fine, but there’s a few issues: bodhi is having trouble updating bugs sometimes, and things that use libreport (anaconda and abrt) are no longer just sending 1 email on new bugs, but an email for every attachment. These issues are being worked on.

I hope everyone has a relaxing holiday season.

Comments Off on Rawhide notes from the trail, mid december 2018

OpenShift in Fedora Infrastructure

by on 2018/12/09 at 5:50 pm
Posted In: fedora, linux

I thought I would write up a quick post to fill folks in on what our OpenShift setup is in Fedora Infrastructure, what we are doing with it now, and what we hope to do with it in coming years.

For those that are not aware, OpenShift is the Red Hat version of OKD, which is a open source, container application platform. That is, it’s a way to deploy and manage application containers. Each of your applications can use a known framework to define how they are built, managed and run. It’s pretty awesome. If you need to move your applicaiton somewhere else, you can just export and import it into another OpenShift/OKD and away you go. Recent versions also include monitoring and logging frameworks too. There is also a very rich permissions model, so you can basically give as much control to a particular application as you like. This means the developer(s) of the applications can also deploy/debug/manage their application without needing any ops folks around for that.

Right now in Fedora Infrastructure we are running two separate OpenShift instances:One in our staging env and one in production. You may note that OpenShift changes the idea of needing a staging env, since you can run a separate staging instance or just test one container of a new version before using it for all of production, however, our main use for the staging OpenShift is not staging applications so much as having another OpenShift cluster to upgrade and test changes in.

In our production instance we have a number of applications already: bodhi (the web part of it, there is still a seperate backend for updates pushes), fpdc, greenwave, release-monitoring, the silverblue web site, and waiverdb. There’s more in staging that are working on getting ready for production.

One of the goals we had from the sysadmin side of things was the ability to be easily able to completely re-install the cluster and all applications, so we have made some different setup choices that others might. First, in order to deploy the cluster we have in our ansible playbooks one that creates and provisions a ‘control’ host. On this control host we pull a exact version of the openshift-ansible git repository and run ansible from the control host with a inventory we generate and the specific openshift-ansible repo. This allows us to provision a cluster exactly the same everytime. One the cluster is setup, we have setup our ansible repo to have the needed definitions for every application and it can provision them all with a few playbook runs. Of course this means no containers with persistent storage in them (or very few using NFS), but so far thats fine. Most of our applications store their state in a database and we just run that outside of the cluster.

Short term moving forward we plan to move as many applications as we can/make sense to OpenShift, as it’s a much easier way to manage and deploy things. We also intend to set things up so our prod cluster can run staging containers (and talk to all the right things, etc). Also we hope to run a development instance in our new private cloud. This instance we hope to open more widely to contributors for their developing applications or proof of concepts. We would like to get some persistent storage setup for our clusters, but it’s unclear right now what that would be.

Longer term we hope to run other clusters in other locations so we can move applications around as makes sense and also for disaster recovery.

I’d have to say that dealing with OpenShift has been very nice, there have been issues, but they are all logical and easy to track down, and the way things are setup just makes sense. Looking forward to 4.0!

Comments Off on OpenShift in Fedora Infrastructure