Flock 2019

by on 2019/08/19 at 12:29 pm
Posted In: fedora, flock, linux

Flock time is upon is! This time in lovely Budapest. As always when flock is in europe, it’s a long flight for me, but otherwise travel was uneventfull: Drive 2 hours to PDX, then PDX to AMS, then a short layover for coffee and stoupwaffles and then AMS to BUD, and finally a taxi ride to the hotel.

The hotel is quite lovely. It’s right next to the danube river and has a nice view. The AC is working nicely too (it’s quite hot outside here right now). After getting into the hotel yesterday and a quick dinner at a very nice place down the road, I managed to sleep for 10+ hours.

A nice complementary breakfast, then flock. I went to:

  • The FPLs yearly talk on the state of Fedora. This went over a lot of decisions and changes the Fedora Council made early this year around mission and how to better try and acheive and articulate our goals. Then some nice charts and graphs, and finally a bunch of things for us to ponder on how to solve.
  • Next up was a coffee break. I Love that we have these scheduled, it’s a great chance to visit with fellow fedorians.
  • The next big talk in the main room was Cate Huston from wordpress talking about how we all fail at projects and some things to try and fail less. It was a nice talk. Lots to think about, and definitely some things that resonated.
  • Final talk in the big room for the day was from the Facebook desktop support team about them rolling Fedora out as a supported desktop for their users. They went over their setup, some problems/issues they hope to work with Fedora to solve and took questions. Very interesting.
  • Nice lunch in the hotel resturant, then back to flock…
  • I went then to the Improving the Packaging Experience with automation. Some great ideas there, we just need people to try and drive them and spend some cycles working on them.
  • Then on to the Fedora Workstation status update and progress. Lots of great stuff here, most of which I knew, but it was great seeing all the progress in one place. Keep up the great work!
  • The next few timeslots I got pulled into the “hallway” track. ie, discussions with friends and co-workers and community members (and sometimes all 3 in one person!).
  • Next I went to Ben Cotton’s session on changing the changes process. I think the changes he wants to do will help out and make things run smoother. He had a nice overview of changes and there was a good bit of discussion with the audience.
  • I sadly missed the “slideshow kareoke” session as I stopped to talk to people.
  • The nights event was board games and relaxing, so we gathered up everyone from the CPE team that was there and had a very nice group dinner at a local place.
  • Then, back to the hotel and time to sleep…

The next day started out with a “Fedora, Red Hat, and IBM” talk from Denise Dumas. I’ve heard most of the content before, but she put a nice Fedora spin on things.

  • Next up I went to a panel talk lead by Brendan Conoboy about how RHEL8 was formed and where and why it diverged from Fedora. Not surprisingly, we want to try and merge things back as closely as we can to make RHEL9 smoother.
  • Lubomír Sedlář‘s talk on making composes faster was definitely something I had to attend. In addition to the ideas he had, the audience suggested a few more and I hope everyone went away with a good understanding of how composes are made.
  • Leigh Griffin had a nice talk about my team (CPE) trying to become more agile. I really like his approach to find what works for us and not just apppoint a scrum master and make everyone go through some ceremonies. What we end up with will depend on what we want and how we work, there’s no one-size-fits-all.
  • Laura Abbot gave a talk about the kernel tree of the future. There’s a lot of will to move things to a better workflow than they are today. I hope they can get it to a nice place with some hard work.
  • Finally up at the end of the day was my 25min communishift talk. The room was packed and there were tons of questions (which I hope I answered). Communishift is out community openshift cluster for Proof of concepts, development, tinkering or the like from the community. We are only responsible for the platform, users are responsible for everything above that. It seemed to get a lot of good feedback.
  • The evening event was a dinner boat ride on the river. The dinner and the boat and the company was great. There was a bit too much walking to get there in the heat, but we all survived.

Saturday started off with a showcase from the outreachy and google summer of code projects. It was awesome to see some of the projects they had worked on and the progress they had made. great stuff!

  • Most of the rest of saturday I spent talking to people and discussing ideas or problems. Got a lot of requests to add people to communishift, which I did later that night.

Sunday started off with a CPE hackfest. I worked it like a barcamp, so we tossed out ideas and topics of things we wanted to go over and voted on them. The top two were: “going over backlog of projects” and “improving the packager experience”. We split the room into two and started working away.

  • We got a ton of requirements / needs for a new monitoring / metrics setup. There was a lot of good discussion from people not on our team, but interacting with us on what they might want/need.
  • There was some discussion about how to try and move forward on our next generation account system. Stay tuned here there’s several good ideas being explored.
  • I finally got to talk to the copr folks and come up with some plans to move copr forward from being on our old cloud. Should be some good movement in this area soon I hope.
  • We got some progress on the CPE docs. Fixed some things and got them generating as we push changes to them.
  • After lunch we had the “meet your FESCo” panel. There were lots of questions and I hope a good discussion. Everyone got some good mic time and it went pretty well.
  • Finally the wrap up session, everyone who lead a session came up and gave a short overview. It was great to hear this for the sessions I couldn’t attend. Finally there was a SUPER great video made by tatica of a bunch of people at flock. We really are a diverse bunch.
  • Sunday night I had a final dinner with the team and then up at 4am to head home. Left the hotel at 4am, got home about 3pm the same day (and a ton of time zone changes in the way).

All in all another great flock in the books. This one seemed to have more energy / excited folks than I have seen in a while, and as always it was good to see old friends. Until next year!

Comments Off on Flock 2019

epel8-playground

by on 2019/07/30 at 11:52 am
Posted In: fedora, linux

We have been working away at getting epel8 ready (short status: we have builds and are building fedpkg and bodhi and all the other tools maintainers need to deal with packages and hope to have some composes next week), and I would like to introduce a new thing we are trying with epel8: The epel8-playground.

epel8-playground is another branch for all epel8 packages. By default when a package is setup for epel8 both branches are made, and when maintainers do builds in the epel8 branch, fedpkg will build for _both_ epel8 and epel8-playground. epel8 will use the bodhi updates system with an updates-testing and stable repo. epel8-playground will compose every night and use only one repo.

The idea here is that maintainers can (if they choose) modify the package.cfg file in the epel8 branch to only build for epel8, and then use the epel8-playground branch (and it’s sperate repos/builds):

  • To test out some new version of the package that might not be stable yet.
  • To test out some new packaging of the package
  • To test a major version change of the package that they want to land at the next epel8 minor release.
  • To build a package that will never be stable enough for epel8, but still could be useful to some.

At minor RHEL releases (ie, 8.1, 8.2) people can pull in big changes from playground to the main epel8 packages. Since people will be upgrading and paying more attention than usual anyhow at those points, it’s a great chance to do that change, but also you want to make sure it’s panned out, so you can test before hand in playground.

We hope that this feature will be useful to some folks. Do let us know on the epel-devel list or in our next EPEL sig meeting!

└ Tags: , , ,
Comments Off on epel8-playground

Changing how we work

by on 2019/07/21 at 2:59 pm
Posted In: fedora, linux

As those of you who read the https://communityblog.fedoraproject.org/state-of-the-community-platform-engineering-team/ blog know, we are looking at changing workflows and organization around in the Community Platform Engineering team (of which, I am a member). So, I thought I would share a few thoughts from my perspective and hopefully enlighten the community more on why we are changing things and what that might look like.

First, let me preface my remarks with a disclaimer: I am speaking for myself, not our entire team or anyone else in it.

So what are the reasons we are looking for change? Well, there are a number of them, some of them inter-related, but:

  • I know I spend more time on my job than any ‘normal’ person would. Thats great, but we don’t want burn out or heroic efforts all the time. It’s just not sustainable. We want to get things done more efficiently, but also have time to relax and not have tons of stress.
  • We maintain/run too many things for the number of people we have. Some of our services don’t need much attention, but even so, we have added lots of things over the years and retired very few.
  • Humans suck at multitasking. There’s been study after study that show that for the vast majority of people, it is MUCH more efficent to do one task at a time, finish it and then move on. Our team gets constant interruptions, and we currently handle them poorly.
  • It’s unclear where big projects are in our backlog. When other teams approach us with big items to do it’s hard to show them when we might work on the thing they want us to, or whats ahead of it, or what priority things have.
  • We have a lot of ‘silos’. Just the way the team has worked, one person usually takes lead on each specific application or area and knows it quite well. This however means no one else does, no one else can help, they can never win the lottery, etc.
  • Things without a ‘driver’ sometimes just languish. If there is not someone (one of our team or even a requestor) pressing a work item forward, sometimes it just never gets done. Look at some of the old tickets in the fedora-infrastructure tracker. We totally want to do many of those, but they never get someone scheduling them and doing them.
  • There’s likely more…

So, what have we done lately to help with these issues? We have been looking a lot at other similar teams and how they became more efficient. We have been looking at various of the ‘agile’ processes, although I personally do not want to cargo cult anything, if there’s a ceremony some process calls for that makes no sense for us, we should not do it.

  • We setup an ‘oncall’ person (switched weekly). This person listens for pings on IRC, tickets or emails to anyone on the team and tries to intercept and triage them. This allows the rest of the team to focus on whatever they are working on (unless the oncall person deems this serious enough to bother them). Even if you stop and tell the person you don’t have time and are busy on something else, the amount of time to swap that out and back in already makes things much worse for you. We of course will still be happy to work with people on IRC, just schedule time in advance in the associated ticket.
  • ticket or it doesn’t exist. We still are somewhat bad about this, but the idea is that every work item should be a ticket. Why? So we can keep track of the things we do, so oncall can triage them and assign priority, so people can look at tickets when they have finished a task and not been interrupted in the middle of it. So we can hand off items that are still being worked on and coordinate. So we know who is doing what. And on and on.
  • We are moving our ‘big project’ items to be handled by teams that assemble for that project. This includes a gathering info phase, priority, who does what, estimated schedule, etc. This ensures that there’s no silo (multiple people working on it), that it has a driver so it gets done and so on. Setting expectations is key.
  • We are looking to retire, outsource or hand off to community members some of the things we ‘maintain’ today. There’s a few things that just make sense to drop because they aren’t used much, or we can just point at some better one. There’s also a group of things that we could run, but we could just outsource to another company that focuses on that application and have them do it. Finally there are things we really like and want to grow, but we just don’t have any time to work on them. If we hand them off to people who are passioniate about them, hopefully they will grow much better than if we were still the bottleneck.

Finally, where are we looking at getting to?

  • We will probibly be setting up a new tracker for work (which may not mean anything to our existing trackers, we may just sync from those to the new one). This is to allow us to get lots more metrics and have a better way of tracking all this stuff. This is all still handwavy, but we will of course take input on it as we go and adjust.
  • Have an ability to look and see what everyone is working on right at a point in time.
  • Much more ‘planning ahead’ and seeing all the big projects on the list.
  • Have an ability for stakeholders to see where their thing is and who is higher priority and be able to negotiate to move things around.
  • Be able to work on single tasks to completion, then grab the next one from the backlog.
  • Be able to work “normal” amounts of time… no heroics!

I hope everyone will be patient with us as we do these things, provide honest feedback to us so we can adjust and help us get to a point where everyone is happier.

└ Tags: , , , ,
Comments Off on Changing how we work

Attention epel6 and epel7 ppc64 users

by on 2019/05/23 at 12:55 pm
Posted In: fedora, linux

If you are a epel6 or epel7 user on the ppc64 platform, I have some sad news for you. If you aren’t feel free to read on for a tale of eol architectures.

ppc64 (the big endian version of power) was shipped with RHEL6 and RHEL7 and Fedora until Fedora 28. It’s been replaced by the ppc64le (little endian) version in Fedora and RHEL8.

The fedora build system (koji) runs Fedora on all it’s builders. We moved to doing this because we often needed new features in rpm or lower level packages like that. For example when rpm switched to xz compression, we needed a rpm package that was new enough to do/understand that on all the builders, so we either had to backport this support to the RHEL version or just switch to Fedora. We found it more supportable to just switch to Fedora.

However, since Fedora stopped supporting ppc64 in Fedora 29, all our current ppc64 builders are Fedora 28 (the last release with support for ppc64). Fedora 28 is about to go end of life and we had to decide what to do about epel6/epel7 since they still support ppc64.

epel6 ppc64 users may be sticking with that platform because their hardware doesnt support ppc64le. epel7 users could move to ppc64le, but they might be keeping ppc64 instances to remain compatible or not wanting to change. ppc64 users are a very low number (around 100 checkins to our mirroring system per day, next to 1.5million for x86_64). Additionally, I would expect few of those ppc64 installs are new deployments or even directly on the internet.

We could make RHEL7 builders just for epel6/7 ppc64. However, our ansible playbooks for builder deployment have counted on them being Fedora for a long time now, so it would be a pretty big retooling effort. Additionally, those builders would sit around mostly idle, taking up resources we could use for other ones.

So, in the end, I think we are going to look at retiring ppc64 in epel6/7 next week when Fedora 28 goes end of life. The old packages would still be available in koji, but the repos would disappear. If there’s some case we haven’t thought of here, do bring it up to the epel steering comittee: https://pagure.io/epel/issue/57 or on the epel-devel list.

Comments Off on Attention epel6 and epel7 ppc64 users

A preliminary review of /e/

by on 2019/03/22 at 7:10 pm
Posted In: fedora, linux

I’ve been running LineageOS on my phone for a while now (and cyanogenmod before that) and been reasonably happy overall. Still even LineageOS is pretty intertwined with the google ecosystem and worries me, especially given that google is first and foremost an ad company.

I happened to run accross mention of /e/ somewhere and since LineageOS did a jump from being based on ASOP15 to ASOP16 which required a new install anyhow, I decided to check it out.

As you may have gathered from the above, /e/ is a phone OS and platform, forked off from LineageOS14.1. It’s located at https://e.foundation based in france (a non profit) headed by Gaël Duval, who Linux folks may know from his Mandrake/Mandriva days. The foundation has a lot of noble goals, starting with “/e/’s first mission is to provide everyone knowledge and good practices around personal data and privacy.” They also have a slogan “Your data is your data!”

I downloaded and installed a 0.5 version here. Since I already had my phone unlocked and TWRP recovery setup, I just backed up my existing LineageOS install (to my laptop), wiped the phone and installed /e/. The install was painless and since (of course) there’s no google connections wanted, I didn’t even have to download a gapps bundle. The install worked just fine and I was off and exploring:

The good:

  • Most everything worked fine. Basically if it worked in LineageOS 14.1, it works here (phone, wifi, bluetooth, etc)
  • Many of the apps I use with my phone seem fine: freeotp, signal, twidere, tiny tiny rss reader, revolution irc are all the same apps I am used to using and are install-able from f-droid just fine.
  • There is of course no google maps anymore, but this was a great chance to try out OsmAnd, which has come a very long way. It’s completely usable except for one thing: The voice navigation uses TTS voices and it sounds like a bad copy of Stephen Hawking is talking to you. Otherwise it’s great!
  • My normal ebook reader app is available: fbreader, but I decided to look around as it’s getting a bit long in the tooth. I settled so far on KOReader, which was orig a kobo app, but works pretty nicely on this OS as well.
  • For podcasts I had been using dogcatcher, but now I am trying out AntennaPod.
  • The security level of the image I got was March 2019, so they are keeping up with at least the “android” security updates now.

The meh:

  • The fdroid app isn’t pre-installed, but it’s easy to install it. They plan to have their own store for apps that will just show additional information over the play store, etc.
  • There is ‘fennec’ in f-droid. You can’t seem to install firefox as all download links lead to the play store.
  • I had been using google photos to store backups/easy web access versions of pictures and movies I took, but of course now I just need to look into alternatives. Perhaps syncthing.

The bad:

  • A few apps I was using are of course non free and not available in f-droid: tello, vizio smartcast, various horrible IOT smart things apps, my credit unions silly app, etc. tello works fine if you can find a apk not on the play store. vizio smartcast seems to fail asking for location services (which should work, but oh well).
  • Untappd doesn’t seem to have a .apk easily available, so I guess twitter will be spared my been drinking adventures. 🙂
  • Some infosec folks looked closely and there was still some traffic to google: https://infosec-handbook.eu/blog/e-foundation-first-look/#e-foundation but they had a very reasonable reply I thought (not trying to reject or ignore anything): https://hackernoon.com/leaving-apple-google-how-is-e-actually-google-free-1ba24e29efb9

The install is all setup with MicroG. “A free-as-in-freedom re-implementation of Google’s proprietary Android user space apps and libraries.” It does a pretty good job pretending to be google for apps that need some google bits.

In addition to the OS, /e/ folks have a server side setup as well. I didn’t play with it too much as I am waiting for their promised containerized versions of the server side so I can run them myself. These provide replacements for google drive, notes, address book, mail, etc.

The name /e/ is a bit strange to try and pronounce, or search for. Turns out they had another name at first, but someone else was using it and took exception. There is some mention that they are going to rename things before the magic 1.0 comes.

All in all I think I am going to keep using /e/ for now. Keeping up on security and the ability to make me look at open source alternatives to the various apps I use seems pretty nice to me. I do hope it catches on and more folks start to use it.

Comments Off on A preliminary review of /e/