EPEL8 packages

by on 2019/10/09 at 12:31 pm
Posted In: fedora, linux

With the opening up of EPEL8, there’s a lot of folks looking and seeing packages they formerly used in EPEL6/7 not being available and wondering why. The reason is simple: EPEL is not a fixed exact list of packages, it’s a framework that allows interested parties to build and provide the packages they are interested in providing to the community.

This means for a package to be in EPEL8, it requires a maintainer to step forward and explicitly ask “I’d like to maintain this in EPEL8” and then build, test and do all the other things needed to provide that package.

The reason for this is simple: We want a high quality, maintained collection of packages. Simply building things once and never again doesn’t allow for someone fixing bugs, updating the package or adjusting it for other changes. We need a active maintainer there willing and able to do the work.

So, if you do see some package missing that you would really like, how do you get it added to the collection? First, open a bug in bugzilla.redhat.com against the package. If it has a Fedora EPEL product version, use that, otherwise use Fedora. Explain that you would really like the current Fedora/EPEL6/7 maintainers to also maintain it for EPEL8. If they are willing, they will answer in the bug. If no answer after a few weeks, you could consider maintaining the package yourself. Consult with the epel-devel list or #epel-devel on IRC for further options.

Do note that mailing maintainer(s) directly isn’t nearly as good as just filing a bug. They would get the bug info anyhow in email, Other users might see that and add that they too want the package, the maintainer might hand off the package and the new packager could see the bug request but have no idea about private emails, some other packager might see the bug and offer to maintain it. All wins for a bug over private emails.

As the collection grows, these sorts of questions will likely die down, but it’s important to remember that every package needs (at least) one maintainer.

Comments Off on EPEL8 packages

MUA++ (or on to neomutt)

by on 2019/09/29 at 1:59 pm
Posted In: fedora, linux

About 18 months ago or so, I posted about switching my Mail User Agent / mail client from claws-mail to thunderbird: https://www.scrye.com/wordpress/nirik/2017/03/24/mua-or-on-to-thunderbird/ last week, I moved on to a new setup.

First let em talk a bit about why I am moving on from thunderbird. Since moving to it thunderbird seems to have gotten slower and slower (no matter how much I compacted mail folders). They made some, IMHO, anoying changes (like in thunderbird 52: “”When replying to a mailing list, reply will be sent to address in From header ignoring Reply-to header”, which is just dead WRONG). Recent thunderbird versions had taken to pausing randomly and just not doing anything at all for like 20-30seconds. I had become annoyed more than once because thunderbird handled all my filtering, so if I was offline for a while and reconnected, it would take a long time for thunderbird to filter my emails and during that time I couldn’t really do anything else. The final straw was thunderbird 68’s changes to add-ons. Since they were moving to the newer engine, all add-ons have to be reworked to be “webextensions”. Firefox went through this last year. However, thunderbird seems to have not handled it well. I didn’t see any press or announcement, just updated and suddenly all my add-ons were gone. Additionally, going to the thunderbird addons site, there’s a filter for version, but at the time, it didn’t even have 68 listed! Pretty much non of my addons were ready for this change.

So, finally I decided I would look at mutt. I had avoided it in the past for a few reasons: I wanted to be able to see html emails easily in the same application I was using to read the rest of my emails, and I just liked the idea of a application that didn’t depend on a terminal. With no real GUI MUA’s left, I decided to get over those and look at mutt.

Doing some reading and pondering, I ran into a number of places talking about patches to mutt (“If you want to use this, you need patch X…”) which lead me to neomutt. I don’t know the details, but my understanding is that mutt development slowed way down for a number of years, and a lot of patches piled up. neomutt is a fork of mutt with all those patchsets applied, with the goal of cleaning them up and getting them into the mainstream mutt package. They already got the sidebar patches in, and hopefully they will get others in over time. Since I like living on the edge, I went with neomutt, which has a handy copr made by the main developer.

Thinking about the filtering annoyance I looked at a number of possible process changes, but finally just went with an easy one: procmail on my mail server. It’s pretty old school, but it was already setup a long time ago, can filter things as they arrive and is easy to add rules to.

neomutt/mutt can of course use imap, but I like to have my laptop setup so I can read and answer emails even when off line (although that state is increasingly rare). So, I looked at syncing my imap store to my laptop. Many people seem to prefer offlineimap, but it seems it’s on life support now and the developers have moved on to a newer tool that isn’t really ready yet to be used. Additionally, offlineimap in rawhide (what my laptop runs) is currently broken because it uses python2 and a module it needs has been retired. So, I settled on mbsync. This is a fast, written in C program to sync between local folders and imap server (both ways). It took just a few minutes to configure and then a bit to sync all my imap folders down.

The final part of the new setup was searching. I really liked the ability in thunderbird to search everything, find threads if I just had one email, etc. The winner here is ‘notmuch’, a xiapan database for emails. After installing the package a simple ‘notmuch new’ and answering a few questions and away it went. After the initial indexing, it can (and should) be run after each mbsync invocation. You can then search from the command line or (if using neomutt) virtual folders and search queries. It much faster to come back than thunderbirds virtual folders. As a bonus all the emailed I had ‘flagged’ in thunderbird were still flagged to mutt, so I can continue to flag things I want to reply to and search for and reply to them later.

neomutt (and now mutt too apparently) has a sidebar setup, which is much like thunderbirds accounts/folders setup. I set mine to show any folder that has unread mail (and how many) and any folder that has flagged messages (and how many). Using that I can decide which one(s) I want to deal with. If I just want to crunch through them all, doing c (change folder) will default to the next folder with unread messages.

I of course have multiple accounts, but thats not a problem for neomutt or mbsync. Just need to define the local and remote folders and tell neomutt to read in those folders when you change to them. One issue that I was stuck on for a bit was how to filter the gmail accounts I have. (Since there’s no way to use procmail on those). Luckily it seems you can get google to do it. Just need to go to the account on the web interface, select the message(s), click on the little 3 dots “more actions” at the top, filter messages like these, set it to archive automatically (skip mbox) and set a label (which makes google put it in a imap folder with that label).

So, it’s been about a week with this setup so far, and I have to say, I really like it. I was off power for about 5 hours last week (power outage) and was impressed how much more battery life my laptop had without thunderbird spinning up fans and taking up CPU. Re setting up all my filters allowed me to find some things that need to be dealt with, which is nice. I think it’s made me more caught up/responsive to emails than before. It’s definitely more responsive/faster than thunderbird and I never really find myself waiting on the client.

There are a few things I miss from the old/previous setups:

  • Calendaring. Thunderbird had calendaring built in, and now I have to just pull up a web version. I did try gnome-calendar for a bit, but it doesn’t allow you to see who attendees are or answer yes or no or maybe on going to the meeting. So, web interface it is for now.
  • Thunderbird could display annoying html only emails well in the interface, but… I always prefer the text/plain part, and turn off remote image loading, so many of the most broken emails werent that pretty anyhow. Additionally, mutt can easily open those parts in a browser or the like.
  • When you are composing an email in neomutt, you can’t super easily just go and look at other emails. Thunderbird having a seperate window for this was nice. You can start another neomutt -R (read only), but that seems like a hassle. You can of course postpone your email, go look at whatever else, then resume it, but still a bit of a pain.
  • My old setup used spamassassin on the mail server to detect spam, but in addition I was using thunderbirds bogofilter support as a ‘second line’. Without this I have been getting more spams to my inbox (mostly to list moderator addresses).

And there’s a number of TODOs left:

  • I’m trying to decide if there is any point in keeping old email list posts or bug tracker emails. I was going to archive them, but then I wondered: why? I can get them from the list archives or the bug tracker application. Is there any great point in having more than say 90days of them around? Still pondering on that…
  • Right now I am running ‘mbsync; notmuch new’ manually (I bound it to f1 in neomutt) when I want. I wonder if it’s worth setting up a cron job? Right now it takes about 90 seconds to run (mostly because gmail is slow), but not sure any interval matters. I’m likely to pull before I start answering emails.
  • There’s still some weirdness around switching accounts. It seems like if I start neomutt and never go to an accounts folders, I never see new emails for that account until I do. Should figure out what I might be doing wrong and file a bug if it seems like it’s not my fault. 🙂
  • A full mbsync takes like 90 seconds. If I nuke a bunch of old emails (see point 1 above), I think I can reduce that time. Also, I looked at doing clever things like moving a bunch of old things to archive and not syncing them all the time, but that requires a lot of manual work, which I would like to avoid.
  • Look at setting up a local bogofilter or second line spam handler to reduce spam even more.

So, all in all, I am pretty happy with this setup. Happy to share any config or help others setup the same mail workflow. We will see in a year or two how well it sticks.

3 Comments

Flock 2019

by on 2019/08/19 at 12:29 pm
Posted In: fedora, flock, linux

Flock time is upon is! This time in lovely Budapest. As always when flock is in europe, it’s a long flight for me, but otherwise travel was uneventfull: Drive 2 hours to PDX, then PDX to AMS, then a short layover for coffee and stoupwaffles and then AMS to BUD, and finally a taxi ride to the hotel.

The hotel is quite lovely. It’s right next to the danube river and has a nice view. The AC is working nicely too (it’s quite hot outside here right now). After getting into the hotel yesterday and a quick dinner at a very nice place down the road, I managed to sleep for 10+ hours.

A nice complementary breakfast, then flock. I went to:

  • The FPLs yearly talk on the state of Fedora. This went over a lot of decisions and changes the Fedora Council made early this year around mission and how to better try and acheive and articulate our goals. Then some nice charts and graphs, and finally a bunch of things for us to ponder on how to solve.
  • Next up was a coffee break. I Love that we have these scheduled, it’s a great chance to visit with fellow fedorians.
  • The next big talk in the main room was Cate Huston from wordpress talking about how we all fail at projects and some things to try and fail less. It was a nice talk. Lots to think about, and definitely some things that resonated.
  • Final talk in the big room for the day was from the Facebook desktop support team about them rolling Fedora out as a supported desktop for their users. They went over their setup, some problems/issues they hope to work with Fedora to solve and took questions. Very interesting.
  • Nice lunch in the hotel resturant, then back to flock…
  • I went then to the Improving the Packaging Experience with automation. Some great ideas there, we just need people to try and drive them and spend some cycles working on them.
  • Then on to the Fedora Workstation status update and progress. Lots of great stuff here, most of which I knew, but it was great seeing all the progress in one place. Keep up the great work!
  • The next few timeslots I got pulled into the “hallway” track. ie, discussions with friends and co-workers and community members (and sometimes all 3 in one person!).
  • Next I went to Ben Cotton’s session on changing the changes process. I think the changes he wants to do will help out and make things run smoother. He had a nice overview of changes and there was a good bit of discussion with the audience.
  • I sadly missed the “slideshow kareoke” session as I stopped to talk to people.
  • The nights event was board games and relaxing, so we gathered up everyone from the CPE team that was there and had a very nice group dinner at a local place.
  • Then, back to the hotel and time to sleep…

The next day started out with a “Fedora, Red Hat, and IBM” talk from Denise Dumas. I’ve heard most of the content before, but she put a nice Fedora spin on things.

  • Next up I went to a panel talk lead by Brendan Conoboy about how RHEL8 was formed and where and why it diverged from Fedora. Not surprisingly, we want to try and merge things back as closely as we can to make RHEL9 smoother.
  • Lubomír Sedlář‘s talk on making composes faster was definitely something I had to attend. In addition to the ideas he had, the audience suggested a few more and I hope everyone went away with a good understanding of how composes are made.
  • Leigh Griffin had a nice talk about my team (CPE) trying to become more agile. I really like his approach to find what works for us and not just apppoint a scrum master and make everyone go through some ceremonies. What we end up with will depend on what we want and how we work, there’s no one-size-fits-all.
  • Laura Abbot gave a talk about the kernel tree of the future. There’s a lot of will to move things to a better workflow than they are today. I hope they can get it to a nice place with some hard work.
  • Finally up at the end of the day was my 25min communishift talk. The room was packed and there were tons of questions (which I hope I answered). Communishift is out community openshift cluster for Proof of concepts, development, tinkering or the like from the community. We are only responsible for the platform, users are responsible for everything above that. It seemed to get a lot of good feedback.
  • The evening event was a dinner boat ride on the river. The dinner and the boat and the company was great. There was a bit too much walking to get there in the heat, but we all survived.

Saturday started off with a showcase from the outreachy and google summer of code projects. It was awesome to see some of the projects they had worked on and the progress they had made. great stuff!

  • Most of the rest of saturday I spent talking to people and discussing ideas or problems. Got a lot of requests to add people to communishift, which I did later that night.

Sunday started off with a CPE hackfest. I worked it like a barcamp, so we tossed out ideas and topics of things we wanted to go over and voted on them. The top two were: “going over backlog of projects” and “improving the packager experience”. We split the room into two and started working away.

  • We got a ton of requirements / needs for a new monitoring / metrics setup. There was a lot of good discussion from people not on our team, but interacting with us on what they might want/need.
  • There was some discussion about how to try and move forward on our next generation account system. Stay tuned here there’s several good ideas being explored.
  • I finally got to talk to the copr folks and come up with some plans to move copr forward from being on our old cloud. Should be some good movement in this area soon I hope.
  • We got some progress on the CPE docs. Fixed some things and got them generating as we push changes to them.
  • After lunch we had the “meet your FESCo” panel. There were lots of questions and I hope a good discussion. Everyone got some good mic time and it went pretty well.
  • Finally the wrap up session, everyone who lead a session came up and gave a short overview. It was great to hear this for the sessions I couldn’t attend. Finally there was a SUPER great video made by tatica of a bunch of people at flock. We really are a diverse bunch.
  • Sunday night I had a final dinner with the team and then up at 4am to head home. Left the hotel at 4am, got home about 3pm the same day (and a ton of time zone changes in the way).

All in all another great flock in the books. This one seemed to have more energy / excited folks than I have seen in a while, and as always it was good to see old friends. Until next year!

Comments Off on Flock 2019

epel8-playground

by on 2019/07/30 at 11:52 am
Posted In: fedora, linux, rawhide

We have been working away at getting epel8 ready (short status: we have builds and are building fedpkg and bodhi and all the other tools maintainers need to deal with packages and hope to have some composes next week), and I would like to introduce a new thing we are trying with epel8: The epel8-playground.

epel8-playground is another branch for all epel8 packages. By default when a package is setup for epel8 both branches are made, and when maintainers do builds in the epel8 branch, fedpkg will build for _both_ epel8 and epel8-playground. epel8 will use the bodhi updates system with an updates-testing and stable repo. epel8-playground will compose every night and use only one repo.

The idea here is that maintainers can (if they choose) modify the package.cfg file in the epel8 branch to only build for epel8, and then use the epel8-playground branch (and it’s sperate repos/builds):

  • To test out some new version of the package that might not be stable yet.
  • To test out some new packaging of the package
  • To test a major version change of the package that they want to land at the next epel8 minor release.
  • To build a package that will never be stable enough for epel8, but still could be useful to some.

At minor RHEL releases (ie, 8.1, 8.2) people can pull in big changes from playground to the main epel8 packages. Since people will be upgrading and paying more attention than usual anyhow at those points, it’s a great chance to do that change, but also you want to make sure it’s panned out, so you can test before hand in playground.

We hope that this feature will be useful to some folks. Do let us know on the epel-devel list or in our next EPEL sig meeting!

└ Tags: , ,
Comments Off on epel8-playground

Changing how we work

by on 2019/07/21 at 2:59 pm
Posted In: fedora, linux

As those of you who read the https://communityblog.fedoraproject.org/state-of-the-community-platform-engineering-team/ blog know, we are looking at changing workflows and organization around in the Community Platform Engineering team (of which, I am a member). So, I thought I would share a few thoughts from my perspective and hopefully enlighten the community more on why we are changing things and what that might look like.

First, let me preface my remarks with a disclaimer: I am speaking for myself, not our entire team or anyone else in it.

So what are the reasons we are looking for change? Well, there are a number of them, some of them inter-related, but:

  • I know I spend more time on my job than any ‘normal’ person would. Thats great, but we don’t want burn out or heroic efforts all the time. It’s just not sustainable. We want to get things done more efficiently, but also have time to relax and not have tons of stress.
  • We maintain/run too many things for the number of people we have. Some of our services don’t need much attention, but even so, we have added lots of things over the years and retired very few.
  • Humans suck at multitasking. There’s been study after study that show that for the vast majority of people, it is MUCH more efficent to do one task at a time, finish it and then move on. Our team gets constant interruptions, and we currently handle them poorly.
  • It’s unclear where big projects are in our backlog. When other teams approach us with big items to do it’s hard to show them when we might work on the thing they want us to, or whats ahead of it, or what priority things have.
  • We have a lot of ‘silos’. Just the way the team has worked, one person usually takes lead on each specific application or area and knows it quite well. This however means no one else does, no one else can help, they can never win the lottery, etc.
  • Things without a ‘driver’ sometimes just languish. If there is not someone (one of our team or even a requestor) pressing a work item forward, sometimes it just never gets done. Look at some of the old tickets in the fedora-infrastructure tracker. We totally want to do many of those, but they never get someone scheduling them and doing them.
  • There’s likely more…

So, what have we done lately to help with these issues? We have been looking a lot at other similar teams and how they became more efficient. We have been looking at various of the ‘agile’ processes, although I personally do not want to cargo cult anything, if there’s a ceremony some process calls for that makes no sense for us, we should not do it.

  • We setup an ‘oncall’ person (switched weekly). This person listens for pings on IRC, tickets or emails to anyone on the team and tries to intercept and triage them. This allows the rest of the team to focus on whatever they are working on (unless the oncall person deems this serious enough to bother them). Even if you stop and tell the person you don’t have time and are busy on something else, the amount of time to swap that out and back in already makes things much worse for you. We of course will still be happy to work with people on IRC, just schedule time in advance in the associated ticket.
  • ticket or it doesn’t exist. We still are somewhat bad about this, but the idea is that every work item should be a ticket. Why? So we can keep track of the things we do, so oncall can triage them and assign priority, so people can look at tickets when they have finished a task and not been interrupted in the middle of it. So we can hand off items that are still being worked on and coordinate. So we know who is doing what. And on and on.
  • We are moving our ‘big project’ items to be handled by teams that assemble for that project. This includes a gathering info phase, priority, who does what, estimated schedule, etc. This ensures that there’s no silo (multiple people working on it), that it has a driver so it gets done and so on. Setting expectations is key.
  • We are looking to retire, outsource or hand off to community members some of the things we ‘maintain’ today. There’s a few things that just make sense to drop because they aren’t used much, or we can just point at some better one. There’s also a group of things that we could run, but we could just outsource to another company that focuses on that application and have them do it. Finally there are things we really like and want to grow, but we just don’t have any time to work on them. If we hand them off to people who are passioniate about them, hopefully they will grow much better than if we were still the bottleneck.

Finally, where are we looking at getting to?

  • We will probibly be setting up a new tracker for work (which may not mean anything to our existing trackers, we may just sync from those to the new one). This is to allow us to get lots more metrics and have a better way of tracking all this stuff. This is all still handwavy, but we will of course take input on it as we go and adjust.
  • Have an ability to look and see what everyone is working on right at a point in time.
  • Much more ‘planning ahead’ and seeing all the big projects on the list.
  • Have an ability for stakeholders to see where their thing is and who is higher priority and be able to negotiate to move things around.
  • Be able to work on single tasks to completion, then grab the next one from the backlog.
  • Be able to work “normal” amounts of time… no heroics!

I hope everyone will be patient with us as we do these things, provide honest feedback to us so we can adjust and help us get to a point where everyone is happier.

└ Tags: , , , ,
Comments Off on Changing how we work