dumpster fire^W^W2020 year in review

by on 2020/12/31 at 3:14 pm
Posted In: cats, dogs, fedora, flock, linux, pets

Since today is new years eve and 2021 begins here tonight, I thought I would take a look back on 2020.

The big item for me in the first and middle part of the year was our Datacenter move. Planning for that started last year and spilled into early 2020. Then, setup of new machines, migration of services for a minimum Fedora, then moving all the rest of the machines, then getting them online. It was a ton of work, not only in planning, but setting up things, migrating and communicating with everyone involved. Overall I think it went pretty well. Downtime was pretty low for the most part and we managed to get things up pretty fast. It wasn’t perfect of course, we _still_ have things we moved to one datacenter that are not back online yet, but hopefully soon in 2021. I really think how well it worked was a testament to everyone involved: Folks on our team, RHIT planning and networking, and even all the Fedora community members (who were super understanding of the outages and issues!). Here’s to never ever moving from the new datacenter ever again. 🙂

I actually did manage to get one trip in in the early part of the year before covid-19 destroyed everything: I made it to devconf.cz. A excellent time as always!

You may think that the pandemic didn’t affect me much. After all, I am an introvert that lives in the forest 20 miles from the nearest town. However, it really did: When everyone started working from home there was a big flood of video meetings and such, I think because everyone who normally talks to lots of people at the office wanted contact more, and that caused a lot more meetings, making things hard to get done. That died down some, but all the stress about whats locked down and what precautions you need to take to go to the store and all the US politics doom mixed in made it pretty stressful, even when avoiding people.

We managed to get Fedora 32 and 33 out the door, and on time again for the most part!

Flock was of course canceled, and I feel sad not being able to see all my Fedora friends in person. However, nest with fedora was pretty great! Not the same, but still good.

I got a new 2020 dell xps 13 laptop. It’s alright, but I am not amazed by it. I might need to upgrade sooner than the normal 3 years if something amazing comes out and finances permit.

I picked up not one, but two pinephones. I have had a bit of time to play with them over the holidays, but need some more. They are still not to the level of what I would need for daily use, but they are rapidly advancing. Here’s to a Fedora pinephone spin in 2021. 🙂

I decided I should try more microblogging, so I setup a mastodon account: https://fosstodon.org/@nirik Come and join me there and dump twitter. 🙂 No idea if it will stick, but I am going to give it a try. I really didn’t blog much in 2020, given all the datacenter work and doom, perhaps I will try more in 2021.

On the work front, we had some great folks migrate out to other places and some new people come in. If there’s one thing that always stays the same, it’s change. I am pretty happy about how we have moved to actually planning larger projects (initatives) that our team works on now. I think it’s much better for everyone to know what we are working on, what the priority is for everyone involved and clear ideas what done is for those things. Hopefully we keep it up and refine it in 2021. Another thing that I think has been a smashing success is our daily ops standups. Just getting a few folks together every day for 30min to triage tickets, process quick things and discuss things has been great! We have really killed the infrastructure ticket queue down before the holidays. In 2021 we hope to finish that and work on the releng ones.

Finally here we gained a kitten (she was a feral cat that decided she loves people), lost the last of our dogs (he was nearly 15) and just had a bunch of stuff done in our back yard (retaining wall, a bunch of paver stones around our deck, it looks really nice!). I expect in 2021 after we get the back yard grass grown we will look at getting some more dogs, it’s… strange without them around. We have also started (before Christmas even) to eat more healthy and excersize more (I’m down 9lbs so far… at least it’s a dent in the pandemic weight gain.).

Hope 2021 is a great year for everyone! Good riddance to 2020!

Comments Off on dumpster fire^W^W2020 year in review

Matrix and Fedora

by on 2020/11/28 at 7:32 pm
Posted In: fedora, linux

Recently the Fedora Council has been floating the idea of modernizing the Fedora community real-time chat platform (currently IRC hosted at freenode.net). The front runner is matrix. I last looked at matrix 4 or so years ago, so I thought it would be a good time to revisit it and see how it looks today.

TLDR: I suspect we will have IRC and Matrix bridged together for a long time to come, if you are new user, use Matrix, if not keep using IRC.

First a few words about IRC (Internet Relay Chat). IRC is a 30+ year old chat protocol. There’s tons of clients. There’s tons of bots and add-ons. There’s tons of gateways and aggregators. So, whats not to like? Well, everything is a add-on mish mash that can be very confusing and frustrating for new users. For example, IRC has no persistance, you only see messages while your client is connected. So, folks invented irc “bouncers” to connect for you to the IRC networks you care about and when you reconnect play back all the messages you missed. Authentication is via messaging various services bots. Encryption is via plugins or other add ons (and often not setup). So, most old timers have a client they have carefully tuned, a bouncer and a bunch of custom bots, which is fine, but new users (not surprisingly) find this all a hassle and frustrating. IRC also has it’s own culture and rituals, some of which still make sense to me, but others that don’t.

Matrix on the other hand is pretty new (6 years). You can interact with it as a guest or have an account on a particular homeserver. If you have an account all your history is saved, and can be synced to your client on login. You can send pictures and moves and fancy gifs. You can (somewhat) have end to end encryption (see clients below) with encrypted rooms where the server can’t know what was said in the room. You can have ‘reactions’ to things people say. You can redact something you said in the past. You can have a nice avatar and a Real Name (if you like). You can join rooms/conversations with other matrix servers (for example the kde, mozilla and others are running servers). You can get read receipts to see who read your message and notifications when someone is typing (also client dependent see below).

Finally there’s matrix <–> IRC bridges. These allow conversations to be relayed from one side to another. Of course some things don’t translate over at all (reactions, receipts, typing notification, avatars) and some look different (if someone in matrix posts a picture, the people on IRC accross the bridge will see a link to the picture). The main text of the conversation is properly relayed to both sides.

I managed to re-setup my matrix homeserver (matrix.scrye.com) from 4 years ago. Ran into a little problem with the version of matrix-synapse in Fedora (It’s too old to federate right, so I built the new one and the other update thats blocking it. Hopefully that blockage will be fixed soon). It’s still a memory hog, but it runs well enough. They are working on the next generation server written in go (dentrite), but it still lacks a lot of features.

After that it was on to testing clients. There’s a bunch more available than there used to be, which is great. The documentation on them is pretty much missing for most of the clients (exception: element has docs).

The premier client is element (used to be riot). It’s available as a web app, an android app and a native linux app (which isn’t available except direct from them that I could find). The web app and android app are likely the most full featured of the clients. They support setting up client encryption and cross signing your connections so all of them can read the same encrypted rooms. For chat clients, I really prefer stand alone, as web apps have a lot of issues (not restarting on browser restart, notifications not working right, poor integration into the desktop, etc).

Fractal is the gnome native client, available as a flatpak. My impression: full screen is horrible as the chat text is centered in a small col in the middle. No way to adjust the text size or font, making it really small and non readable by me. On the plus side, it does have a ‘take me to next room with unread messages’ key, which is really nice.

nheko is a QT client with a mix of features implemented, it’s available as a rpm packaged in Fedora. My impression: Looks pretty nice, nice to be able to tag rooms into groups. Looks good full screen. There’s only really a “this room has unread messages in it”, not any indicator of _how many_ and no easy way to go to the next room with unread in it (that I could figure out). No docs at all anywhere that I could find. 🙁

Quaternion is another QT client with a mix of features implemented. No end to end encryption, but lots of the other features. It’s also available in Fedora as a rpm. My impression: looks pretty nice, lets you tag rooms and seperates them easily. Doesn’t seem to have a ‘go to next unread room’ function. ;( No docs.

spectral is a c++ client. Its packaged in Fedora, but it seems to crash on launch for me, so I didn’t get much chance to look it over. ;(

FluffyChat is a port of a native android matrix client. It’s pretty full featured and available as a flatpak. Does the chat sort of ‘sms’ style, which is cute and all, and fine for small rooms, but bad for larger discussions and such. Otherwise looks outstanding and is really fast!

Neochat is another QT client available as a rpm in Fedora. I had to tinker with my server setup and get /.well-known/matrix/client and server working before I could get neochat connecting. After that it connected fine, it was really fast, but

All of the clients seemed to handle basic chatting and history fine. Other features are all over the map. Element was the only one I saw with a search feature (to search the history). None of them had logging, which I guess could be mooted at least partly by a good search of the history/backlog. Element was the only one that seemed to have the url previews working (where the server fetches them for you and shows a preview in the client). I am not sure why so many of the clients are using QT, perhaps because kde is running their own matrix server?

So, as far as clients, I’m really missing easy ways to go to the next room with unread messages in most of them (I use this ALLLLLLLLLLLLL the time in hexchat). Logging/searching is really important to me too. I often have to look up what happened back on day X or see the exact command I used to solve something a year ago.

If you’re a new user/contributor these days I think it completely makes sense to just use a matrix client. You get history without having to deal with a bouncer and some nice other features and you can bridge to all the old fogeys on IRC. If Fedora gets it’s own home server this will be even easier (as I assume you will just be able to use your fedora account to login and create an account for you).

The real question is how long should we keep the current situation with Matrix and IRC bridged? What advantages would be dropping the irc bridges bring? Right now, not too much. End to end encryption isn’t that interesting for an open source project. Reactions are interesting (think about using them to vote up or down proposals in meetings?), but we have done without them so far. I think migration from IRC is going to be a long process, nor is there great advantage to pushing things to go faster. I hope that over coming years matrix clients continue to get better and implement more features. Someday (probably years down the road) more Fedora users will be on Matrix than IRC, then sometime after that things will have shifted enough that the community will start assuming you are on Matrix.

I have also a few other things I use my existing IRC client with: a bitlbee server to pull in other IM/twitter/etc, and a few old IRC servers that I still hang out in, so it probably doesn’t make sense for me to move over to matrix full time yet.

One additional thought: we have several IRC bots that do various things on IRC. Matrix has a handful of bots, but nothing like IRC. It’s practically a programming rite of passage to make a IRC bot. 🙂 I think we could safely look at starting design on bots for our needs for matrix and switch to them when ready (but again, no hurry at all here).

Comments Off on Matrix and Fedora

ansible 2.10.x and Fedora/EPEL

by on 2020/11/02 at 3:19 pm
Posted In: fedora, linux

As some of you all may know, there were some big changes around how ansible upstream is distributed and maintained with the 2.10.x release(es). I thought I would recap for everyone who was not aware of these changes, then share my plans for the Fedora/EPEL ansible packages. Everyone is going to end up in a better place after this settles out, so there’s no need to panic. 🙂

ansible has some good problems with their community: They are very popular, very easy to work on and very widely used. This means there’s a flood of people always submitting bug reports, pull requests, fixes, enhancements, and all manner of things. In the old ansible setup before 2.10, this resulted in bottlenecks. The core ansible maintainers couldn’t review, merge and handle all the incoming flow of things. There were a bunch of things that were tried to help this (ansibot marking up issues and PR’s), making modules git submodules of ansible, making more people ‘community maintainers’ with merge and other powers over some modules, etc. Even with these measures however, that hits another problem: ansible only releases every N months. If you have fixed some big bug in your module, users aren’t going to get that fix very quickly at all.

The 2.10 changes are another stab at fixing or at least mitigating things so that ansible can surge forward with all the community energy behind it. Basically: The ansible “engine” has been split off into ‘ansible-base’. This allows it to move as quickly (or slowly!) as it needs to to be stable and improve things. ansible-base does have some modules included, but only the very base ansible modules like copy or the like. Most of the rest of the modules that used to be in older ansible versions are now shipped as ‘ansible-community’ in a group of collections that have all agreed to ship at the same time as a bundle. Each of those collections are seperate and able to be managed by people who care about that collection. Finally there’s all the other collections out there that can self manage, release when they like, etc. the ‘ansible’ collection contents can be seen in: https://github.com/ansible-community/ansible-build-data/blob/2.10.1/2.10/ansible-2.10.1.deps (for example for 2.10.1).

So now we have ansible-base (the engine and just a few base modules), ansible-community (a collection of collections that is most of the modules in old ansible +- some) and all the collections in ansible galaxy. Each can now be maintained by people who care about just that collection/thing and update on their schedule. This removes the central bottleneck and hopefully gets things cranking along at a good pace.

So, what does this mean for Fedora/EPEL? Well, for one thing, upstream is no longer going to ship rpms or tar.gz releases of ansible-base (“ansible”) or ansible-community. They have decided distributions can do a fine job there and it’s just duplicating effort (releases of ansible/ansible-base and ansible-community will be on pypi just like many other python upstreams). Next, of course, ansible-2.9.x is not going away anytime soon. It should continue to get updates for a while. I’m planning on submitting ‘ansible-base’ package to Fedora review today (it’s all ready except that you cannot build full web docs currently as it needs a stack of new packages and a bugfix/update to python-pydandic, so I will be disabling full web docs to start with). Then I am going to start on a ansible-community package. Once they are both done in rawhide and the packaging is all sorted out, I’m planning on encouraging collections in the ansible-community meta collection to packaging and maintaining their own collections along with any collections that are not part of that meta bundle that want to package up. Once thats all working, we will look at retiring the old ‘ansible’ (2.9.x) package in favor of ansible-base + collections and push that out to other Fedora branches and EPEL. Some versions of EPEL (I’m looking at you 7) may stick with 2.9.x for as long as possible as 2.10.x may start to use newer python features that won’t work on EPEL7. EPEL8 may need to switch to newer python module, but time will tell.

I’m looking forward to this new setup and the added velocity it gives ansible!

Any feedback/comments/better ideas are welcome in the tracking bug for ansible-2.10.x: https://bugzilla.redhat.com/show_bug.cgi?id=1848739

Comments Off on ansible 2.10.x and Fedora/EPEL

Bugzilla and the art of package maintenance

by on 2020/10/31 at 6:15 pm
Posted In: fedora, linux

The first part of 2020 (in addition to all the horrible things happening in the world) was pretty heads down in our datacenter move. Now that that is over, and Fedora 33 is out the door (hurray!), I’m slowly trying to catch up on other things I normally do that are now a bit backlogged. This weekend, I decided to try and catch up (at least some) on my package maintaining and bugzilla bugs.

In the distant past (before 2020) I basically tried to work on this stuff mostly as it came in. Update packages when release monitoring let me know there was a release, ask questions of bug reporters as they reported and try and get something actionable. Of course now I’m behind, so lets try a different process to catch up.

First, some of you may be aware that there is a python module for bugzilla (python-bugzilla), but did you know that it also includes a handy command line interface?

Lets try some initial queries and see if we can triage things to work on things in (hopefully) some semblance of their priority.

% bugzilla query -a kevin@scrye.com -s NEW,ASSIGNED  | wc -l      
102

So, 102 bugs. Not bad at all! Oh, but wait, those are only the ones I am directly assigned to, I am in a number of groups that get assignment and I am just CCed. How bad are those:

% bugzilla query --cc kevin@scrye.com -s NEW,ASSIGNED  | wc -l
412

Well, thats not great. But some of these might be bugs that are since solved or someone else is busy working on them or they have been made moot somehow. So, I will dump that query into a text file (The default output is to give you bugnumber, state (NEW,ASSIGNED), and the summary of the bug. Amusingly, I see the oldest bug I am assigned to was filed in 2011. 🙂 So, I have those to look through and do something with, but perhaps we can make some smaller lists:

bugzilla query -a kevin@scrye.com -s NEW,ASSIGNED  | grep CVE | wc -l
3

Cool, Only 3 CVE’s are assigned to me directly. 2 of those are tracking (fedora,epel) for a ansible bug thats not fixed yet and one is for a libntlm bug. Hurray, a CVE I can check on.

So, what I usually do is read the CVE bug, looking in particular for versions (this one affects versions through 1.5). I then look on https://src.fedoraproject.org/rpms/libntlm to see what versions are in what releases. All 1.5 here. Next, I look at other libntlm bugs by going to the handy: https://bugz.fedoraproject.org/libntlm In addition to the 2 CVE tracking bugs (fedora,epel), I see there’s a release monitoring bug about 1.6 being available. Great. Time to do some packaging work.

So, this is a library, so we need to be carefull about changes and what might depend on it. Looking at the NEWS file for 1.6 however, we see: “** API and ABI modifications. No changes since last version.” Great. Thats an update and 3 less bugs. 🙂

ok, on to messing with a few packages/components that more ‘system wide’. epel-release, distribution, etc.

Then, lets look at packages that are needing updates:

bugzilla query --cc kevin@scrye.com -s NEW,ASSIGNED --outputformat '%{id}: %{component}: %{summary}'  |sed -e 's|^\#||' | sort -n | grep 'is available'
1271027: python-stevedore: python-stevedore-3.2.2 is available
1757650: python-feedparser: python-feedparser-6.0.2 is available
1771091: python-mechanize: python-mechanize-0.4.5 is available
1801531: awscli: awscli-2.0.61 is available
1813734: hiredis: hiredis-1.0.0 is available
1815307: bodhi: bodhi-5.5.0 is available
1829595: python-redis: python-redis-3.5.3 is available
1839836: blueberry: blueberry-1.3.9 is available
1846161: iperf3: iperf3-3.9 is available
1848739: ansible: ansible-2.10.1 is available
1862178: python-eventlet: python-eventlet-0.29.1 is available
1862531: python-vcrpy: python-vcrpy-4.1.1 is available
1880754: calibre: calibre-5.4.2 is available
1881455: python-greenlet: python-greenlet-0.4.17 is available
1886993: mxml: mxml-3.2 is available
1888249: python-pygal: python-pygal-3.0.0.dev1 is available
1890435: koji: koji-1.23.0 is available
1890954: GitPython: GitPython-3.1.11 is available
1891597: libnftnl: libnftnl-1.1.8 is available
1891769: nftables: nftables-0.9.7 is available
1893366: wordpress: wordpress-5.5.3 is available
1893453: iptables: iptables-1.8.6 is available

Some of those are blocked (calibre needs sip5 packaged, ansible needs 2.10.x packaged (which I have done, but need to clean up and submit)), but some of the others should be pretty easy. 🙂 Lets see how far I can get today!

A few hours later… 

% bugzilla query --cc kevin@scrye.com -s NEW,ASSIGNED  | wc -l
380

Dealt with 32 bugs. Not a bad afternoon I guess. 🙂 I hope the command line bugzilla use and triage helps others deal with bugs as well.

Comments Off on Bugzilla and the art of package maintenance

Nest with Fedora

by on 2020/08/25 at 2:28 pm
Posted In: fedora, flock, linux

With everything going on in the world, sadly the normal “in person” Fedora conference, flock, had to be canceled. In it’s place a ‘all virtual’ conference was scheduled for Aug 7th through 9th: “nest with fedora”

Sadly the keynotes were all way too early for me to make it to live (4-5am my time is just too early), so I am going to catch those via recordings as soon as they are up. I did manage to make it to some sessions however.

First a quick note about the platform that was used: hopin.io. I’ve been to a number of virtual events of late, and most of them have not been that great. However, this time I was pleasantly surprised! hopin is actually pretty well done. There was a ‘reception’ area and a live chat for conference wide discussions, talks appeared a few minutes before their scheduled time to allow people to join and each talk had it’s own chat. Speakers could invite other people to join in the video chat part if they wanted. There was a conference wide ‘poll’ section with lots of silly polls in it for people to take. Mixed in the talks were some ‘hallway’ track places where everyone who joined could join via video. There was also a timed thing where you could get 5min with a random other person then it would switch everyone to the next person. Not as good as a real hallway track, but not bad given the constraints. Video and audio worked pretty well.

There was a number of nice talks I went to, as well as some fun ones (CPE put on a pub quiz and a bingo, both were great!).

Smooge and I gave a really quick talk about our recent Datacenter Move. I also thought the ‘meet your FESCo’ talk was interesting, there were some great questions.

All in all, not as good as a real conference, but a lot safer and easier and not too bad overall. 🙂 Many thanks to all the organizers!

Comments Off on Nest with Fedora