Flock to Fedora in Cork, IE (2023)

by on 2023/08/10 at 7:26 pm
Posted In: fedora, linux

Just got back from our first in person flock since 2019 and it was amazing. I thought I’d share my journey and thoughts from it here in longer form.

I did do some prep work the previous week in that I tried to make sure as best as I could that infrastructure and releng stuff was stable and wouldn’t need any interventions from me, which mostly worked out in the end, with only some releng work (starting mass signing of f40 in prep for branching this coming week) and some troubles with the matrix/irc bridge (which I can’t do much about aside from letting people know).

Travel to flock was fine. I took advantage of my favorite flight from PDX to AMS and then to Cork directly. In AMS I ran into Justin Flory and David Cantrell, who were both not planning on being there at the same time as me, but due to flight changes were. The flight to Cork was quick and we took a cab to the hotel.

The hotel was fine. It was a bit out of the center of town, which meant you had to take a cab or bus most of the time, but it wasn’t too bad. The rooms oddly didn’t have heating or cooling, but did have windows that opened, so I just kept mine open and was just fine. The conference center was in a seperate building behind the hotel and it was a bit weird to walk all the back there on the 3rd floor to get to it, but it worked out fine. The elevator had a amusing ‘off by one’ error in it. Ground floor was “1”, the first floor was “2”, etc. The food was all fine to me. The hotel did a buffet breakfast and did lunch/coffee breaks, etc.

I left Monday, but arrived Tuesday afternoon, got settled and looked around, then off to dinner with various folks. It was really nice. We went to a place called Market Lane and they did a pretty good job accomodating a large group of roundy nerds.

Wednesday the conference kicked off with some introductions from Justin and then Matthew Millers ‘State of Fedora’ talk. This was great as always, but sadly the streaming / recording had problems at the beginning, so Matthew redid the talk on the last day in the afternoon for a smaller audience. I think both were great. I think I liked the first one better, but might be because I was less tired. Next I went to Justin Forbes talk on the state of the kernel, informative as always. Then on to a roundtable about packaging problems in modern languages from Jens Peterson. Some good comments from a number of folks, but I am not sure we came up with much plan to help aside from moving to more bundling. I was hoping we could look at sharing tooling to handle large package sets, but we can discuss this more on other platforms moving forward. Then, on to a talk about the current state of Infra and Releng applications from Akash and Aoife. They made this talk really nice and fun and gave out prizes! Great work by them putting this together. Next I wanted to go to the RiscV talk, but somehow decided to go to the AI/ML in QA talk instead. Really interesting seeing how to try and use AI for our QA efforts. I’m not a big fan of AI/LLM’s in general, but I think they do have uses, and this is a clever one. Next up was a state of Fedora CI. It’s amazing how long it’s taken us, but we are finally getting there with CI. Next to a talk about EPEL. Great to get more visibility for a subproject of Fedora thats so popular and useful. The great Flock international candy swap took place. So much cool and interesting candy. Someone (Carl!) even brought some jerky. Tons of good stuff. The day ended out with a lot of talking to various people at the game night, the hotel bar and then finally the hotel lobby when they kicked us out of the bar.

Thursday started in too early at 8:30 with a great keynote by Jen Madriaga on Diversity, Equity and Inclusion. Some very good points and things to think about. We can all be better here and help each other. Next up was a “Meet your FESCo” session. We only had 4 of the 9 FESCo members present this time, but we talked and answered questions and hopefully made some amount of sense. I then headed out to a talk on whats new in systemd. Wow, so many things I had no idea about. I hope the slides for this are up somewhere because even though I followed along on my laptop, there were things I didn’t get to trying out. So many good things. Next was a talk on ansible packaging in Fedora / EPEL. An excellent overview from gotmax23, who I was finally able to meet in person. So nice to have someone take over the ansible maintainer mantle. I almost always enjoyed it, but I just don’t have the time to devote to it that I used to. It’s in good hands. Next was Matthew Millers discourse discourse, but it didn’t really get into how to move it away from him doing so much and more got into a general discussion about it and how and if we should convince people to move there. Still lots of good info and things that were good to bring up. Next was the upstream colaboration in Enterprise Linux panel. It was all surprisingly cordial and for much of it the panelists seem to all agree on things. Then the evening events: dinner at a mexican food place and a scavenger hunt. I was wiped out, so I headed back to sleep after the dinner.Even though I passed out at like 8:30pm, I didn’t really seem to catch up on sleep much. Seems to be what happens at flock.

The last day of flock, Friday, again started off at 8:30am. This day was devoted to a Mentor Summit. After a introduction I was on a Mentoring panel. This session was one of the best of the conference I thought. Some really great questions from the audience and from our moderator, Amita Sharma. She kept us going with great questions and the time flew by. My fellow panelists did an awesome job too. Next I went to a workshop I was running with James Richardson on revaming our onboarding and mentoring and docs about those in Fedora Infrastructure and Release Engineering. Surprisingly we had a really nice crowd of folks and we dived right in. Tons of good ideas and suggestions. As soon as we are recovered from travel, James and I will be writing things up for a round of review with the community and then we can dive in and revamp the docs and start trying ideas. Infra and Releng are great, fun areas to contibute and I look forward to onboading and mentoring a bunch of new folks. The workshop was scheduled to go on after lunch, but we lost a lot of people (either leaving early or going to other sessions), so we did just a bit and wrapped things up. Then off to the final night. There was a ghost tour, but I was tired and a bit footsore so I tagged along with some folks getting dinner in Cork and then passed out early for my super early travel back.

My saturday started at 4:30am or so. Got up, showered, packed and checked out and met the cab to the airport at 5:30am. Then flight to heathrow (which I had never been to before). Amusingly, my Brother had been vacationing in the area and was in fact flying back to the US that same morning. So, we met up in the airport, he got me into the lovely Virgin Atlantic Lounge where we had breakfast and caught up a bit. Sometimes these weird things work out. Next was my 9.5 hour flight to Seattle. That went mostly fine, I usually just read or listen to podcasts and ignore the world. I have to say that noise canceling headphones are sure nice for these trips. I picked up a new sony WH-1000MX5 set before this trip and they did an outstanding job. Tons of battery life, super good noise canceling. Landed in Seattle and then walked. And Walked. And walked. I think it was probibly a mile or two of corridors and up and down and around before getting to the passport control line. It was really a lot of walking after being in planes all day. Finally got past that and… my next flight was in another terminal, so more walking to a train and more walking. 🙂 (I did over 10k steps saturday). Finally got to my gate and… they changed the gate. Got to the new gate and… they didn’t have a driver for the bus from the gate to the plane. When they finally did they made me check my bag because of limited space. Finally off and landed in Portland, then in to wait on my bag. Then shuttle to the parking lot where I parked and finally the 2 hour drive home. Whew. Pretty epic day.

I have to say the hallway track was excellent as always. I had a number of really nice coversations with all kinds of people on all kinds of topics, from Irish taxes to Community Building to Mobile devices and hardware support to Boating to Weather to Books, etc.

So, whirlwind travel, but really really nice to see people in person again who I talk with most days over the network. I really hope next year we get more of the folks who were not able to make it this time around. As always flock leaves me body tired, but mind bursting with all the posibilities!

Comments Off on Flock to Fedora in Cork, IE (2023)

A story of a Distributed Denial Of service

by on 2023/06/26 at 5:15 pm
Posted In: fedora, linux

We had a DDoS hit our DNS servers a few weeks ago, so I thought I would write up what happened for anyone interested.

First, a bit of background: Why do we ( Fedora Infrastructure ) run DNS servers? Well, we run them to provide users resolution of our domains. It’s worth noting that we don’t provide recursive servers that will answer queries for any domain, but just authoritative servers for the domains we manage. Doing this allows us to quickly update things (which we depend on to take proxy servers in and out of rotation) as well as make sure we have dnssec working and other configuration. If we were setting this up these days, we might very well go with a trusted 3rd party provider, but we predate those really existing and for the most part it’s worked fine for us. We have a number of DNS servers, 2 of them in our main IAD2 datacenter and the rest spread out to various other places we have presence.

So, whats a DDoS (Distributed Denial of Service)? Basically it’s flooding something with requests, but not from one point/ip/subnet, but from a distributed network of machines. You can’t simply block requests from an ip/subnet, the requests are coming from everywhere!

The DDoS that hit us started pretty sharply. We started getting alerts from our monitoring, and then… couldn’t reach any of our machines in our main datacenter. Luckily we have close contact with the networking folks there and they were able to see that we had hit a connection limit in our firewall there (10million connections). They were all connections going to our DNS servers, so we had the networking folks clear the connections and block DNS from our main datacenter until we got a handle on things. That got most everything back working, but DNS was still getting flooded, leaving some users unable to resolve our domains.

I shifted to working with the remaining DNS servers outside our main datacenter. Looking at the requests that were coming in, it seemed that many or most were from legit looking nameservers or ips. So, this attack was in fact a indirect one. They were querying recursive DNS servers to query us for domains we controlled. So, blocking would not do anything but block legit users mixed in with the recursive servers. So, I took several tacks: First, I increased limits we had in place to allow those servers to process a lot more connections at once (from 1000 or so to 10000). Secondly, I setup some limits per zone. All the queries were for our old fedorahosted.org domain, which had a wildcard (ie, anything.fedorahosted.org). Limiting per zone (which bind can now do) helped other more important domains get replied to while the queries against fedorahosted.org were limited. Finally I removed the wildcard on fedorahosted.org (we haven’t used that domain in many years, nothing should us uing it now).

After doing those things (or perhaps cooincidence?) the attack stopped. Just as our servers were fully able to handle the queries coming in. So, hopefully we are better set if this happens again. If it becomes too common we may have to look at moving DNS off to a 3rd party that specializes in handling this kind of thing.

Finally, several people asked me what motive the people doing this might have. I have no idea, we are a humble Linux distribution here. I don’t think we will every know more.

Comments Off on A story of a Distributed Denial Of service

That new car smell

by on 2023/05/27 at 11:09 am
Posted In: cars, linux

After about 9 months on a waiting list, we finally got our new car last tuesday. It’s a rav4 prime XSE with premium package (ie, the highest end one they make).

A short thing at the top here for those interested in Open Source/Linux. The manual for the car points to a site that shows 53 components as having source available in one place via LG and another site for the informational display. It’s got all the usual things you might expect: kernel patches, busybox, glibc, systemd, util-linux, etc. It’s of course unclear how it all builds or works, but thats typical. At least there’s a ton of open source here (like there is everywhere).

Now, read on for long boring car stuff. 🙂

So why did I decide to go for this make/model? Well, it’s pretty ideal for us here. We live about 15mi from the grocery store/shops, so when we go into town for shopping or appointments it’s about a 30-40mi round trip (depending on which side of town the thing we are going to is on). The rav4 prime has a advertised 42mile all electric range, so 95% of the time we should be able to be all EV and burn no gas at all. On those occasions when we go further (trip to the coast and back: ~90mi, trip to the portland airport: ~230mi, longer road trips, etc) we can just run in hybrid mode and get great millage and not have to worry about recharging or range. The prime has a 14.5gal tank, although toyota is apparently really concerned with anyone running out of gas, so when it says “0 miles” and the fuel gage reads empty, there’s still about 3 gal in the tank (~100+ miles). That gives it a range of around ~500miles+~100mi reserve. I also really really liked the engineering involved in this car. They really did a very clever job using motor/generators and planetary gears. It’s nice to know I never need to worry about: belts (there are none), starters (There isn’t one, it uses a motor generator and the battery to start the engine), alternators (there’s a dc/dc converter, no alternator), or driveshaft (the back 2 wheels have their own generator). The climate system is all heat pump based, moving hot/cold from where it is to where it’s desired.

Some first impressions: Loving it.

  • Yesterday went to the store and back in 100% EV mode, round trip 26.4 miles, and used 36% of the battery.
  • The magnetic grey metallic color just looks super sharp.
  • The “wave your foot under the back and it opens/closes the back hatch” is awesome.
  • It’s going to take some getting used to to just opening the door and having it know you have the key and unlocking for you.
  • Thursday went to the portland airport and back in HV (Hybrid vehicle) mode. 232miles. It claims I got 51mpg, which I suppose is possible. There was a lot of stop and go traffic when the engine just turned off and I think the fastest I got going was 65mph. Ave speed was like 38mph. Still have about 2/3rds of a tank of gas.
  • Radar cruise control with lane assist is pretty cool, but will take some getting used to also. The lane assist seems to want you right in the middle of the lane, but I found that my habbit was to be more toward the outside of the road (to stay further from crazy drivers). The radar cruise control wanted to brake later than I would have, ie, I saw break lights ahead and would have started slowing, but it waited until the car in front was slowing a lot before it started breaking.
  • The 2023 infotainment thing (they redid it completely for 2023) is pretty great. Voice commands seem to work as well as those things always do, navigation was solid, music via my phone bluetooth was great, phone calls and texts worked great. One interesting thing I found out is that my phone won’t do android auto. This is because I’m running grapheneos, and android auto requires google play to have a lot of privs that graphineos doesn’t let it have in the sandbox it runs it in. As far as I can tell right now though, I don’t care. The regular phone via bluetooth integration seems to do everything I normally would want. Perhaps I’m missing some great android auto thing I am unaware of though.
  • Ventilated seats are awesome. Really makes things cooler and more pleasant.
  • I had to really punch it a few times on the airport trip in HV and… man does it go. (302hp)
  • The cameras for parking/backing up are outstanding. We put a doormat in the garage and when I pull the car in I can clearly see looking down where the mat is so I know where to position the car.
  • Blind spot monitoring is great. No more craning around to see if you can switch lanes. It just lights a light on the side mirrors when someone is there.
  • Digital rearview mirror took a little getting used to also (this is basically showing you behind the car with the camera, so you don’t need to worry about peoples heads in the way,etc). I found it hard to refocus on it at first, but after a while I really liked it. It’s much clearer than the non digital one since it doesn’t have to look through the ‘privacy glass’ (ie, tint) in the back and can gather light much better. I will probibly use this all the time now.
  • The heads up display is awesome. Being able to see speed, charge / eco /power, lane assist/radar cruse, speed limit and songs when they change is great.
  • Sign detection is awesome. It basically looks for signs (speed limit, stop, yield, etc) and shows them on the panel/HUD. I can’t count the number of times when driving in the past I said to myself “Hey, what was the speed limit here?” Now you know. It does put the sign in red/yellow when you speed, but it doesn’t beep or care otherwise.
  • There’s some gamification on ‘eco driving’ that I haven’t really paid much attention to. Basically it gives you a score on starts / stops / etc from 1-100. My score way jumping around, but I was mostly ignoring it to play with the car, perhaps I’ll look at it more in coming years.
  • The charging schedule stuff seems to work fine (if you fully plug it in, I didn’t have it actually fully plugged in the other day and surprisingly it didn’t charge until I did). So you can tell it you are departing at 2pm on friday and it will start charging so that it’s done just before 2pm friday. You can even have it run the a/c before you leave to get the interior all ready.
  • The android app is… functional, but has a lot of quirks. But you can lock/unlock, start the car (and turn on a/c), see range for hv/ev, plugged in or not, charging or not, how long until charged. It also sends you notices anytime the doors lock/unlock, charging starts, etc. It’s really chatty with notices and I don’t see a way to adjust them, but thats mostly fine. It also has a little ‘Drive pulse and trips” thing where it tells you about your trips (about a day after you made them). You can opt out of this, but I am leaving it on for now. This all requires a subsription of course, but new cars come with 1 year included. I guess I’ll see in a year if I want to renew.
  • There is of course a ODBII port and… wow… theres a lot of data there. I will have to play around with it and see if any of it is useful to collect.
  • Panoramic moon roof is awesome. Lets in tons of light and looks really nice.

A few annoyances / things to sort out:

  • When you stop and open the drivers door it gives you a cool little report about your trip. time, mpg or KWh/mile, miles, etc. If you have a charging schedule set it shows you a thing about that and asks if you want to charge now, then shows the trip report and then… it disappears. I see no way to get it back, so it’s anoying that you can’t look back at your last trip and see the stats.
  • Similarly for mpg/kwhpm you can get a graph that shows your ‘best mpg’ and bars for your trips, but mpg is pretty worthless when you are in ev mode (it shows 99.9mpg), and there’s no way to see more details about any of the trips. (at least from the infotainment/display).
  • The “guess-o-meter” as other folks call the millage estimates is pretty off for ev mode especially. I went 26.4 miles and used 36% of battery, but 100% full the guess-o-meter said I had 33miles of range. I understand that it has to learn/calibrate. Will be interesting to see how well it adjusts in the next month or so.
  • As folks noted on the net, the pedestrian warning noise at low speeds in ev mode is pretty weird. Many people call it ‘space ship like’. Moving forward it’s just weird, but not too anoying, but backing up… wow. it’s really quite loud. Will have to see if I get used to it, or want to do something about it.
  • Amusingly, insurance was about the same as the 2013 gas rav4 we traded in on this one. I guess because all the crazy safety features.

Not yet tested / need to look into it more:

  • Will be interesting to drive at night. I think the HUD will look really sharp and nice at night. Will have to see how the headlights do.
  • Also living in the pacific northwest, will be interesting to see how the wipers do, and how it handles in rain. The OEM tires are… not great, but I am planning on just using them until they wear out.
  • Longer trips in HV at higher speeds will be interesting. I am sure millage will be less (hopefully around the 36-40mpg thats advertised). Still will likely double the mpg we would get in our other car.
  • Didn’t play around with satellite radio at all. I’m happy with my music/podcasts
  • Didn’t play around yet with the wifi hotspot. This service only has a 1 month trial included, so I’ll have to decide if it’s worth getting. My phone does just fine as a hotspot most of the time, but it might be nice to have just because it’s a different provider (AT&T) and will have different coverage.
  • I avoided the dealer extended warentee and ceramic coating sell. I might get these things down the road, but I think I can do cheaper/better than the dealer on them.
  • Speaking of, considering a ceramic wax, but there’s a ton of choices out there.
  • It came with roof cross bars, which I had the dealer take off. We don’t usually carry things on the roof, and we can always put them back on if we do.
  • I’d like to try a trip sitting in the back. It looks really quite nice, but would be interesting to see how comfortable it is. So far I have always been in the front.

Anyhow, thats enough rambling. Overall I am very happy and enjoying playing around with it.

Comments Off on That new car smell

Tracking down a bug

by on 2023/03/01 at 6:51 pm
Posted In: fedora, linux

I thought I’d share a fun hour or so of my afternoon and how I approached tracking down a bug in the Fedora Xfce spin.

The bug is https://bugzilla.redhat.com/show_bug.cgi?id=2170682 “Updates cause Xfce to lose all menu icons”. A curious bug, on updating to the latest version icons in menus no longer were enabled (although the user didn’t disable them and they are enabled by default).

Since the report happily had the update where it started, I went and looked first upstream (which turned out to be a mistake, but you never know). I looked through the recent commits for xfce4-settings to see if anything stood out as possibly being related to this. Nothing really did. All the changes seemed minor and unrelated.

Next I grabbed the latest Fedora 38 prebeta/branched Xfce live media and fired it up in a vm. Sure enough, the icons were not there at all. I checked the live users xsettings.xml vs the default one and they both seemed fine from a quick glance. Then, I saved off the user xsettings.xml file and reset the menu icons option. Sure enough, that one option was added, but the file was otherwise completely the same.

Next I checked the user journal. Bingo. There is a message saying the system xml file is broken/not parsable. Thats the problem! So where does the offending line come from? It’s from the patch we have to set the default theme… the xml wasn’t formed fully correctly and thus xfsettingsd just gave up reading it and never got to the default for menu icons. 🙂

Knowing when something broke really helps you narrow down what to look at for the problem. In some ideal world I would have checked the user journal for errors first, but I did get there eventually.

So if you hit a bug: try and figure out when it started, use that to bound your search for _what changed_ and that will a lot of the time find the culprit.

Comments Off on Tracking down a bug

error: rpmdbNextIterator: skipping in Fedora 38+

by on 2023/01/31 at 2:50 pm
Posted In: fedora, linux
I've seen this question enough times recently to decide to just write up a blog post on it and point people here. :)

If you are running Fedora 38 (currently rawhide, but will be branching off soon) and you are getting errors like this from dnf and/or rpm:

Running transaction check
error: rpmdbNextIterator: skipping h#   47749 
Header V4 DSA/SHA1 Signature, key ID 7fac5991: BAD
Header SHA256 digest: OK
Header SHA1 digest: OK
error: rpmdbNextIterator: skipping h#   47749 
Header V4 DSA/SHA1 Signature, key ID 7fac5991: BAD
Header SHA256 digest: OK
Header SHA1 digest: OK

This post is for you. What is going on here? And how can you get around the problem?

Well, what happened is that rpm used to have internal code to handle signatures on packages. This code was really something rpm didn’t want to have to maintain, so they switched recently to using sequoia, which is a new gnupg handling project written in rust. With this switch, sequoia actually honors the site wide fedora crypto policy (which the internal old rpm version did not).

Back in Fedora 33, the distro wide crypto policy was updated to disallow SHA-1 as a signature algorithm. See https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 for more information.

You might wonder why, if Fedora changed the distro wide crypto policy to disallow SHA-1 in signatures, why didn’t they update things so nothing used SHA-1 now? Well, the short answer is: they did. No rpms that Fedora produces now use SHA-1 signatures. However, some third-party rpms do. One of the big ones that many people are hitting is google’s “chrome” web browser. There’s probably others.

Now that we know _why_ this is happening, what can you do? Well, first you need to do something so dnf and rpm allows you to remove/update/change your package set. You can do this by (temporarily!) allowing SHA-1 Signatures with: 

sudo update-crypto-policies --set DEFAULT:SHA1

rpm and dnf should now work for you again. You might remove packages that have SHA-1 signatures and switch to alternatives. Or wait until google updates their signing key (the current one is from 2007). Once you have done what you need to do you can set the policy back to it’s sane default:

sudo update-crypto-policies --set DEFAULT
Comments Off on error: rpmdbNextIterator: skipping in Fedora 38+