Living on the Edge: Rawhide day to day

Kevin Fenzi

Fedora Infrastructure Lead, Red Hat CC-BY-SA-4.0

This talk will be go over tips, tricks and workflows to use rawhide or branched as your everyday Fedora. Learn how to troubleshoot and work around issues and help make rawhide or branched more stable for everyone else.

My Background

• Who am I?

  • Running rawhide for 3+ years on my day to day laptop
  • Sysadmin by trade, I'm on my laptop a LOT
  • Been involved with Fedora for 11 years now, the last 5 as a Red Hatter
  • Maintain a bunch of packages (Xfce stack, python for infrastructure, etc)

Rawhide Background

• What is rawhide?
  • Always rolling on development version
  • New major versions can appear anytime
  • New minor versions do appear all the time
  • Not 100% signed (but we are close!)
  • A daily FULL compose of all packages/images/trees
  • Tested by OpenQA

Rawhide Goals

(direct from the wiki)

• To allow package maintainers to integrate the newest usable versions of their packages into Fedora.
• To allow advanced users access to the newest usable packages in a rolling manner.
• To allow incremental changes to packages that are either too minor or major to go to stable Fedora releases.
• To identify and fix issues with packages before they reach a stable release of Fedora.

How Rawhide is made

• Cron script from fedora-pungi runs pungi 4 at 05:15UTC
• Creates trees/multilib from f-N tag in koji
• Creates all images/spins/labs/install media, etc

Why run Rawhide?

• You like getting new software every day
• You like troubleshooting problems or issues and seeing them get fixed
• You love a challenge
• Help out those folks using stable releases, you find bugs so they do not need to

Some Rawhide Stats

In the last month:

• One day with 18 packages upgraded, another with 2487 (varies a lot)
• 12,000-25,000 mirrormanager requests a day
• Mass rebuild days almost all packages update
• First announced Aug 1998 - 18 years ago now!

Getting on the Rawhide trail

• Automated testing (OpenQA) is mostly install path
• Once you are on the trail it's easier to stay on
• Fresh installs: Finding images: https://nightly.fedoraproject.org/
• Upgrading from stable to rawhide with dnf
• Mixing rawhide and stable releases

Getting OFF the rawhide trail

• branching points (we just had one 2016-07-26)
• distro-sync to another branch
• fedora-repos-rawhide subpackage and repo file
• Re-install

A note about hardware

• Things that work for stable in general work for rawhide
• Sometimes rawhide has support for newer things
• If you use/need non free video or other packages, rawhide might not be a good choice
• If you have really old hardware you might have issues as support is dropped.

Tools: dnf

• dnf is your first go to tool
  1. downgrading
  2. history
  3. bisecting when a problem appeared
  4. broken deps / --best
  5. dnf repoquery --unsatisfied ( yum check )
  6. excluding/pinning

Tools: koji

• Signed vs Unsigned packages
• Downloading large collections of packages (koji download-build)
• Grabbing things before they finish building
• Finding older working packages
• Scratch builds to test fixes

Tools: selinux

• enforcing=0
• dontaudit rules

Tools: abrt and setroubleshootd

• Good for simple problems
• Can downgrade after report filed

Tools: Gathering info

• Changelog / NEWS / git
• dnf logs
• Rawhide reports

Tools: Bugs and reporting

• The packages bug pagehttps://apps.fedoraproject.org/packages/<name>/bugs/all
• Look in changelog for related bugs

Tools: mailing lists / IRC

• Rawhide reports
• Reports of serious issues
• Place to ask if you aren't sure where breakage is
• #fedora-devel, #fedora-qa

Strategy: Update and Reboot cadience

• Daily updates ideal
• Reboots everytime there is a new kernel
• Have a usb/rescue media handy just in case

Strategy: alternatives

• Have and know alternatives
• Have alternate desktops/browsers/etc
• Alternatives help you troubleshoot
• Testing with a new user

Strategy: Backups

• Backups: as important as ever
• snapshots: good in pretty limited cases

Branched

• 'bodhi -D' to download updates to test
• updates-testing enabled by default, can use base repo to downgrade
• fedora-easy-karma to help moving things along

Every further on the edge

• koji buildroot as repo
  • No multilib
  • Not signed
  • Could be untagged
• Packages from koji before they finish

The Future!

• Gating rawhide (install path)
• Gating rawhide (compose path)
• More automated testing
• Multiple versions?
• Taskotron tests to gate composes

Questions and Discussion

Rawhide is awesome!

Lets all make it MORE awesome