I did a bit more playing around with running Fedora with no firmware.
Josh Boyer pointed out that the microcode_ctl package also had firmware in it. This is the ‘errata’ for cpus that fixes various bugs after they were released. Also, in my previous tests I didn’t regenerate the initramfs without the firmware being available. So, this could have meant firmware was getting loaded in initramfs before the Linux kernel booted up.
So, I removed all the firmware packages again, along with microcode_ctl and updated to the latest rawhide kernel. Amusingly I forgot that the kernel has a Requires(post) on linux-firmware, so the install pulled that in and used it when it generated the initramfs. I had to remove it again and re-run dracut to make a new initramfs to be sure it was not being used. A reboot later and… everything seemed to come up fine (again except for the iwl wireless that needed it’s firmware package to work). This time I didn’t notice any lags in X/input. So, that could have been just a bug in that previous rawhide kernel, or some interaction with firmware loading and debug kernels? Hard to say. In any case things seem to work fine here.
I’m not really going to keep running like this, as I don’t mind firmware that makes things better for me, but it’s nice to know if you can run without some or all of them if you so choose.