On 26 Jul 2021, at 11:44, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
Hello all,
I recently discovered that the kernel config generated by tools/packaging/kernel has CONFIG_PCI_MMCONFIG disabled.
I'm curious how much difference it makes in memory size (if at all). I suspect things that were configured out were most likely selected to reduce the kernel size more than for any other reason.
That's probably not a good idea, since that's a very atypical config for modern systems, and for complicated reasons it completely stops SHPC hotplug from working.
That's weird. Shouldn't that be abstracted behind pci_read_config_byte() and the like? Now you make me curious why this breaks SHPC hotplug. If you have the time to explain ;-) Does this seem like a kernel bug or some hardware / architectural limitation?
So, I've filed https://github.com/kata-containers/kata-containers/pull/2311 to change it.
But..
I have reason to suspect that anyone using Kata's VFIO support is actually *relying* on this bogus guest behaviour to make things work. In all my experiements when I try to use VFIO devices with a more normal kernel, the device is detected in the guest after the agent's rescan and works, but only briefly. When the SHPC hotplug completes 5s later, this interferes somehow and puts the device in a badly broken state (config space accesses fail completely)[0].
To make things harder, seeing how this affects the CI is also non-trivial: 1) Changing the config in the kata-containers repo will not (AFAICT) trigger a rebuild of the cached guest kernel in the CI, and so it will be tested effectively without the change. 2) The vfio CI testcase is pretty rudimentary. I think it's quite likely it could complete fast enough to succeed and shut down before the SHPC hotplug completes and breaks the device
At this point I'm inclined to just push the change anyway and deal with the fallout as and when it occurs. But does any one have a better idea?
[0] In fact, if disable_idle_d3=N for the vfio-pci module, it even puts the device into a broken state in the *host*, requiring a host reboot to restore.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev