On Mon, Jul 26, 2021 at 07:44:34PM +1000, David Gibson wrote:
Hello all,
I recently discovered that the kernel config generated by tools/packaging/kernel has CONFIG_PCI_MMCONFIG disabled. That's probably not a good idea, since that's a very atypical config for modern systems, and for complicated reasons it completely stops SHPC hotplug from working.
So, I've filed https://github.com/kata-containers/kata-containers/pull/2311 to change it.
But..
I have reason to suspect that anyone using Kata's VFIO support is actually *relying* on this bogus guest behaviour to make things work. In all my experiements when I try to use VFIO devices with a more normal kernel, the device is detected in the guest after the agent's rescan and works, but only briefly. When the SHPC hotplug completes 5s later, this interferes somehow and puts the device in a badly broken state (config space accesses fail completely)[0].
To make things harder, seeing how this affects the CI is also non-trivial: 1) Changing the config in the kata-containers repo will not (AFAICT) trigger a rebuild of the cached guest kernel in the CI, and so it will be tested effectively without the change. 2) The vfio CI testcase is pretty rudimentary. I think it's quite likely it could complete fast enough to succeed and shut down before the SHPC hotplug completes and breaks the device
So, I realized I was misunderstanding how the config_version stuff works. Once I fix that up, I can trigger a guest kernel rebuild easily enough. .. and it does indeed fail, specifically when it tries to shut down the Kata VM, which is interesting.
At this point I'm inclined to just push the change anyway and deal with the fallout as and when it occurs. But does any one have a better idea?
[0] In fact, if disable_idle_d3=N for the vfio-pci module, it even puts the device into a broken state in the *host*, requiring a host reboot to restore.
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson