Guest kernel and CONFIG_PCI_MMCONFIG
Hello all, I recently discovered that the kernel config generated by tools/packaging/kernel has CONFIG_PCI_MMCONFIG disabled. That's probably not a good idea, since that's a very atypical config for modern systems, and for complicated reasons it completely stops SHPC hotplug from working. So, I've filed https://github.com/kata-containers/kata-containers/pull/2311 to change it. But.. I have reason to suspect that anyone using Kata's VFIO support is actually *relying* on this bogus guest behaviour to make things work. In all my experiements when I try to use VFIO devices with a more normal kernel, the device is detected in the guest after the agent's rescan and works, but only briefly. When the SHPC hotplug completes 5s later, this interferes somehow and puts the device in a badly broken state (config space accesses fail completely)[0]. To make things harder, seeing how this affects the CI is also non-trivial: 1) Changing the config in the kata-containers repo will not (AFAICT) trigger a rebuild of the cached guest kernel in the CI, and so it will be tested effectively without the change. 2) The vfio CI testcase is pretty rudimentary. I think it's quite likely it could complete fast enough to succeed and shut down before the SHPC hotplug completes and breaks the device At this point I'm inclined to just push the change anyway and deal with the fallout as and when it occurs. But does any one have a better idea? [0] In fact, if disable_idle_d3=N for the vfio-pci module, it even puts the device into a broken state in the *host*, requiring a host reboot to restore. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On Mon, Jul 26, 2021 at 07:44:34PM +1000, David Gibson wrote:
Hello all,
I recently discovered that the kernel config generated by tools/packaging/kernel has CONFIG_PCI_MMCONFIG disabled. That's probably not a good idea, since that's a very atypical config for modern systems, and for complicated reasons it completely stops SHPC hotplug from working.
So, I've filed https://github.com/kata-containers/kata-containers/pull/2311 to change it.
But..
I have reason to suspect that anyone using Kata's VFIO support is actually *relying* on this bogus guest behaviour to make things work. In all my experiements when I try to use VFIO devices with a more normal kernel, the device is detected in the guest after the agent's rescan and works, but only briefly. When the SHPC hotplug completes 5s later, this interferes somehow and puts the device in a badly broken state (config space accesses fail completely)[0].
To make things harder, seeing how this affects the CI is also non-trivial: 1) Changing the config in the kata-containers repo will not (AFAICT) trigger a rebuild of the cached guest kernel in the CI, and so it will be tested effectively without the change. 2) The vfio CI testcase is pretty rudimentary. I think it's quite likely it could complete fast enough to succeed and shut down before the SHPC hotplug completes and breaks the device
So, I realized I was misunderstanding how the config_version stuff works. Once I fix that up, I can trigger a guest kernel rebuild easily enough. .. and it does indeed fail, specifically when it tries to shut down the Kata VM, which is interesting.
At this point I'm inclined to just push the change anyway and deal with the fallout as and when it occurs. But does any one have a better idea?
[0] In fact, if disable_idle_d3=N for the vfio-pci module, it even puts the device into a broken state in the *host*, requiring a host reboot to restore.
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On 26 Jul 2021, at 11:44, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
Hello all,
I recently discovered that the kernel config generated by tools/packaging/kernel has CONFIG_PCI_MMCONFIG disabled.
I'm curious how much difference it makes in memory size (if at all). I suspect things that were configured out were most likely selected to reduce the kernel size more than for any other reason.
That's probably not a good idea, since that's a very atypical config for modern systems, and for complicated reasons it completely stops SHPC hotplug from working.
That's weird. Shouldn't that be abstracted behind pci_read_config_byte() and the like? Now you make me curious why this breaks SHPC hotplug. If you have the time to explain ;-) Does this seem like a kernel bug or some hardware / architectural limitation?
So, I've filed https://github.com/kata-containers/kata-containers/pull/2311 to change it.
But..
I have reason to suspect that anyone using Kata's VFIO support is actually *relying* on this bogus guest behaviour to make things work. In all my experiements when I try to use VFIO devices with a more normal kernel, the device is detected in the guest after the agent's rescan and works, but only briefly. When the SHPC hotplug completes 5s later, this interferes somehow and puts the device in a badly broken state (config space accesses fail completely)[0].
To make things harder, seeing how this affects the CI is also non-trivial: 1) Changing the config in the kata-containers repo will not (AFAICT) trigger a rebuild of the cached guest kernel in the CI, and so it will be tested effectively without the change. 2) The vfio CI testcase is pretty rudimentary. I think it's quite likely it could complete fast enough to succeed and shut down before the SHPC hotplug completes and breaks the device
At this point I'm inclined to just push the change anyway and deal with the fallout as and when it occurs. But does any one have a better idea?
[0] In fact, if disable_idle_d3=N for the vfio-pci module, it even puts the device into a broken state in the *host*, requiring a host reboot to restore.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On Tue, Jul 27, 2021 at 11:58:56AM +0200, Christophe de Dinechin wrote:
On 26 Jul 2021, at 11:44, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
Hello all,
I recently discovered that the kernel config generated by tools/packaging/kernel has CONFIG_PCI_MMCONFIG disabled.
I'm curious how much difference it makes in memory size (if at all). I suspect things that were configured out were most likely selected to reduce the kernel size more than for any other reason.
That's probably not a good idea, since that's a very atypical config for modern systems, and for complicated reasons it completely stops SHPC hotplug from working.
That's weird. Shouldn't that be abstracted behind pci_read_config_byte() and the like? Now you make me curious why this breaks SHPC hotplug. If you have the time to explain ;-) Does this seem like a kernel bug or some hardware / architectural limitation?
Yeah, it's not as direct as that, hence "complicated reasons". *checks notes* specifically the path is: When mmconfig is disabled, raw_pci_ext_ops is never set by pci_mmcfg_arch_init(). That makes pci_ext_cfg_avail() return 0, which makes the 'support' variable in negotiate_os_control() lack the OSC_PCI_EXT_CONFIG_SUPPORT bit - a.k.a. the ACPI_PCIE_REQ_SUPPORT bit. That makes negotiate_os_control() exit early (" _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]"). So, it never gets to acpi_pci_osc_control_set(), which would set root->osc_control_set. That means that host_bridge->native_shpc_hotplug is set to 0 later in acpi_pci_root_create(). That means that acpi_get_hp_hw_control_from_firmware() will return -ENODEV, and therefore so will shpc_probe(). I don't know enough about ACPI or PCI on x86 to know if this is a kernel bug per se. If it is, Kata's in more trouble, since if the kernel bug is fixed it will break us (since we're actually relying on SHPC not working properly). -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
participants (2)
-
Christophe de Dinechin
-
David Gibson