Hi Stefan, Nice experiment. I notice that you got some trouble with the vsock code, thanks for reporting this. We're planning to add a proper support of vsock pretty soon because it's clearly not there yet. Thanks, Sebastien ________________________________________ From: Stefan Hajnoczi [stefanha@redhat.com] Sent: Friday, June 29, 2018 10:42 AM To: kata-dev@lists.katacontainers.io Cc: Lokesh Mandvekar Subject: [kata-dev] Kata on Fedora experiment (modular kernel) Hi, I wanted to share some thoughts after running Kata on Fedora. I built Kata from source and used the Fedora rootfs together with a vanilla modular kernel. This configuration is similar to what I imagine official Fedora RPMs would eventually use, because they probably want to use a stock Fedora kernel instead of building a special monolithic kernel for Kata. The Fedora rootfs was built like this: # KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora My understanding is Kata wants *either* an initramfs or a disk image. Modular kernels typically need to load drivers before they can access the disk, so I opted for the initramfs even though it is several hundred megabytes. The initramfs could probably be shrunk significantly with some work on the Fedora rootfs script. This configuration required several tweaks to get Kata running. I'm not going to create a patch series or send a pull request, because at this stage these are dirty hacks that cannot be merged. But it's likely that other people will encounter these issues, so I'm including them below. Stefan --- * kata-agent assumes initramfs is equivalent to running the agent as init. In our case kata-agent *is* in an initramfs but *not* init, so it calculates the wrong value for noPivotRoot: diff --git a/agent.go b/agent.go index ec01250..182c2de 100644 --- a/agent.go +++ b/agent.go @@ -830,7 +832,7 @@ func realMain() { running: false, // pivot_root won't work for init, see // Documention/filesystem/ramfs-rootfs-initramfs.txt - noPivotRoot: os.Getpid() == 1, + noPivotRoot: true, // TODO check initramfs, not pid=1! kata-agent might have been started by systemd form an initramfs os.Getpid() == 1, subreaper: r, pciDeviceMap: make(map[string]string), deviceWatchers: make(map[string](chan string)), * kata-agent assumes AF_VSOCK should be used if the guest kernel has vsock support. A generic kernel like the Fedora kernel will have AF_VSOCK but that doesn't mean the virtio-serial device should be ignored. Perhaps change the check to see if /dev/virtio-ports/agent.channel.0 is present? diff --git a/channel.go b/channel.go index b4f8977..34ef292 100644 --- a/channel.go +++ b/channel.go @@ -29,25 +29,29 @@ type channel interface { } func newChannel() (channel, error) { + // This is broken, it ignores the virtio-serial agent.channel.0 device + // in favor of AF_VSOCK support, even on systems that are not going to + // use AF_VSOCK but happen to have the drivers installed. + // Check for vsock support. - vSockSupported, err := isAFVSockSupported() - if err != nil { - return nil, err - } - - if vSockSupported { - // Check if vsock socket exists. We want to cover the case - // where the guest OS can support vsock, but the runtime is - // still using virtio serial connection. - exist, err := vSockPathExist() - if err != nil { - return nil, err - } - - if exist { - return &vSockChannel{}, nil - } - } +// vSockSupported, err := isAFVSockSupported() +// if err != nil { +// return nil, err +// } +// +// if vSockSupported { +// // Check if vsock socket exists. We want to cover the case +// // where the guest OS can support vsock, but the runtime is +// // still using virtio serial connection. +// exist, err := vSockPathExist() +// if err != nil { +// return nil, err +// } +// +// if exist { +// return &vSockChannel{}, nil +// } +// } return &serialChannel{}, nil } * The Fedora initramfs and modular kernel rely on udev to load the virtio_console.ko driver. Make sure to start kata-agent *after* the agent device appears. Note this change will break AF_VSOCK if you choose to use that instead of virtio-serial! diff --git a/kata-agent.service.in b/kata-agent.service.in index 861757d..5395047 100644 --- a/kata-agent.service.in +++ b/kata-agent.service.in @@ -2,6 +2,8 @@ Description=Kata Containers Agent Documentation=https://github.com/kata-containers/agent Wants=kata-containers.target +BindsTo=dev-virtio\x2dports-agent.channel.0.device +After=dev-virtio\x2dports-agent.channel.0.device [Service] # Send agent output to tty to allow capture debug logs * Make sure the rootfs has modprobe and udev. Needed by the modular kernel. diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh index a0ca15a..19fbc7b 100644 --- a/rootfs-builder/fedora/config.sh +++ b/rootfs-builder/fedora/config.sh @@ -14,4 +14,4 @@ PACKAGES="iptables" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true