Hi Stefan,
Thanks for sharing your experiments. Your approach to integrate Kata shows some points improve to support this use case.
I did not consider in the past use the host kernel as the kata kernel but definitely is a use case we could support.
A couple of questions related on how to mange the kernel and image when you add Fedora RPMs.
Are you going to add a symlink to allow the kata runtime always find the latest installed kata kernel ?
How are you planning to update the modules if a kernel changes ? The image will by generated after a new kernel update? Or every time the kernel changes you are going to update the kernel from package?
On Fri, Jun 29, 2018 at 06:42:27PM +0100, Stefan Hajnoczi wrote:
Hi, I wanted to share some thoughts after running Kata on Fedora. I built Kata from source and used the Fedora rootfs together with a vanilla modular kernel. This configuration is similar to what I imagine official Fedora RPMs would eventually use, because they probably want to use a stock Fedora kernel instead of building a special monolithic kernel for Kata.
The Fedora rootfs was built like this:
# KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora
My understanding is Kata wants *either* an initramfs or a disk image. Modular kernels typically need to load drivers before they can access the disk, so I opted for the initramfs even though it is several hundred megabytes. The initramfs could probably be shrunk significantly with some work on the Fedora rootfs script.
This configuration required several tweaks to get Kata running. I'm not going to create a patch series or send a pull request, because at this stage these are dirty hacks that cannot be merged. But it's likely that other people will encounter these issues, so I'm including them below.
Stefan
- kata-agent assumes initramfs is equivalent to running the agent as init. In our case kata-agent *is* in an initramfs but *not* init, so it calculates the wrong value for noPivotRoot:
diff --git a/agent.go b/agent.go index ec01250..182c2de 100644 --- a/agent.go +++ b/agent.go @@ -830,7 +832,7 @@ func realMain() { running: false, // pivot_root won't work for init, see // Documention/filesystem/ramfs-rootfs-initramfs.txt
noPivotRoot: os.Getpid() == 1,
noPivotRoot: true, // TODO check initramfs, not pid=1! kata-agent might have been started by systemd form an initramfs os.Getpid() == 1, subreaper: r, pciDeviceMap: make(map[string]string), deviceWatchers: make(map[string](chan string)),
- kata-agent assumes AF_VSOCK should be used if the guest kernel has vsock support. A generic kernel like the Fedora kernel will have AF_VSOCK but that doesn't mean the virtio-serial device should be ignored. Perhaps change the check to see if /dev/virtio-ports/agent.channel.0 is present?
diff --git a/channel.go b/channel.go index b4f8977..34ef292 100644 --- a/channel.go +++ b/channel.go @@ -29,25 +29,29 @@ type channel interface { }
func newChannel() (channel, error) {
// This is broken, it ignores the virtio-serial agent.channel.0 device
// in favor of AF_VSOCK support, even on systems that are not going to
// use AF_VSOCK but happen to have the drivers installed.
// Check for vsock support.
vSockSupported, err := isAFVSockSupported()
if err != nil {
return nil, err
}
if vSockSupported {
// Check if vsock socket exists. We want to cover the case
// where the guest OS can support vsock, but the runtime is
// still using virtio serial connection.
exist, err := vSockPathExist()
if err != nil {
return nil, err
}
if exist {
return &vSockChannel{}, nil
}
}
+// vSockSupported, err := isAFVSockSupported() +// if err != nil { +// return nil, err +// } +// +// if vSockSupported { +// // Check if vsock socket exists. We want to cover the case +// // where the guest OS can support vsock, but the runtime is +// // still using virtio serial connection. +// exist, err := vSockPathExist() +// if err != nil { +// return nil, err +// } +// +// if exist { +// return &vSockChannel{}, nil +// } +// }
return &serialChannel{}, nil
}
- The Fedora initramfs and modular kernel rely on udev to load the virtio_console.ko driver. Make sure to start kata-agent *after* the agent device appears. Note this change will break AF_VSOCK if you choose to use that instead of virtio-serial!
diff --git a/kata-agent.service.in b/kata-agent.service.in index 861757d..5395047 100644 --- a/kata-agent.service.in +++ b/kata-agent.service.in @@ -2,6 +2,8 @@ Description=Kata Containers Agent Documentation=https://github.com/kata-containers/agent Wants=kata-containers.target +BindsTo=dev-virtio\x2dports-agent.channel.0.device +After=dev-virtio\x2dports-agent.channel.0.device
[Service] # Send agent output to tty to allow capture debug logs
- Make sure the rootfs has modprobe and udev. Needed by the modular kernel.
diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh index a0ca15a..19fbc7b 100644 --- a/rootfs-builder/fedora/config.sh +++ b/rootfs-builder/fedora/config.sh @@ -14,4 +14,4 @@ PACKAGES="iptables" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true
kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev