July 2018 - kata-dev - lists.katacontainers.io

Denver meeting: Decision by Aug 7
by Anne Bertucio 08 Aug '18

08 Aug '18

Hi all, The Denver in-person teams meeting [1] will be here before we know it (Sept 11-13), and I’ve been keeping an eye on the etherpad to make sure we have enough active contributors present. The Denver meeting is a time for our community to have dedicated space in a focused environment—no conference distractions or trying to create ad hoc space. The next opportunity to have in-person time like this is the OpenStack Summit Berlin, although knowing that it overlaps with KubeCon Shanghai, our community might be split in two that week. The Denver meeting will only be valuable if we have enough active contributors there to push things forward, and to be considerate of travel plans, I’d like to propose that we decide to commit to meeting in Denver by August 7, or let the space go to another team. If you have confirmed your travel approval, please add your name to the etherpad before August 7: https://etherpad.openstack.org/p/KataContainersPTG4 <https://etherpad.openstack.org/p/KataContainersPTG4> [1] https://www.openstack.org/ptg <https://www.openstack.org/ptg> Anne Bertucio OpenStack Foundation anne(a)openstack.org | irc: annabelleB

1 1

Kata metrics deck for Monday 2017-07-30
by Graham Whaley 03 Aug '18

03 Aug '18

Hi folks. I'm lined up to present the current state and plans of the metrics tests and systems to the Kata Architecture Committee call on Monday. You can preview the deck on gdrive at the link https://docs.google.com/presentation/d/1eKLQX2g_4fRdM0QLzP06p16WcuEYWWrKcA5… Let me know if you cannot see that or would like a pdf mailing out etc. And you can fire questions at me on the list, or keep them ready for Monday. Graham

2 1

Improving isolation in Kata
by Stefan Hajnoczi 24 Jul '18

24 Jul '18

Kata currently launches QEMU as user 'root' and I haven't found isolation mechanisms that would confine the QEMU process in the event of a guest escape. Inevitably there will be a security bug that allows a malicious guest to escape to QEMU, so it's important that the QEMU process is unprivileged and isolated. Running QEMU as 'root' simplifies 9p configuration by allowing the QEMU process to access files with any uid/gid. QEMU does support other 9p configurations where the QEMU process does not run as root, including the virtfs-proxy-helper privilege separation approach. Are there any other reasons for running the QEMU process as root? For comparison, libvirt launches QEMU as the unprivileged 'qemu' user with a unique SELinux context. This makes it much harder to access other guests or escape to the host if the guest compromises QEMU. Is there any work underway to improve isolation in Kata? Two directions that come to mind are: 1. Design isolation from scratch for Kata, including unprivileged uid/gid, namespaces, and SELinux. This requires changes to Kata because the QEMU process will not be able to open arbitrary files (e.g. device hotplug, 9p configuration). FWIW, relying on namespaces doesn't add a lot of security because if the guest is malicious, then that may well be because of a Linux namespaces security hole - the attacker could just use the same exploit again to escape from QEMU to the host. 2. Use libvirt-go (API) or libvirt-go-xml (XML) instead of govmm to get strong isolation. This would also slim down virtcontainers and remove the dependency on govmm (no more command-line munging and defining object models for QEMU command-line concepts). Stefan

5 15

[kata-release] release cadence, process
by Ernst, Eric 20 Jul '18

20 Jul '18

Carlos et al, I am proposing that we continue to roll releases; we are having a good amount of fixes merge and I think it warrants having a regular weekly cadence (likely a 0.0.x update). I believe we discussed this before – let’s start driving it. Carlos, Raju, Akshay – from prior emails/request for volunteers, you were all listed for folks to help with packaging and release. Carlos, is the process defined enough that you think we can start rotating the release duties among more stakeholders? WDYT? Thx, Eric

7 7

Re: [kata-dev] [Qemu-devel] [PATCH] migration: add capability to bypass the shared memory
by Peng Tao 18 Jul '18

18 Jul '18

On Tue, Jul 3, 2018 at 6:05 PM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: > On Mon, Jul 02, 2018 at 09:52:08PM +0800, Peng Tao wrote: >> On Mon, Jul 2, 2018 at 9:10 PM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: >> > On Sat, Mar 31, 2018 at 04:45:00PM +0800, Lai Jiangshan wrote: >> > Risks: >> > 1. If one cloned VM is exploited then all other VMs are more likely to >> > be exploitable (e.g. kernel address space layout randomization). >> w.r.t. KASLR, any memory duplication technology would expose it. I >> remember there are CVEs (e.g., CVE-2015-2877) specific to this kind >> attack against KSM and it was stated that "Basically if you care about >> this attack vector, disable deduplication.". Share-until-written >> approaches for memory conservation among mutually untrusting tenants >> are inherently detectable for information disclosure, and can be >> classified as potentially misunderstood behaviors rather than >> vulnerabilities. [1] >> >> I think the same applies to vm templating as well. Actually VM >> templating is more useful (than KSM) in this regard since we can >> create a template for each trusted tenant where as with KSM all VMs on >> a host are treated equally. >> >> [1] https://access.redhat.com/security/cve/cve-2015-2877 > > That solves the problem between untrusted users but a breach in one > clone may reveal secrets of all other clones belonging to the same > tenant. As a user, I would be uncomfortable knowing that if one of my > machines is breached then secrets used by all of my machines might be > exposed. > Secrets are really point 2 in your list and I'll answer it below :) >> > 2. If you give VMs cloned from the same template to untrusted users, >> > they may be able to determine the secrets other users' VMs. >> In kata and runv, vm templating is used carefully so that we do not >> use or save any secret keys before creating the template VM. IOW, the >> feature is not supposed to be used generally to create any template >> VMs at any stage. > > At what point are templates captured to avoid these problems? Is there > code that shows how to do this? > Both runv and kata pauses the VM right after the agent inside guest is up and running, which, in the initramfs case, translates into the point that kernel boots and the init process starts. If you are interested in seeing the actual code, you can look at https://github.com/hyperhq/hyperstart/ and https://github.com/kata-containers/agent for what is done in the guest at that point. If you see any secrets being saved there, I'll be more than happy to fix it. :) >> > Security is a >> > major factor for using Kata, so it's important not to leak secrets >> > between cloned VMs. >> > >> Yes, indeed! And it is all about trade-offs, VM templating or KSM. If >> we want security above anything, we should just disable all the >> sharing. But there is actually no ceiling (think about physical >> isolation!). So it's more about trade-offs. With Kata, VM templating >> and KSM give users options to achieve better performance and lower >> memory footprint with little sacrifice. The security advantage of >> running VM-based containers is still there. > > Adding options to enable/disable features leads to confusion among > users, makes performance comparisons harder, and increases support > overhead. > > Technical solutions to the security problems are possible. I'm > interested in progress in this area because it means users don't need to > make a choice, they can benefit from the feature without sacrificing > security. > Well, that is really beyond the scope of the reviewing of this particular QEMU patch. But as a Kata developer, I can answer it anyway. For one thing, Kata already has quite a few configuration options that let users choose different features. For another thing, Kata already ships with KSM support by default and VM templating in Kata is better off than KSM in many aspects (e.g., by providing similar level of memory conservation w/o affecting host and guest performance). So from Kata containers point of view, it makes sense to have VM templating support and let users decide which one they want to use. PS: CCing kata-dev since the discussion starts to concern about Kata specific usage. Cheers, Tao

2 3

ARM CI status for Kata container
by Kaly Xin 17 Jul '18

17 Jul '18

Hi, These days we are working on the ARM CI setting up for Kata container project. For hardware, thanks for Hyper friends who applied for a ThunderX2 server on packet.net WorksOnArm project. And we plan to utilize this machine as a Jenkins slave to be scheduled by Jenkins master with CI tasks on ARM. For software preparation, we are going through the scripts under https://github.com/kata-containers/tests/tree/master/.ci to setup the testing environment. We met some issues in environment setting up for ARM. We resolved the issues and the PR were merged. See https://github.com/kata-containers/tests/issues/472 for example. Currently, the environment setting up is successful. Now we are going through the integration test cases under https://github.com/kata-containers/tests/tree/master/integration on our internal server. Unfortunately, most of the cases fail. We are trying to figure out what happens now. We believe that it might not be a short term development to pass most of the test cases. It will lag the process a bit longer. The reasonable solution could be setting up the CI infrastructure on ARM first - just to add one ARM server as a Jenkins slave which can be scheduled by Jenkins master. And it can be configured to run a limited number of successful tasks as Jenkins jobs in initial stage. At this step, we need the help from the CI infrastructure team - at least, both the master and slave have to know the IP/hostname, slave has to know the plugin to be installed and the credential things, etc. After the infrastructure is setting up, we could add more Jenkins jobs in. But that almost means different architecture could have different Jenkins job to test. To make the process happen, it's better to have some kind of mechanism (indicated by @Xu Wang<mailto:xu@hyper.sh> as "filter") to define and generate the set of Jenkins jobs running by different architectures. We have a proposal for this and will raise an issue to discuss it in community very soon. We plan to have Zuul support as the further step after above is working. Thanks Kaly IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0

Kata Mission Statement
by Anne Bertucio 16 Jul '18

16 Jul '18

Hi all, On the Architecture Committee call today, I brought up the creation of a mission statement for Kata Containers. The idea of a mission statement is to be a succinct expression of what this project aims to do and any critical principles driving our work. Process: The Zuul community found that this process[1] worked quite well for them. The TL;DR: 1. Generate ideas and share feedback for set time period (they did one week) (open GitHub issue here[2]) 2. The [Kata Arch Committee] and Foundation team have a veto on suggestions (This is to lean on our community leadership and the Foundation marketing team for their expertise in avoiding statements that might come back to bite us later, etc) 3. Run a vote for the final choice if there isn’t a clear winning amongst the community Thoughts? Ideas? [1] http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-May/000394.html <http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-May/000394.html> [2] https://github.com/kata-containers/community/issues/47 <https://github.com/kata-containers/community/issues/47> Anne Bertucio OpenStack Foundation anne(a)openstack.org | irc: annabelleB

1 0

OpenStack Summit Berlin CFP closes in one week: Submit your Kata talks!
by Anne Bertucio 13 Jul '18

13 Jul '18

Hi all, See the note below from Ashlee at OSF about the OpenStack Summit Berlin CFP, which closes in one week. There was lots of good Kata Containers content in the Container Infrastructure track in Vancouver and it’d be great to see that in Berlin! If you gave a Kata talk in Vancouver, it’s okay to submit the same for Berlin. There will be lots of people in Berlin who weren’t in Vancouver and will want to hear about Kata (but also bring your new ideas and topics!). Please let me know if you have any questions about the Summit CFP process or would like someone to look over your CFP before submission. Cheers, Anne ---------------------------------------- Hi everyone, The CFP deadline for the OpenStack Summit Berlin is less than one week away, so make sure to submit your talks <https://www.openstack.org/summit/berlin-2018/call-for-presentations/> before July 18th at 6:59am UTC (July 17 at 11:59pm PST). Tracks: • CI/CD • Container Infrastructure • Edge Computing • Hands on Workshops • HPC / GPU / AI • Open Source Community • Private & Hybrid Cloud • Public Cloud • Telecom & NFV SUBMIT HERE <https://www.openstack.org/summit/berlin-2018/call-for-presentations/> Community voting, the first step in building the Summit schedule, will open in mid July. Once community voting concludes, a Programming Committee for each Track will build the schedule. Programming Committees are made up of individuals from many different open source communities working in open infrastructure, in addition to people who have participated in the past. Read the full selection process here <https://www.openstack.org/summit/berlin-2018/call-for-presentations/selecti…>. Register for the Summit <https://www.eventbrite.com/e/openstack-summit-november-2018-berlin-tickets-…> - Early Bird pricing ends August 21 Become a Sponsor <https://www.openstack.org/summit/berlin-2018/sponsors/> Cheers, Ashlee Ashlee Ferguson OpenStack Foundation ashlee(a)openstack.org <mailto:ashlee@openstack.org>

2 1

Kata on Fedora experiment (modular kernel)
by Stefan Hajnoczi 11 Jul '18

11 Jul '18

Hi, I wanted to share some thoughts after running Kata on Fedora. I built Kata from source and used the Fedora rootfs together with a vanilla modular kernel. This configuration is similar to what I imagine official Fedora RPMs would eventually use, because they probably want to use a stock Fedora kernel instead of building a special monolithic kernel for Kata. The Fedora rootfs was built like this: # KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora My understanding is Kata wants *either* an initramfs or a disk image. Modular kernels typically need to load drivers before they can access the disk, so I opted for the initramfs even though it is several hundred megabytes. The initramfs could probably be shrunk significantly with some work on the Fedora rootfs script. This configuration required several tweaks to get Kata running. I'm not going to create a patch series or send a pull request, because at this stage these are dirty hacks that cannot be merged. But it's likely that other people will encounter these issues, so I'm including them below. Stefan --- * kata-agent assumes initramfs is equivalent to running the agent as init. In our case kata-agent *is* in an initramfs but *not* init, so it calculates the wrong value for noPivotRoot: diff --git a/agent.go b/agent.go index ec01250..182c2de 100644 --- a/agent.go +++ b/agent.go @@ -830,7 +832,7 @@ func realMain() { running: false, // pivot_root won't work for init, see // Documention/filesystem/ramfs-rootfs-initramfs.txt - noPivotRoot: os.Getpid() == 1, + noPivotRoot: true, // TODO check initramfs, not pid=1! kata-agent might have been started by systemd form an initramfs os.Getpid() == 1, subreaper: r, pciDeviceMap: make(map[string]string), deviceWatchers: make(map[string](chan string)), * kata-agent assumes AF_VSOCK should be used if the guest kernel has vsock support. A generic kernel like the Fedora kernel will have AF_VSOCK but that doesn't mean the virtio-serial device should be ignored. Perhaps change the check to see if /dev/virtio-ports/agent.channel.0 is present? diff --git a/channel.go b/channel.go index b4f8977..34ef292 100644 --- a/channel.go +++ b/channel.go @@ -29,25 +29,29 @@ type channel interface { } func newChannel() (channel, error) { + // This is broken, it ignores the virtio-serial agent.channel.0 device + // in favor of AF_VSOCK support, even on systems that are not going to + // use AF_VSOCK but happen to have the drivers installed. + // Check for vsock support. - vSockSupported, err := isAFVSockSupported() - if err != nil { - return nil, err - } - - if vSockSupported { - // Check if vsock socket exists. We want to cover the case - // where the guest OS can support vsock, but the runtime is - // still using virtio serial connection. - exist, err := vSockPathExist() - if err != nil { - return nil, err - } - - if exist { - return &vSockChannel{}, nil - } - } +// vSockSupported, err := isAFVSockSupported() +// if err != nil { +// return nil, err +// } +// +// if vSockSupported { +// // Check if vsock socket exists. We want to cover the case +// // where the guest OS can support vsock, but the runtime is +// // still using virtio serial connection. +// exist, err := vSockPathExist() +// if err != nil { +// return nil, err +// } +// +// if exist { +// return &vSockChannel{}, nil +// } +// } return &serialChannel{}, nil } * The Fedora initramfs and modular kernel rely on udev to load the virtio_console.ko driver. Make sure to start kata-agent *after* the agent device appears. Note this change will break AF_VSOCK if you choose to use that instead of virtio-serial! diff --git a/kata-agent.service.in b/kata-agent.service.in index 861757d..5395047 100644 --- a/kata-agent.service.in +++ b/kata-agent.service.in @@ -2,6 +2,8 @@ Description=Kata Containers Agent Documentation=https://github.com/kata-containers/agent Wants=kata-containers.target +BindsTo=dev-virtio\x2dports-agent.channel.0.device +After=dev-virtio\x2dports-agent.channel.0.device [Service] # Send agent output to tty to allow capture debug logs * Make sure the rootfs has modprobe and udev. Needed by the modular kernel. diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh index a0ca15a..19fbc7b 100644 --- a/rootfs-builder/fedora/config.sh +++ b/rootfs-builder/fedora/config.sh @@ -14,4 +14,4 @@ PACKAGES="iptables" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true

6 8

FYI: containerd shim API v2
by Tim Allclair 06 Jul '18

06 Jul '18

There is a proposal out in the containerd community for a v2 shim API. This API is specifically targeted at enabling VM-based runtimes, so I wanted to make sure folks in the Kata community have a chance to review it and ensure it is a good (compatible) match for the current Kata runtime APIs. https://github.com/containerd/containerd/issues/2426 Thanks!

3 3