June 2018 - kata-dev - lists.katacontainers.io

Kata on Fedora experiment (modular kernel)
by Stefan Hajnoczi 12 Jul '18

12 Jul '18

Hi, I wanted to share some thoughts after running Kata on Fedora. I built Kata from source and used the Fedora rootfs together with a vanilla modular kernel. This configuration is similar to what I imagine official Fedora RPMs would eventually use, because they probably want to use a stock Fedora kernel instead of building a special monolithic kernel for Kata. The Fedora rootfs was built like this: # KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora My understanding is Kata wants *either* an initramfs or a disk image. Modular kernels typically need to load drivers before they can access the disk, so I opted for the initramfs even though it is several hundred megabytes. The initramfs could probably be shrunk significantly with some work on the Fedora rootfs script. This configuration required several tweaks to get Kata running. I'm not going to create a patch series or send a pull request, because at this stage these are dirty hacks that cannot be merged. But it's likely that other people will encounter these issues, so I'm including them below. Stefan --- * kata-agent assumes initramfs is equivalent to running the agent as init. In our case kata-agent *is* in an initramfs but *not* init, so it calculates the wrong value for noPivotRoot: diff --git a/agent.go b/agent.go index ec01250..182c2de 100644 --- a/agent.go +++ b/agent.go @@ -830,7 +832,7 @@ func realMain() { running: false, // pivot_root won't work for init, see // Documention/filesystem/ramfs-rootfs-initramfs.txt - noPivotRoot: os.Getpid() == 1, + noPivotRoot: true, // TODO check initramfs, not pid=1! kata-agent might have been started by systemd form an initramfs os.Getpid() == 1, subreaper: r, pciDeviceMap: make(map[string]string), deviceWatchers: make(map[string](chan string)), * kata-agent assumes AF_VSOCK should be used if the guest kernel has vsock support. A generic kernel like the Fedora kernel will have AF_VSOCK but that doesn't mean the virtio-serial device should be ignored. Perhaps change the check to see if /dev/virtio-ports/agent.channel.0 is present? diff --git a/channel.go b/channel.go index b4f8977..34ef292 100644 --- a/channel.go +++ b/channel.go @@ -29,25 +29,29 @@ type channel interface { } func newChannel() (channel, error) { + // This is broken, it ignores the virtio-serial agent.channel.0 device + // in favor of AF_VSOCK support, even on systems that are not going to + // use AF_VSOCK but happen to have the drivers installed. + // Check for vsock support. - vSockSupported, err := isAFVSockSupported() - if err != nil { - return nil, err - } - - if vSockSupported { - // Check if vsock socket exists. We want to cover the case - // where the guest OS can support vsock, but the runtime is - // still using virtio serial connection. - exist, err := vSockPathExist() - if err != nil { - return nil, err - } - - if exist { - return &vSockChannel{}, nil - } - } +// vSockSupported, err := isAFVSockSupported() +// if err != nil { +// return nil, err +// } +// +// if vSockSupported { +// // Check if vsock socket exists. We want to cover the case +// // where the guest OS can support vsock, but the runtime is +// // still using virtio serial connection. +// exist, err := vSockPathExist() +// if err != nil { +// return nil, err +// } +// +// if exist { +// return &vSockChannel{}, nil +// } +// } return &serialChannel{}, nil } * The Fedora initramfs and modular kernel rely on udev to load the virtio_console.ko driver. Make sure to start kata-agent *after* the agent device appears. Note this change will break AF_VSOCK if you choose to use that instead of virtio-serial! diff --git a/kata-agent.service.in b/kata-agent.service.in index 861757d..5395047 100644 --- a/kata-agent.service.in +++ b/kata-agent.service.in @@ -2,6 +2,8 @@ Description=Kata Containers Agent Documentation=https://github.com/kata-containers/agent Wants=kata-containers.target +BindsTo=dev-virtio\x2dports-agent.channel.0.device +After=dev-virtio\x2dports-agent.channel.0.device [Service] # Send agent output to tty to allow capture debug logs * Make sure the rootfs has modprobe and udev. Needed by the modular kernel. diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh index a0ca15a..19fbc7b 100644 --- a/rootfs-builder/fedora/config.sh +++ b/rootfs-builder/fedora/config.sh @@ -14,4 +14,4 @@ PACKAGES="iptables" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true

6 8

Want to contribute?
by Hunt, James O 29 Jun '18

29 Jun '18

Hi All, If you're interested in contributing to the Kata Project but don't know where to start, here are some pointers... = TL;DR = The general advice is: - Start small. - If in doubt, raise an issue. - Talk to us! ;) = Read = - https://github.com/kata-containers/documentation/blob/master/install/README… - https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md - https://github.com/kata-containers/documentation/blob/master/architecture.md - https://github.com/kata-containers/documentation/blob/master/Developer-Guid… If something is unclear/missing/wrong, raise an issue so we (or you, if you want) can fix it. = Review = Take a look at some of the PRs that are in the queue: - https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+org%3Akata-cont… Add comments if you don't understand something. We want the code to be as easy to understand as possible, so asking questions (even "silly questions" aka "questions nobody else though of" or "questions nobody else was brave enough to ask"!) helps to ensure that. = Tackle = Take a look at the "bitesize" and "good first issue" issues: - https://github.com/issues?utf8=%E2%9C%93&q=is%3Aopen+org%3Akata-containers+… - https://github.com/issues?q=is%3Aopen+org%3Akata-containers+label%3Abitesize Add a comment stating that you'd like to have a go. If you don't know how to start, ask for help! Note that multiple people can be assigned to an issue if necessary. = Lurk = Attend Architecture Committee and Working Committee meetings: - https://katacontainers.io/ You don't need to be an architect / programmer / existing contributor to join! ;) Hang out on slack+irc - see below. = Talk = We don't bite - come along and chat on slack/irc and let us know the sorts of things you'd like to get involved in: - https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md#co… = Promote = We'd love to have a regular blog post summarising the "week in Kata". If you'd like to get involved, see: - http://lists.katacontainers.io/pipermail/kata-dev/2018-April/000113.html. = Play = Install Kata and see how it works: - https://github.com/kata-containers/documentation/blob/master/install/README… = Test = We're always looking to increase our test coverage. Each repo has it's own set of unit tests but we keep all other types of tests in it's own repo and all those tests also get run against all PRs for system components: - https://github.com/kata-containers/tests Take a look at the unit-test coverage for, say, the runtime repo and see if you can find a way to increase it: - https://github.com/kata-containers/runtime = Customise = Try building a custom image with osbuilder: - https://github.com/kata-containers/osbuilder = Optimise = Enable full debug ( https://github.com/kata-containers/documentation/blob/master/Developer-Guid…) create a Kata Container and then start analysing the logs. We've got a basic tool called kata-log-parser that can convert the kata log entries to other formats (text, XML, JSON, TOML, YAML, CSV). Along with high-resolution timestamps, the tool also collates all the log entries and calculates time deltas. That isn't quite as good as system level tracing but can give you an indication of which parts of the system might need further investigation: - https://github.com/kata-containers/tests/tree/master/cmd/log-parser = Overcome = Take a look at the limitations doc and see if you can help to chip away at the outstanding issues: - https://github.com/kata-containers/documentation/blob/master/Limitations.md Cheers, James --- https://katacontainers.io/ | https://github.com/clearcontainers Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ.

1 1

September in-person meeting dates released
by Anne Bertucio 29 Jun '18

29 Jun '18

Hi all, Kata Containers has been allotted space on the Tuesday-Thursday of the OpenStack PTG (September 11-13). If you’re not familiar with the PTG, it is in-person meeting time for projects that live at the OpenStack Foundation. The meeting is for active contributors of the project to get together for focused development time. By the end of the week in Vancouver, we were getting into a good flow with each other and had some good discussions. You can think of the PTG meeting as an extension of this time, minus the distractions of conference sessions and activities. To make this time worthwhile, we need a critical mass from our project to attend. If you are planning to attend (even if you haven’t sorted travel), please add your name to the etherpad [1] so we can ensure we’ll have a good group. Hope many of you are able to make it! [1] https://etherpad.openstack.org/p/KataContainersPTG4 <https://etherpad.openstack.org/p/KataContainersPTG4> [2] https://www.openstack.org/ptg <https://www.openstack.org/ptg> Anne Bertucio OpenStack Foundation anne(a)openstack.org | irc: annabelleB

1 0

Kata September In-Person Meeting
by Anne Bertucio 23 Jun '18

23 Jun '18

Hi all, Please see the note below from Kendall Nelson (OSF staff) regarding the September OSF project meeting in Denver. For Kata, we will have meeting space for approximately 3 days, and presumably there will be some cross-OSF project meetings as well. The success of this time hinges on participation, so I would strongly encourage you to attend if you’re able. In Vancouver, we were able to move through some longstanding topics quickly by being in person, and I’m looking forward to doing more of this in Denver. You can read more about the project meeting at openstack.org/ptg <http://openstack.org/ptg>, or shoot me a note if you have any questions about attending the Kata meeting. If you plan on attending, please drop your name in this etherpad[1]. Cheers, Anne [1] https://etherpad.openstack.org/p/KataContainersPTG4 ----------------------------------------------------------------- Hello Everyone! Wanted to give you some updates on PTG4 planning. We have finalized the list of SIGs/ Groups/WGs/Teams that are attending. They are as follows: Airship API SIG Barbican/Security SIG Blazar Chef OpenStack Cinder Cyborg Designate Documentation Edge Computing Group First Contact SIG Glance Heat Horizon Infrastructure Interop WG Ironic Kata Containers Keystone Kolla LOCI Manila Masakari Mistral Monasca Neutron Nova Octavia OpenStack Ansible OpenStack Charms OpenStack Helm OpenStackClient Operator Meetup Puppet OpenStack QA Oslo Public Cloud WG Release Management Requirements Sahara Scientific SIG Self-Healing SIG SIG- K8s StarlingX Swift Technical Committee TripleO Upgrades SIG Watcher Zuul (pending confirmation) Thierry and I are working on placing them into a strawman schedule to reduce conflicts between related or overlapping groups. We should have more on what that will look like and a draft for you all to review in the next few weeks. We also wanted to remind you all of the Travel Support Program. We are again doing a two phase selection. The first deadline is approaching: July 1st. At this point we have less than a dozen applicants so if you need it or even think you need it, I urge you to apply here[1]. Also! Reminder that we have a finite number of rooms in the hotel block so please book early to make sure you get the discounted rate before they run out. You can book those rooms here[2] (pardon the ugly URL). Can't wait to see you all there! -Kendall Nelson (diablo_rojo) P.S. Gonna try to do a game night again since you all seemed to enjoy it so much last time :) [1] https://openstackfoundation.formstack.com/forms/travelsupportptg_denver_2018 <https://openstackfoundation.formstack.com/forms/travelsupportptg_denver_2018> [2] https://www.marriott.com/meeting-event-hotels/group-corporate-travel/groupC… <https://www.marriott.com/meeting-event-hotels/group-corporate-travel/groupC…> Anne Bertucio OpenStack Foundation anne(a)openstack.org | irc: annabelleB

2 1

What to do about release versioning ?
by Boeuf, Sebastien 23 Jun '18

23 Jun '18

Hi folks, This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation). But before we can answer this, we have to figure out what is that we consider a "breakage". Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages: First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it. Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command. I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues. The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account. So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly. Numbers are cheap, but here is the status and I/we need some feedback before we can push the pending release (as we don't want to introduce a wrong version number in the history). Thanks, Sebastien

11 11

kata-containers issues when running CRI-O
by Fuentes, Salvador 22 Jun '18

22 Jun '18

Hi all, Recently we have seen issues with the CRI-O tests in our CI. We have recorded some of the failures on https://github.com/kata-containers/tests/issues/361 I have tried to reproduce the failures and these are my findings: - I found that several kernel warnings are being displayed on the kata-proxy log (you can see the warnings on a comment in the issue above). - There seems to be a race condition somewhere because when running cri-o (crictl) commands manually, I cannot reproduce the issue. - The cri-o random failures seem to only appear in Ubuntu 16-04 host machines and the tests that mostly fail are: 6,7,8 and 9. - I am not sure that the cri-o tests failures are 100% related to the kernel warnings. Sometimes the kernel warning occurs, but the tests pass. - It may be related to the storage driver been used: On ubuntu we use devicemapper, and in the fedora and centos we use overlay. If someone can help debugging this, it would be very helpful. As for the CI failures, which are very annoying, an interim action could be to disable the tests 6-9. With this, we can have a stable CI, while we continue investigating the issue. How to reproduce the issue: 1. Update your kata-containers components to the master branch 2. You'll need to install CRI-O 1.10 and the CNI plugins. If you already have the kata-containers/tests repository, you can run: .ci/install_crio.sh .ci/install_cni_plugins.sh 3. Run the tests using kata-containers runtime: In The kata-containers/tests repository, run: export RUNTIME=kata-runtime sudo -E PATH=$PATH make crio 4. Check logs: sudo journalctl -t kata-proxy sudo journalctl -t kata-runtime Thanks, Salvador Fuentes.

1 0

CPU constraint translation to Qemu SMP options
by Edupuganti, Saikrishna 21 Jun '18

21 Jun '18

Hi Team, When translating the K cpu count from Docker/k8s to Qemu options, what is reasoning behind configuring K sockets with 1 core per socket instead of K cores with 1 socket? Did any performance measurements guide this decision? When we bring up kata container with 8 cpus it brings up a VM with (8+1) sockets - kubectl exec -it ubuntu-kata -- lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 kubectl exec -it ubuntu-kata-8cpu-limit -- lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 9 On-line CPU(s) list: 0-8 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 9 Thanks, Sai

4 5

Kata with AMD Secure Encrypted Virtualization (SEV)
by Larrew, Jesse 21 Jun '18

21 Jun '18

Hi all, The virtualization instructions in the latest AMD EPYC server processors have been enhanced with a memory encryption feature that could provide projects like Kata Containers with unique security capabilities compared to their non-virtualized counterparts. We call this feature Secure Encrypted Virtualization (or SEV) and would be interested in collaborating with the Kata Container community to enable support for containers with encrypted memory. In short, we've added an inline AES engine to our memory controller that encrypts data written to system DRAM and decrypts data read from DRAM. The encryption keys are generated from a TRNG in the onboard AMD Secure Processor (SP) and programmed into the memory controller as needed in a manner that is never visible to software. Additionally, our virtualization instructions have been enhanced to be able to associate a VM ASID with a unique encryption key, so each VM (or container) can keep the contents of its memory confidential from the host and/or other tenant VMs/containers. The guest kernel can choose which pages to encrypt and which to share with the host by setting a bit in the guest page tables, which puts the guest in complete control of the visibility of their data in the cloud. More information can be found in our Memory Encryption whitepaper [1] and in the Architecture Programmer's Manual [2]. Linux kernel support for SEV has been merged into the 4.15 and upcoming 4.16 kernels. OVMF BIOS support has been merged as well. The qemu changes are still being upstreamed, but the patches are available for testing on github [3]. With the above support in place, we have developed a proof-of-concept demo that is based on Clear Containers. Since the Clear Containers project had already done the heavy lifting to run container workloads inside of a VM, it was rather straightforward to add support to encrypt those VMs using SEV. The required changes are summarized below: * Container kernel: - Add SEV support patches from the Linux kernel repo in [3]. - Force virtio to use the DMA API (and hence SWIOTLB) when adding/removing buffers to/from the virtio ring buffer. - SEV requires a memory copy in order to perform the encryption, so zero-copy solutions using DAX for the container initial user space will not work. + Build in a small initramfs to use as the guest kernel initial user space. + Include the updated container agent binary and supporting libs (~14MB total). * Container agent: - Update the agent not to use the pivot_root() method from the initramfs environment, and perform the pivot to the container workload filesystem manually instead. * Container runtime: - Add the new qemu command line options for starting an SEV guest. * Qemu-lite: - Add the SEV support patches from the qemu repo in [3]. With the above changes, we are able to start docker containers inside of SEV-protected VMs: amd@pecanporter:~/src/git$ sudo docker run -ti --runtime sev-runtime busybox sh / # whoami root / # dmesg | grep SEV [ 0.001000] AMD Secure Encrypted Virtualization (SEV) active [ 0.219196] SEV is active and system is using DMA bounce buffers As a check, dumping the contents of a page from the qemu heap reveals plaintext data: amd@pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem bs=4096 count=1 skip=23058854513 | xxd | tail dd: /proc/38572/mem: cannot skip to specified offset 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.8437e-05 s, 46.3 MB/s 00000f60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f80: 0000 0000 0000 0000 7100 0000 0000 0000 ........q....... 00000f90: 2f72 756e 2f76 6972 7463 6f6e 7461 696e /run/virtcontain 00000fa0: 6572 732f 706f 6473 2f33 3565 3233 6565 ers/pods/35e23ee 00000fb0: 3330 6466 6237 3266 3135 3730 6265 3432 30dfb72f1570be42 00000fc0: 6665 3165 6331 3366 3331 3332 6138 6133 fe1ec13f3132a8a3 00000fd0: 6463 3336 6463 3131 6235 6365 3837 6236 dc36dc11b5ce87b6 00000fe0: 3437 3930 3736 6339 612f 636f 6e73 6f6c 479076c9a/consol 00000ff0: 652e 736f 636b 0000 0104 0000 0000 0000 e.sock.......... However, any attempt to read the container memory from the host produces only ciphertext: amd@pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem bs=4096 count=1 skip=34165702144 | xxd | head dd: /proc/38572/mem: cannot skip to specified offset 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.9039e-05 s, 46.0 MB/s 00000000: e9b8 e14d c063 ee18 fd85 5ecc 4d1f c1a2 ...M.c....^.M... 00000010: d681 cdf2 259b a97e c43b 5cde bf9e 695b ....%..~.;\...i[ 00000020: db3c 778b 8e77 89f4 f795 e5a6 9ebb 765b .<w..w........v[ 00000030: 0905 e1d3 c7ec 6f2b bada ed15 b2e0 db7f ......o+........ 00000040: d5e9 6d15 cf28 0ca1 4a45 3b9a 1779 e3ff ..m..(..JE;..y.. 00000050: 9ee0 b562 2311 6e5a e972 4c06 3f6a 6ebf ...b#.nZ.rL.?jn. 00000060: 909a 88ea 737a 6226 5d87 8968 b31b d096 ....szb&]..h.... 00000070: 9360 cbb0 4f34 d811 89a7 048f 01e8 d19e .`..O4.......... 00000080: 5429 995a 4de0 6fba 3360 8bb4 a2dc 17e4 T).ZM.o.3`...... 00000090: 80f5 6657 9fd7 0347 e78d 4d13 6b6c c649 ..fW...G..M.kl.I Our threat model is to allow container workloads to reduce their risk exposure to security vulnerabilities in the hosting environment, which seems to overlap nicely with the threat model of Kata Containers. Is this a feature that the Kata community would find useful? If so, we would be very interested to work with the community to enable SEV memory encryption for Kata Containers. Any and all feedback is welcome! Thanks! References: [1] AMD Memory Encryption Whitepaper: http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_E… [2] AMD64 Architecture Programmer's Manual Volume 2: System Programming: http://developer.amd.com/wordpress/media/2012/10/24593_APM_v21.pdf [3] AMD SEV github repo: https://github.com/AMDESE/AMDSEV Sincerely, Jesse Larrew MTS Software Security Architect AMD Security Architecture R&D jesse.larrew(a)amd.com O: +(1) 512-602-0092 (x50092) M: +(1) 512-791-4852

12 32

RFC: Let's do a 1.0.1 release
by Ernst, Eric 20 Jun '18

20 Jun '18

Hey Carlos, all – I chatted a bit on #kata-dev on it and got +1 from Xu/Wei, but I think it is time we do a 1.0.1 release. We’ve had some good fixes since May 22nd land, including some to issues identified during hackathon in Beijing this week. Can we do a release before end of week (sooner is better of course) for 1.0.1? In a perfect world, we’ll have [1] landed, allowing us to post all of our build artifacts on the github release page. WDYT? Any other bug fixes in flight which we should try to land first? Thanks, Eric [1] - https://github.com/kata-containers/packaging/pull/67

3 3

Read & reply please
by edward 18 Jun '18

18 Jun '18

Hello and good day. My name is Edward, and I am contacting you for partnership by reason of the significance of your Nationality, which concerns claim/transfer of Approximately 21 Million US D0llars ia a fixed deposit account left by a deceased Boss and Friend for onward sharing between you and I in a ratio to be agreed. Kindly get back to me as soon as you receive this email for complete details as time is of the essence giving that we have less than 2 month period to achieve this. Regards Edward.

1 0