Hey folks,

A few of us had the chance to have some follow up discussions with some more members of the AWS firecracker, containers and open source teams. 

This was a great chance to meet and talk, highlighting some of the progress the Kata community has made with enabling Firecracker, while also learning more about Amazon’s goals with the Firecracker project.  I’m hopeful that next time it can be done in a more open setting.  With this in mind, I am starting a working agenda for tasks and topics at the Denver PTG (project team gathering – May 2-4) which coincides with the OpenInfra conference at the beginning of May.  This is a great opportunity to meet for a couple of days face to face, without any agenda except what developers want to do for the project: hacking and collaboration.  I’ll follow up on this next week. 

Please find my rough notes below; I’m hoping to get the recording from the call to share on the mailing list.

Thanks,

Eric

Discussions between Kata + AWS Firecracker/Containers– Jan 11 2019

Attendees:

Eric Ernst

Manohar Castelino

Claire Massey

Arun Gupta

Samuel Ortiz

Adrian Cockroft

Noah Meyerhans

Samuel Karp

Matt Wilson

Meena Gowdar

Actions:

• Eric to provide Arun with estimated Kata+firecracker CI utilization
• Claire to work with Arun on coordination of 1.5 Kata release blog highlighting Firecracker support
• Manohar to prepare and setup agenda item for comparing the containerd and cri-o storage driver interfaces, as a basis for LVM work which could be shared
• Eric to reach out to Sig-Node to discuss next steps on runtimeClass, and in particular around resource management.
• Eric to start a “Kubernetes feature requirements” document, which the greater community (AWS containers team + Kata) can work on as a basis for better defining the scope of Firecracker K8S integration (ie: required, nice-to-have, and planned lack of support).

Notes:

• After general introductions, Eric/Manohar provided an overview / background of status and scope of Kata containers as of 1.5.0-rc2 -- see [1].  Manohar provided a demonstration of Kata + FC + QEMU in K8S (with CRI-O) provided.  Most of this is covered in the screen cast at [2].  Support expected to be in the 1.5.0 release, week of January 23rd.
• Kata team to plan on a blog post which can be published with AWS (work with Arun) to highlight release of Firecracker support within Kata.  
• From Kata point of view, Firecracker micro-VMM fills a gap that users have expressed around the size and complexity of QEMU.  With the hypervisor’s limited feature set, fully featured Kubernetes integration will be out of scope.  Kata + firecracker-containers team should jointly create a Kubernetes feature list which can be sorted as required-features, nice to have, and planned lack of support.  With this in place, feature requests and issues with relative priority can be opened on the Firecracker GitHub, and where applicable we can work with the Kubernetes community.
• Highlighting admission controller work which has been utilized for effectively enabling Kata in a cluster with multiple runtime classes.
• Kata + Firecracker Continuous Integration on bare-metal: Kata team would like to make sure Firecracker integration with Kata is well tested, including on AWS cloud.  We are adding today for nested-virtualization nodes, but would like to also make use of AWS. Action: Eric to follow up with CI load estimation for the current suite of Kata tests.
• Kata project has some group/namespace utilization on the host, but has had some gaps with respect to fully constraining the hypervisor, and have been looking at libvirt and home-grown solutions.  Jailer looks very attractive, and is the “right level” of complexity and utility we’ve been looking for.  We’ll start taking a look at using this irrespective of the hypervisor (currently will have gaps due to default vhost for networking when utilizing QEMU).  This will probably land post 1.5 (schedule 1/23).
• Snapshotter: A great place for immediate, direct collaboration.   Since docker/containerd support for devicemapper dropped, AWS team has been working on implementing block based snapshotters to fit into containerd’s snaphotter interface.  Having had discussions during CRI-O contributors call, we have an opportunity to collaborate on creating an LVM based snapshotter.  Action: Manohar - add CRI-O,  containerd snapshotter interface comparison to Kata Arch Committee meeting, [3], and use as basis for starting LVM work.   Kata folks to test existing firecracker-containers team’s snapshotters.
• General discussion on solution complexity: Similar to prior discussions, some good highlights into the design principles of Firecracker Hypervisor: reduced feature set by design to achieve an appropriate security profile.  On Kata Containers, while adding firecracker support was relatively straightforward, it did highlight some areas where we can reduce features and configuration options.  Based on this, Kata is starting to become more opinionated to help reduce complexity.  You can track a lot of this in [4].  Suggestions, comments and PRs welcome :)
• Discussions around production use of Kata Containers: There’s a lot of activity right now, and expected a few new users in production in Q1.  Over the next couple of months, there should be a lot more that can be highlighted in this space.
• Discussion on resource management work to look at in short term:  There are some gaps in the initial implementation of runtimeClass in Kubernetes today.  One example is a lack of insight into overheads introduced by runtime implementations, which requires workarounds to make sure node resources available are calculated accurately. Would be good to see if developers from the Kata community can help with some of the features in consideration at sig-node.  
• Kata currently leverages the swagger API to interface with Firecracker.  Work is currently under way to migrate to the Go SDK for FC, and augment where needed (in particular, the block device rescan support Kata utilizes to mimic hotplug).
• Highlighted the gRPC API utilized for interfacing with the guest agent, kata-agent.  Other projects have looked at re-using this interface - may be a good for the firecracker-containerd team to review.
• Discussion on aligning firecracker-containerd work with what is being developed today in Kata Containers.  This is still an open on the Amazon side,  though there has been good collaboration to date, including common problem areas (see snapshotter notes).
• Many CFPs (OSCON, OpenInfra, KubeCon, ONS) are closing in the next two weeks: pick a couple to highlight the work that has been done to date.  See [5].

[1] - https://drive.google.com/file/d/1aEzHHesNoVsZOqK-IIP0g2bbPIRw48Ev/view

[2] - https://asciinema.org/a/219790

[3] - https://etherpad.openstack.org/p/katacontainers-2019-architecture-committee-mtgs

[4] - https://github.com/kata-containers/runtime/issues/1113

[5] - https://etherpad.openstack.org/p/KataTalks2019