<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style>
<!--
@font-face
{font-family:Wingdings}
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif}
a:link, span.MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif}
p.msonormal0, li.msonormal0, div.msonormal0
{margin-right:0in;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.EmailStyle19
{font-family:"Calibri",sans-serif;
color:windowtext}
.MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
ol
{margin-bottom:0in}
ul
{margin-bottom:0in}
-->
</style><style type="text/css" id="owaParaStyle">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body fpstyle="1" ocsi="0" vlink="#954F72" link="#0563C1" lang="EN-US">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div>Thanks for this nice summary Eric!</div>
<div><br>
</div>
<div>A few questions:</div>
<div><br>
</div>
<div>- Did you discuss about having virtio-pci being implemented for FC, as it was mentioned Amazon were not opposed to replace virtio-mmio during the meeting we had in Seattle a month ago.</div>
<div><br>
</div>
<div>- When you mention "<span lang="en-US"><font size="3" face="Calibri,sans-serif"><span style="font-size:12pt;"><font size="2"><span style="font-size:11pt;">defining the scope of Firecracker K8S integration (ie: required, nice-to-have, and planned lack of
support)</span></font></span></font></span>", do you mean for instance that not having the support for passing files and directories (yes I'm thinking about fs sharing not being supported by FC) would be defined at the k8s level so that the pod would not try
to pass any of those?</div>
<div><br>
</div>
<div>- More generally, is there any news regarding filesystem sharing plans?<br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Sebastien<br>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF207548" style="direction: ltr;"><font size="2" face="Tahoma" color="#000000"><b>From:</b> Ernst, Eric [eric.ernst@intel.com]<br>
<b>Sent:</b> Friday, January 11, 2019 11:33 PM<br>
<b>To:</b> kata-dev@lists.katacontainers.io<br>
<b>Subject:</b> [kata-dev] notes: kata + aws firecracker discussions - Jan 11 2019<br>
</font><br>
</div>
<div></div>
<div>
<div class="WordSection1">
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Hey folks,</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">A few of us had the chance to have some follow up discussions with some more members of the AWS firecracker, containers and open source teams.
</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">This was a great chance to meet and talk, highlighting some of the progress the Kata community has made with enabling Firecracker, while also learning more about Amazon’s goals with the
Firecracker project. I’m hopeful that next time it can be done in a more open setting. With this in mind, I am starting a working agenda for tasks and topics at the Denver PTG (project team gathering – May 2-4) which coincides with the OpenInfra conference
at the beginning of May. This is a great opportunity to meet for a couple of days face to face, without any agenda except what developers want to do for the project: hacking and collaboration. I’ll follow up on this next week.
</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Please find my rough notes below; I’m hoping to get the recording from the call to share on the mailing list.</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Thanks,</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Eric</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Discussions between Kata + AWS Firecracker/Containers– Jan 11 2019</span></p>
<p class="MsoNormal"> </p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Attendees:</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Eric Ernst</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Manohar Castelino</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Claire Massey</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Arun Gupta</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Samuel Ortiz</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Adrian Cockroft</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Noah Meyerhans</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Samuel Karp</span></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Matt Wilson</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Meena Gowdar</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Actions:</span></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in"><span style="font-size:11.0pt">Eric to provide Arun with estimated Kata+firecracker CI utilization</span></li><li class="MsoListParagraph" style="margin-left:0in"><span style="font-size:11.0pt">Claire to work with Arun on coordination of 1.5 Kata release blog highlighting Firecracker support</span></li><li class="MsoListParagraph" style="margin-left:0in"><span style="font-size:11.0pt">Manohar to prepare and setup agenda item for comparing the containerd and cri-o storage driver interfaces, as a basis for LVM work which could be shared</span></li><li class="MsoListParagraph" style="margin-left:0in"><span style="font-size:11.0pt">Eric to reach out to Sig-Node to discuss next steps on runtimeClass, and in particular around resource management.</span></li><li class="MsoListParagraph" style="margin-left:0in"><span style="font-size:11.0pt">Eric to start a “Kubernetes feature requirements” document, which the greater community (AWS containers team + Kata) can work on as a basis for better defining the scope of
Firecracker K8S integration (ie: required, nice-to-have, and planned lack of support).
</span></li></ul>
<p class="MsoNormal"> </p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">Notes:</span></p>
<p class="MsoNormal"> </p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">After general introductions, Eric/Manohar provided an overview / background of status and scope of Kata containers as of 1.5.0-rc2 -- see [1]. Manohar provided a demonstration of Kata
+ FC + QEMU in K8S (with CRI-O) provided. Most of this is covered in the screen cast at [2]. Support expected to be in the 1.5.0 release, week of January 23rd.</span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style="color:black"><span style="font-size:11.0pt">Kata team to plan on a blog post which can be published with AWS (work with Arun) to highlight release of Firecracker support within Kata. </span></li><li class="MsoNormal" style="color:black; vertical-align:baseline"><span style="font-size:11.0pt; color:windowtext">From Kata point of view,
</span><span style="font-size:11.0pt">Firecracker micro-VMM fills a gap that users have expressed around the size and complexity of QEMU. With the hypervisor’s limited feature set, fully featured Kubernetes integration will be out of scope. Kata + firecracker-containers
team should jointly create a Kubernetes feature list which can be sorted as required-features, nice to have, and planned lack of support. With this in place, feature requests and issues with relative priority can be opened on the Firecracker GitHub, and where
applicable we can work with the Kubernetes community.</span></li><li class="MsoNormal" style="color:black; vertical-align:baseline"><span style="font-size:11.0pt">Highlighting admission controller work which has been utilized for effectively enabling Kata in a cluster with multiple runtime classes.</span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">Kata + Firecracker Continuous Integration on bare-metal: Kata team would like to make sure Firecracker integration with Kata is well tested, including on AWS cloud. We are adding today
for nested-virtualization nodes, but would like to also make use of AWS. <b>Action: Eric
</b>to follow up with CI load estimation for the current suite of Kata tests.</span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">Kata project has some group/namespace utilization on the host, but has had some gaps with respect to fully constraining the hypervisor, and have been looking at libvirt and home-grown
solutions. Jailer looks very attractive, and is the “right level” of complexity and utility we’ve been looking for. We’ll start taking a look at using this irrespective of the hypervisor (currently will have gaps due to default vhost for networking when
utilizing QEMU). This will probably land post 1.5 (schedule 1/23).</span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">Snapshotter: A great place for immediate, direct collaboration. Since docker/containerd support for devicemapper dropped, AWS team has been working on implementing block based snapshotters
to fit into containerd’s snaphotter interface. Having had discussions during CRI-O contributors call, we have an opportunity to collaborate on creating an LVM based snapshotter. Action: Manohar - add CRI-O, containerd snapshotter interface comparison to
Kata Arch Committee meeting, [3], and use as basis for starting LVM work. Kata folks to test existing firecracker-containers team’s snapshotters.</span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">General discussion on solution complexity: Similar to prior discussions, some good highlights into the design principles of Firecracker Hypervisor: reduced feature set by design to achieve
an appropriate security profile. On Kata Containers, while adding firecracker support was relatively straightforward, it did highlight some areas where we can reduce features and configuration options. Based on this, Kata is starting to become more opinionated
to help reduce complexity. You can track a lot of this in [4]. Suggestions, comments and PRs welcome :)</span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">Discussions around production use of Kata Containers: There’s a lot of activity right now, and expected a few new users in production in Q1. Over the next couple of months, there should
be a lot more that can be highlighted in this space.</span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">Discussion on resource management work to look at in short term: There are some gaps in the initial implementation of runtimeClass in Kubernetes today. One example is a lack of insight
into overheads introduced by runtime implementations, which requires workarounds to make sure node resources available are calculated accurately. Would be good to see if developers from the Kata community can help with some of the features in consideration
at sig-node. </span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">Kata currently leverages the swagger API to interface with Firecracker. Work is currently under way to migrate to the Go SDK for FC, and augment where needed (in particular, the block
device rescan support Kata utilizes to mimic hotplug). </span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt; color:black">Highlighted the gRPC API utilized for interfacing with the guest agent, kata-agent. Other projects have looked at re-using this interface - may be a good for the firecracker-containerd
team to review.</span><span style="font-size:11.0pt"></span></li><li class="MsoNormal" style="color:black"><span style="font-size:11.0pt">Discussion on aligning firecracker-containerd work with what is being developed today in Kata Containers. This is still an open on the Amazon side, though there has been good collaboration
to date, including common problem areas (see snapshotter notes).</span></li><li class="MsoNormal" style=""><span style="font-size:11.0pt">Many CFPs (OSCON, OpenInfra, KubeCon, ONS) are closing in the next two weeks: pick a couple to highlight the work that has been done to date. See [5].</span></li></ul>
<p class="MsoNormal" style="margin-bottom:12.0pt"> </p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">[1] - </span>
<a href="https://drive.google.com/file/d/1aEzHHesNoVsZOqK-IIP0g2bbPIRw48Ev/view" target="_blank" rel="noopener noreferrer"><span style="color:#1155CC">https://drive.google.com/file/d/1aEzHHesNoVsZOqK-IIP0g2bbPIRw48Ev/view</span></a></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">[2] - </span>
<a href="https://asciinema.org/a/219790" target="_blank" rel="noopener noreferrer"><span style="color:#1155CC">https://asciinema.org/a/219790</span></a></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">[3] - </span>
<a href="https://etherpad.openstack.org/p/katacontainers-2019-architecture-committee-mtgs" target="_blank" rel="noopener noreferrer"><span style="color:#1155CC">https://etherpad.openstack.org/p/katacontainers-2019-architecture-committee-mtgs</span></a></p>
<p style="margin:0in; margin-bottom:.0001pt"><span style="color:black">[4] - https://github.com/kata-containers/runtime/issues/1113</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[5] - <span style="color:black">
<a href="https://etherpad.openstack.org/p/KataTalks2019" target="_blank" rel="noopener noreferrer"><span style="color:purple">https://etherpad.openstack.org/p/KataTalks2019</span></a></span></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span></p>
</div>
</div>
</div>
</div>
</body>
</html>