<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:195503217;
mso-list-type:hybrid;
mso-list-template-ids:-1919138464 610409404 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:"Times New Roman";
color:black;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:1033580292;
mso-list-template-ids:-568181218;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2
{mso-list-id:1241601550;
mso-list-template-ids:492842364;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3
{mso-list-id:1828008352;
mso-list-type:hybrid;
mso-list-template-ids:1354152468 610409404 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l3:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:"Times New Roman";
color:black;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Hey folks,<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">A few of us had the chance to have some follow up discussions with some more members of the AWS firecracker, containers and open source teams.
<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">This was a great chance to meet and talk, highlighting some of the progress the Kata community has made with enabling Firecracker, while also learning more about Amazon’s goals with the Firecracker
project. I’m hopeful that next time it can be done in a more open setting. With this in mind, I am starting a working agenda for tasks and topics at the Denver PTG (project team gathering – May 2-4) which coincides with the OpenInfra conference at the beginning
of May. This is a great opportunity to meet for a couple of days face to face, without any agenda except what developers want to do for the project: hacking and collaboration. I’ll follow up on this next week.
<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Please find my rough notes below; I’m hoping to get the recording from the call to share on the mailing list.<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Thanks,<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Eric<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Discussions between Kata + AWS Firecracker/Containers– Jan 11 2019</span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Attendees:</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Eric Ernst</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Manohar Castelino</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Claire Massey</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Arun Gupta</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Samuel Ortiz</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Adrian Cockroft</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Noah Meyerhans</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Samuel Karp</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Matt Wilson</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Meena Gowdar<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Actions:<o:p></o:p></span></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Eric to provide Arun with estimated Kata+firecracker CI utilization<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Claire to work with Arun on coordination of 1.5 Kata release blog highlighting Firecracker support<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Manohar to prepare and setup agenda item for comparing the containerd and cri-o storage driver interfaces, as a basis for LVM work which could be shared<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Eric to reach out to Sig-Node to discuss next steps on runtimeClass, and in particular around resource management.<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Eric to start a “Kubernetes feature requirements” document, which the greater community (AWS containers team + Kata) can work on as a basis for better
defining the scope of Firecracker K8S integration (ie: required, nice-to-have, and planned lack of support).
<o:p></o:p></span></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">Notes:</span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">After general introductions, Eric/Manohar provided an overview / background of status and scope of Kata containers as of 1.5.0-rc2 -- see [1]. Manohar provided
a demonstration of Kata + FC + QEMU in K8S (with CRI-O) provided. Most of this is covered in the screen cast at [2]. Support expected to be in the 1.5.0 release, week of January 23rd.</span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo6"><span style="font-size:11.0pt">Kata team to plan on a blog post which can be published with AWS (work with Arun) to highlight release of Firecracker support within Kata. <o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo6;vertical-align:baseline">
<span style="font-size:11.0pt;color:windowtext">From Kata point of view, </span><span style="font-size:11.0pt">Firecracker micro-VMM fills a gap that users have expressed around the size and complexity of QEMU. With the hypervisor’s limited feature set, fully
featured Kubernetes integration will be out of scope. Kata + firecracker-containers team should jointly create a Kubernetes feature list which can be sorted as required-features, nice to have, and planned lack of support. With this in place, feature requests
and issues with relative priority can be opened on the Firecracker GitHub, and where applicable we can work with the Kubernetes community.<o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo6;vertical-align:baseline">
<span style="font-size:11.0pt">Highlighting admission controller work which has been utilized for effectively enabling Kata in a cluster with multiple runtime classes.<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">Kata + Firecracker Continuous Integration on bare-metal: Kata team would like to make sure Firecracker integration with Kata is well tested, including on AWS cloud.
We are adding today for nested-virtualization nodes, but would like to also make use of AWS.
<b>Action: Eric </b>to follow up with CI load estimation for the current suite of Kata tests.</span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">Kata project has some group/namespace utilization on the host, but has had some gaps with respect to fully constraining the hypervisor, and have been looking at
libvirt and home-grown solutions. Jailer looks very attractive, and is the “right level” of complexity and utility we’ve been looking for. We’ll start taking a look at using this irrespective of the hypervisor (currently will have gaps due to default vhost
for networking when utilizing QEMU). This will probably land post 1.5 (schedule 1/23).</span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">Snapshotter: A great place for immediate, direct collaboration. Since docker/containerd support for devicemapper dropped, AWS team has been working on implementing
block based snapshotters to fit into containerd’s snaphotter interface. Having had discussions during CRI-O contributors call, we have an opportunity to collaborate on creating an LVM based snapshotter. Action: Manohar - add CRI-O, containerd snapshotter
interface comparison to Kata Arch Committee meeting, [3], and use as basis for starting LVM work. Kata folks to test existing firecracker-containers team’s snapshotters.</span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">General discussion on solution complexity: Similar to prior discussions, some good highlights into the design principles of Firecracker Hypervisor: reduced feature
set by design to achieve an appropriate security profile. On Kata Containers, while adding firecracker support was relatively straightforward, it did highlight some areas where we can reduce features and configuration options. Based on this, Kata is starting
to become more opinionated to help reduce complexity. You can track a lot of this in [4]. Suggestions, comments and PRs welcome :)</span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">Discussions around production use of Kata Containers: There’s a lot of activity right now, and expected a few new users in production in Q1. Over the next couple
of months, there should be a lot more that can be highlighted in this space.</span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">Discussion on resource management work to look at in short term: There are some gaps in the initial implementation of runtimeClass in Kubernetes today. One example
is a lack of insight into overheads introduced by runtime implementations, which requires workarounds to make sure node resources available are calculated accurately. Would be good to see if developers from the Kata community can help with some of the features
in consideration at sig-node. </span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">Kata currently leverages the swagger API to interface with Firecracker. Work is currently under way to migrate to the Go SDK for FC, and augment where needed (in
particular, the block device rescan support Kata utilizes to mimic hotplug). </span>
<span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt;color:black">Highlighted the gRPC API utilized for interfacing with the guest agent, kata-agent. Other projects have looked at re-using this interface - may be a good for the
firecracker-containerd team to review.</span><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo6"><span style="font-size:11.0pt">Discussion on aligning firecracker-containerd work with what is being developed today in Kata Containers. This is still an open on the Amazon side, though there
has been good collaboration to date, including common problem areas (see snapshotter notes).<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l3 level1 lfo6"><span style="font-size:11.0pt">Many CFPs (OSCON, OpenInfra, KubeCon, ONS) are closing in the next two weeks: pick a couple to highlight the work that has been done to date. See [5].<o:p></o:p></span></li></ul>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">[1] - </span>
<a href="https://drive.google.com/file/d/1aEzHHesNoVsZOqK-IIP0g2bbPIRw48Ev/view"><span style="color:#1155CC">https://drive.google.com/file/d/1aEzHHesNoVsZOqK-IIP0g2bbPIRw48Ev/view</span></a></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">[2] - </span>
<a href="https://asciinema.org/a/219790"><span style="color:#1155CC">https://asciinema.org/a/219790</span></a></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">[3] - </span>
<a href="https://etherpad.openstack.org/p/katacontainers-2019-architecture-committee-mtgs"><span style="color:#1155CC">https://etherpad.openstack.org/p/katacontainers-2019-architecture-committee-mtgs</span></a></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="color:black">[4] - https://github.com/kata-containers/runtime/issues/1113</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[5] - <span style="color:black">
<a href="https://etherpad.openstack.org/p/KataTalks2019"><span style="color:purple">https://etherpad.openstack.org/p/KataTalks2019</span></a><o:p></o:p></span></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</body>
</html>