<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hey Jesse,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I had the chance to meet Brent @ Red Hat Summit yesterday, and this was a good reminder to reach out regarding SEV support in Kata.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">First, as expressed to Brent, this is a sweet use case.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Second, one of the things that has come to mind for me is how we can verify.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our CI is running on a mix of baremetal machines for metrics and machines in the cloud (via Azure) for functional testing. For other architectures, and including this feature, it’d be best to have this exercised in CI. Basically, I’d
want to make sure we can replicate the CI on a single AMD Epyc (or other SEV enabled system) which could help gate our CI process. We have the test setup designed to be easily reproduced, and can work together on getting this setup, assuming we find a machine
which this can run on.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">With this feature enabled in our CI, we’d be able to guarantee that it continues to work (and if it fails, it should be an easy fix).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<br>
Eric<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">"Larrew, Jesse" <Jesse.Larrew@amd.com><br>
<b>Date: </b>Tuesday, May 1, 2018 at 10:28 AM<br>
<b>To: </b>Eric Ernst <eric.ernst@intel.com><br>
<b>Cc: </b>Jon Olson <jonolson@google.com>, "Singh, Brijesh" <brijesh.singh@amd.com>, "Kaplan, David" <David.Kaplan@amd.com>, "Hollingsworth, Brent" <brent.hollingsworth@amd.com>, "kata-dev@lists.katacontainers.io" <kata-dev@lists.katacontainers.io>, "Woller,
Thomas" <thomas.woller@amd.com><br>
<b>Subject: </b>RE: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">Hi Eric,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">I got SEV working with the latest 0.0.1 kata runtime. Is there still a chance of getting this in before the 1.0 release on the 22<sup>nd</sup>? Or are we looking at 1.1.0 at this point?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Sincerely,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Jesse<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b>From:</b> Ernst, Eric [mailto:eric.ernst@intel.com]
<br>
<b>Sent:</b> Saturday, April 7, 2018 9:18 PM<br>
<b>To:</b> Larrew, Jesse <Jesse.Larrew@amd.com><br>
<b>Cc:</b> Jon Olson <jonolson@google.com>; Singh, Brijesh <brijesh.singh@amd.com>; Kaplan, David <David.Kaplan@amd.com>; Hollingsworth, Brent <brent.hollingsworth@amd.com>; kata-dev@lists.katacontainers.io; Woller, Thomas <thomas.woller@amd.com><br>
<b>Subject:</b> Re: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">There isn’t really a deadline. While we are still discussing release cadence for Kata, this seems like a nice feature to get in, perhaps after our initial 1.0 release (targeting ~June 1). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Eric<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>
On Apr 7, 2018, at 6:22 PM, Larrew, Jesse <<a href="mailto:Jesse.Larrew@amd.com">Jesse.Larrew@amd.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi Eric,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">I’m seeking internal approval to contribute. Do you have a deadline for a decision?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Sincerely,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Jesse<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b>From:</b> Ernst, Eric [<a href="mailto:eric.ernst@intel.com">mailto:eric.ernst@intel.com</a>]
<br>
<b>Sent:</b> Friday, April 6, 2018 2:08 PM<br>
<b>To:</b> Larrew, Jesse <<a href="mailto:Jesse.Larrew@amd.com">Jesse.Larrew@amd.com</a>>; Jon Olson <<a href="mailto:jonolson@google.com">jonolson@google.com</a>><br>
<b>Cc:</b> Singh, Brijesh <<a href="mailto:brijesh.singh@amd.com">brijesh.singh@amd.com</a>>; Kaplan, David <<a href="mailto:David.Kaplan@amd.com">David.Kaplan@amd.com</a>>; Hollingsworth, Brent <<a href="mailto:brent.hollingsworth@amd.com">brent.hollingsworth@amd.com</a>>;
<a href="mailto:kata-dev@lists.katacontainers.io">kata-dev@lists.katacontainers.io</a>; Woller, Thomas <<a href="mailto:thomas.woller@amd.com">thomas.woller@amd.com</a>><br>
<b>Subject:</b> Re: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Jesse,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">I wanted to follow up here… I think this is a pretty exciting feature and I wanted to see what next steps are. Is this something that you’re planning to or can start contributing to the Kata project?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Thanks<br>
Eric<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.0in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">"Larrew, Jesse" <<a href="mailto:Jesse.Larrew@amd.com">Jesse.Larrew@amd.com</a>><br>
<b>Date: </b>Friday, February 23, 2018 at 7:11 PM<br>
<b>To: </b>Jon Olson <<a href="mailto:jonolson@google.com">jonolson@google.com</a>><br>
<b>Cc: </b>"Singh, Brijesh" <<a href="mailto:brijesh.singh@amd.com">brijesh.singh@amd.com</a>>, "Kaplan, David" <<a href="mailto:David.Kaplan@amd.com">David.Kaplan@amd.com</a>>, "Hollingsworth, Brent" <<a href="mailto:brent.hollingsworth@amd.com">brent.hollingsworth@amd.com</a>>,
"<a href="mailto:kata-dev@lists.katacontainers.io">kata-dev@lists.katacontainers.io</a>" <<a href="mailto:kata-dev@lists.katacontainers.io">kata-dev@lists.katacontainers.io</a>>, "Woller, Thomas" <<a href="mailto:thomas.woller@amd.com">thomas.woller@amd.com</a>><br>
<b>Subject: </b>Re: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.0in"><b>From:</b> Jon Olson [<a href="mailto:jonolson@google.com">mailto:jonolson@google.com</a>]
<br>
<b>Sent:</b> Friday, February 23, 2018 6:58 PM<br>
<b>To:</b> Larrew, Jesse <<a href="mailto:Jesse.Larrew@amd.com">Jesse.Larrew@amd.com</a>><br>
<b>Cc:</b> Castelino, Manohar R <<a href="mailto:manohar.r.castelino@intel.com">manohar.r.castelino@intel.com</a>>; Samuel Ortiz <<a href="mailto:sameo@linux.intel.com">sameo@linux.intel.com</a>>; Hollingsworth, Brent <<a href="mailto:brent.hollingsworth@amd.com">brent.hollingsworth@amd.com</a>>;
<a href="mailto:kata-dev@lists.katacontainers.io">kata-dev@lists.katacontainers.io</a>; Singh, Brijesh <<a href="mailto:brijesh.singh@amd.com">brijesh.singh@amd.com</a>>; Kaplan, David <<a href="mailto:David.Kaplan@amd.com">David.Kaplan@amd.com</a>>; Woller,
Thomas <<a href="mailto:thomas.woller@amd.com">thomas.woller@amd.com</a>><br>
<b>Subject:</b> Re: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:1.0in">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">On Fri, Feb 23, 2018 at 4:42 PM Larrew, Jesse <<a href="mailto:Jesse.Larrew@amd.com">Jesse.Larrew@amd.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:1.0in">> From: Castelino, Manohar R [mailto:<a href="mailto:manohar.r.castelino@intel.com" target="_blank">manohar.r.castelino@intel.com</a>]<br>
> Sent: Friday, February 23, 2018 3:53 PM<br>
> To: Larrew, Jesse <<a href="mailto:Jesse.Larrew@amd.com" target="_blank">Jesse.Larrew@amd.com</a>>; Samuel Ortiz<br>
> <<a href="mailto:sameo@linux.intel.com" target="_blank">sameo@linux.intel.com</a>><br>
> Cc: Hollingsworth, Brent <<a href="mailto:brent.hollingsworth@amd.com" target="_blank">brent.hollingsworth@amd.com</a>>; Woller, Thomas<br>
> <<a href="mailto:thomas.woller@amd.com" target="_blank">thomas.woller@amd.com</a>>; Kaplan, David <<a href="mailto:David.Kaplan@amd.com" target="_blank">David.Kaplan@amd.com</a>>; kata-<br>
> <a href="mailto:dev@lists.katacontainers.io" target="_blank">dev@lists.katacontainers.io</a><br>
> Subject: RE: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)<br>
><br>
> Jesse,<br>
><br>
> > On EPYC, our IOMMU doesn't yet support SEV, so DMA to/from devices<br>
> needs<br>
> > to be done using unencrypted pages. It was easy to implement this using<br>
> the<br>
> > bounce buffers provided by SWIOTLB. As you guessed, a quick change to<br>
> > vring_use_dma_api() got virtio support working properly:<br>
><br>
> Clear Containers today supports direct device assignment via SRIOV.<br>
> This requires pre-allocation and pinning of VM memory.<br>
> Will this continue to work?<br>
><br>
> Also we have been working on reverse ballooning. i.e. free unused memory<br>
> from the VM back to the host. Is there is a way to get this to work with<br>
> encrypted memory<br>
><br>
> For more details about the patches<br>
> <a href="https://gist.github.com/sboeuf/fc71f0218a81997251ee0d7668df2bd9" target="_blank">
https://gist.github.com/sboeuf/fc71f0218a81997251ee0d7668df2bd9</a><br>
><br>
> -manohar<br>
><br>
><br>
<br>
Hi Manohar,<br>
<br>
SEV also requires the guest memory to be pre-allocated and pinned [1], so that's not a problem. As long as the PF drivers in the guest are using the dma apis, everything should continue to work.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">That's surprising -- I can see where reclaim would be challenging without something like a balloon, but why must the be initially backed? What happens if you leave a page unbacked and attempt to lazily back it
on an EPT fault?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><b><i> </i></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><b><i>[JDL] You’re right Jon. I misspoke above. The memory only needs to be pinned; the backing pages can be faulted in on demand. Sorry for the confusion.</i></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:1.0in">
<br>
Similarly, the reverse ballooning patches should also work with SEV. In fact, I would argue that SEV compliments this feature by ensuring that physical page contents aren't exposed to the host when the guest uses MADV_FREE. I've CC'ed our KVM expert, Brijesh
Singh, just in case he sees something that I missed.<br>
<br>
Sincerely,<br>
Jesse<br>
<br>
[1] In order to ensure that memory blocks with identical data will encrypt to different ciphertext, SEV mixes the physical address into the encryption algorithm. As a result, if a page of memory is moved to a different physical address, it will not decrypt
properly. This also defeats block-move attacks on the guest memory, but it also requires all guest memory to be pinned.<br>
<br>
<br>
<br>
<o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>