<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"> <head> <meta name="viewport" content="width=device-width,initial-scale=1"> <!--[if gte mso 9]>
<xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
<o:PixelsPerInch>96</o:PixelsPerInch>
</o:OfficeDocumentSettings>
</xml>
<![endif]--> <style type="text/css">* a:hover{cursor:pointer;}</style> <style>body {-webkit-animation:bugfix infinite 1s;}@-webkit-keyframes bugfix {from {position:relative;}to {position:relative;}}</style> </head> <body style="word-wrap:normal;word-break:break-word"> <style>a {word-wrap:normal;word-break:break-word;}.background-contain {background-size:contain;}@media only screen and (max-width:600px) {.container {-webkit-text-size-adjust:none !important;}.container,.palm-one-whole {width:100% !important;min-width:100% !important;}.palm-one-half {width:50% !important;min-width:50% !important;box-sizing:border-box;}blockquote .container,blockquote .container div,blockquote .container table {width:auto !important;min-width:0 !important;position:relative !important;}img {max-width:100%;}.border-outer,.border-middle,.border-inner,.inner,[title="separator"] {width:100% !important;}.innercell {padding:8px !important;}.palm-block {display:block !important;}td.palm-one-whole {display:inline-block !important;padding:0;}td.palm-one-whole:first-child:not(:only-child) {margin-bottom:16px;}td.hostname {padding-top:3px !important;}}@media only screen and (min-width:601px) {.preview-card {max-width:600px !important;}}@media only screen and (min-device-width :320px) and (max-device-width :568px),only screen and (min-device-width :768px) and (max-device-width :1024px),only screen and (max-device-width:640px),only screen and (max-device-width:667px),only screen and (max-width:480px){.container {width:100% !important;min-width:100% !important;}.p,.small,li,font[size="2"],font[size="3"] {font-size:1em !important;}}@media only screen and (min-device-width :320px) and (max-device-width :568px),only screen and (min-device-width :768px) and (max-device-width :1024px),only screen and (min-device-width :1224px) {.message-wrapper {padding-top:6px;}.apple-only[style] {display:block !important;max-height:none !important;line-height:normal !important;overflow:visible !important;height:auto !important;width:100% !important;position:relative !important;}.no-apple {display:none !important;}form {font-size:inherit;}input[type="text"] {height:43px;padding-left:4px !important;}button:hover {cursor:pointer;}}@media only screen and (min-device-width :1224px) {.apple-mail-form {display:block !important;background-color:white !important;}}* [office365] .outlook-com-hidden {display:none !important;}* [office365] .outlook-com-button {display:block;}* [office365] .outlook-com-only {display:block !important;max-height:none !important;line-height:normal !important;overflow:visible !important;height:auto !important;width:100% !important;position:relative !important;}.ExternalClass {width:100%;}.ExternalClass .outlook-com-button {display:block;}.ExternalClass button {height:auto;}.ExternalClass .outlook-com-hidden {display:none !important;}.ExternalClass .outlook-com-only {display:block !important;max-height:none !important;line-height:normal !important;overflow:visible !important;height:auto !important;width:100% !important;position:relative !important;}.ExternalClass .ecxlabels {display:none !important;}.ExternalClass .ecxlabels {display:none !important;}.ExternalClass .ecxarrow {display:none !important;}.ExternalClass cite >div + div {padding:0 0 4px 0;}.ExternalClass .h1 {padding-bottom:5px;}.ExternalClass .h2 {padding-bottom:5px;}.ExternalClass .h3 {padding-bottom:5px;}.ExternalClass [lang="brand-pinterest"] {width:280px !important;}</style> <!--[if (gte mso 9)|(IE)]>
<style>
a, body {
font-family: 'Calibri', Arial, sans-serif;
}
img {
border: none !important;
-ms-interpolation-mode:bicubic;
}
td {
mso-line-height-rule:exactly !important;
}
.mso-card-inner table {
border-collapse: collapse !important;
mso-table-lspace:0pt;
mso-table-rspace:0pt;
vertical-align: top;
}
.outlook-com-only {
display: none !important;
font-size: 0 !important;
}
#mso-one-whole {
width: 100% !important;
}
.border-outer,
.border-middle,
.border-inner {
border: none !important;
}
.border-middle,
.border-inner {
width: 100% !important;
}
.mso-border-outer,
.mso-border-middle,
.mso-border-inner {
padding: 1px;
}
.mso-border-outer { background-color: rgb(245, 255, 255); }
.mso-border-middle { background-color: rgb(223, 246, 255); }
.mso-border-inner { background-color: rgb(153, 176, 225); }
</style>
<![endif]--> <table class="container" lang="container" dir="ltr" border="0" cellpadding="0" cellspacing="0" valign="top" style="width:100%"> <tr> <td valign="top" class="message-wrapper" style="color:#222;font-family:arial,sans-serif"> <!--[if mso]><table border="0" cellpadding="0" cellspacing="0" valign="top" style="border-collapse:separate;"><tr><td valign="top"><![endif]--> <div>Hi Jesse,</div><div><br></div><div>As Sebastien said, the memory encrypt is an exciting feature for VMs.</div><div><br></div><div>Based on the discuss of runtime merging proposal [1], the kata-runtime 1.0 will based on the runV </div><div>hypervisor drivers [2] and vm factory [3]. And we should consider if this should be enable on all qemu</div><div>based drivers if CPU supports this feature, or implemented as a new hypervisor which let users</div><div>decide.</div><div><br></div><div>Looks like, by enabling the memory encryption, we will get</div><div><br></div><div>- stronger isolation we want</div><div>- bigger memory footprint (vm clone and dax won't work, and ksm won't work as well?)</div><div><br></div><div>Will read the references in detail after today's meeting. Will you attend the online meeting[4], Jesse?</div><div><br></div><div>[1] <a href="https://github.com/kata-containers/runtime/issues/33">https://github.com/kata-containers/runtime/issues/33</a></div><div>[2] https://github.com/hyperhq/runv/tree/master/hypervisor</div><div>[3] https://github.com/hyperhq/runv/tree/master/factory</div><div>[4] https://etherpad.openstack.org/p/katacontainers-2018-architecture-committee-mtgs</div><div><br></div><div>-Xu</div><img align="left" width="0" height="0" style="border:0;width:0px;height:0px" src="https://track.mixmax.com/api/track/v2/3FndH1zO4SHg5tYq3/gIoNnLyVGc5hGQ1hnI" alt=""> <!--[if mso]></td></tr></table><![endif]--> </td> </tr> </table> <div> <div> <p data-m-apply-default-font="true"><br></p> <div class="gmail_extra"> <p data-m-apply-default-font="true"><br></p> <div class="gmail_quote"> On Thu, Feb 22, 2018 6:48 AM, Larrew, Jesse <span dir="ltr"> <a href="mailto:Jesse.Larrew@amd.com" target="_blank">Jesse.Larrew@amd.com</a></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <u></u> <p>Hi Sebastien,</p><p><br></p><p>Thanks for the pointer to the virtcontainers PR. I'll keep an eye on it as you suggested.</p><p><br></p><p>That's interesting that you plan to use vanilla qemu for Kata Containers. Do you intend to upstream the "nofw" and "static-prt" accelerators from qemu-lite? Or will those optimizations be abandoned? They seem like a clever solution for reducing the guest boot time.</p><p><br></p><p>Sincerely,</p><p>Jesse</p><p><br></p><p>> -----Original Message-----</p><p>> From: Boeuf, Sebastien [mailto:sebastien.boeuf@intel.com]</p><p>> Sent: Wednesday, February 21, 2018 4:31 PM</p><p>> To: Larrew, Jesse <Jesse.Larrew@amd.com>; kata-</p><p>> dev@lists.katacontainers.io</p><p>> Cc: Hollingsworth, Brent <brent.hollingsworth@amd.com>; Kaplan, David</p><p>> <David.Kaplan@amd.com>; Woller, Thomas <thomas.woller@amd.com></p><p>> Subject: RE: Kata with AMD Secure Encrypted Virtualization (SEV)</p><p>> </p><p>> Hi Jesse,</p><p>> </p><p>> This is very exciting that you have been able to get Clear Containers working</p><p>> with this new AMD technology. I am sure this is something Kata Containers</p><p>> will need since the goal of this project is to run virtualized and secure</p><p>> containers for any architecture able to do so. There are currently some</p><p>> discussions about using virtcontainers as the core API/library for Kata</p><p>> Containers runtime, and as an example of the way you could contribute to</p><p>> this, here is the link to the recent PR that has been raised to bring simple</p><p>> support for ARM architecture (through Qemu):</p><p>> https://github.com/containers/virtcontainers/pull/614</p><p>> </p><p>> If it gets confirmed that virtcontainers will become part of the Kata</p><p>> Containers runtime, then I would suggest that you could raise a similar PR in</p><p>> order to support AMD.</p><p>> </p><p>> Don't forget to add some documentation about constraints and limitations</p><p>> for users that might want to use your AMD support, especially regarding the</p><p>> host kernel version. As long as you manage to get the Qemu patches merged</p><p>> into upstream, everything will be fine since the goal is to rely on a vanilla</p><p>> version of Qemu for Kata.</p><p>> </p><p>> Thanks,</p><p>> Sebastien</p><p>> ________________________________________</p><p>> From: Larrew, Jesse [Jesse.Larrew@amd.com]</p><p>> Sent: Wednesday, February 21, 2018 2:06 PM</p><p>> To: kata-dev@lists.katacontainers.io</p><p>> Cc: Hollingsworth, Brent; Kaplan, David; Woller, Thomas</p><p>> Subject: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)</p><p>> </p><p>> Hi all,</p><p>> </p><p>> The virtualization instructions in the latest AMD EPYC server processors have</p><p>> been enhanced with a memory encryption feature that could provide</p><p>> projects like Kata Containers with unique security capabilities compared to</p><p>> their non-virtualized counterparts. We call this feature Secure Encrypted</p><p>> Virtualization (or SEV) and would be interested in collaborating with the Kata</p><p>> Container community to enable support for containers with encrypted</p><p>> memory.</p><p>> </p><p>> In short, we've added an inline AES engine to our memory controller that</p><p>> encrypts data written to system DRAM and decrypts data read from DRAM.</p><p>> The encryption keys are generated from a TRNG in the onboard AMD Secure</p><p>> Processor (SP) and programmed into the memory controller as needed in a</p><p>> manner that is never visible to software. Additionally, our virtualization</p><p>> instructions have been enhanced to be able to associate a VM ASID with a</p><p>> unique encryption key, so each VM (or container) can keep the contents of</p><p>> its memory confidential from the host and/or other tenant VMs/containers.</p><p>> The guest kernel can choose which pages to encrypt and which to share with</p><p>> the host by setting a bit in the guest page tables, which puts the guest in</p><p>> complete control of the visibility of their data in the cloud. More information</p><p>> can be found in our Memory Encryption whitepaper [1] and in the</p><p>> Architecture Programmer's Manual [2].</p><p>> </p><p>> Linux kernel support for SEV has been merged into the 4.15 and upcoming</p><p>> 4.16 kernels. OVMF BIOS support has been merged as well. The qemu</p><p>> changes are still being upstreamed, but the patches are available for testing</p><p>> on github [3].</p><p>> </p><p>> With the above support in place, we have developed a proof-of-concept</p><p>> demo that is based on Clear Containers. Since the Clear Containers project</p><p>> had already done the heavy lifting to run container workloads inside of a VM,</p><p>> it was rather straightforward to add support to encrypt those VMs using SEV.</p><p>> The required changes are summarized below:</p><p>> * Container kernel:</p><p>> - Add SEV support patches from the Linux kernel repo in [3].</p><p>> - Force virtio to use the DMA API (and hence SWIOTLB) when</p><p>> adding/removing buffers to/from the virtio ring buffer.</p><p>> - SEV requires a memory copy in order to perform the encryption, so</p><p>> zero-copy solutions using DAX for the container initial user space will not</p><p>> work.</p><p>> + Build in a small initramfs to use as the guest kernel initial user</p><p>> space.</p><p>> + Include the updated container agent binary and supporting libs</p><p>> (~14MB total).</p><p>> * Container agent:</p><p>> - Update the agent not to use the pivot_root() method from the</p><p>> initramfs environment, and perform the pivot to the container workload</p><p>> filesystem manually instead.</p><p>> * Container runtime:</p><p>> - Add the new qemu command line options for starting an SEV guest.</p><p>> * Qemu-lite:</p><p>> - Add the SEV support patches from the qemu repo in [3].</p><p>> </p><p>> With the above changes, we are able to start docker containers inside of SEV-</p><p>> protected VMs:</p><p>> </p><p>> amd@pecanporter:~/src/git$ sudo docker run -ti --runtime sev-runtime</p><p>> busybox sh / # whoami root / # dmesg | grep SEV</p><p>> [ 0.001000] AMD Secure Encrypted Virtualization (SEV) active</p><p>> [ 0.219196] SEV is active and system is using DMA bounce buffers</p><p>> </p><p>> As a check, dumping the contents of a page from the qemu heap reveals</p><p>> plaintext data:</p><p>> </p><p>> amd@pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem</p><p>> bs=4096 count=1 skip=23058854513 | xxd | tail</p><p>> dd: /proc/38572/mem: cannot skip to specified offset</p><p>> 1+0 records in</p><p>> 1+0 records out</p><p>> 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.8437e-05 s, 46.3 MB/s</p><p>> 00000f60: 0000 0000 0000 0000 0000 0000 0000 0000 ................</p><p>> 00000f70: 0000 0000 0000 0000 0000 0000 0000 0000 ................</p><p>> 00000f80: 0000 0000 0000 0000 7100 0000 0000 0000 ........q.......</p><p>> 00000f90: 2f72 756e 2f76 6972 7463 6f6e 7461 696e /run/virtcontain</p><p>> 00000fa0: 6572 732f 706f 6473 2f33 3565 3233 6565 ers/pods/35e23ee</p><p>> 00000fb0: 3330 6466 6237 3266 3135 3730 6265 3432 30dfb72f1570be42</p><p>> 00000fc0: 6665 3165 6331 3366 3331 3332 6138 6133 fe1ec13f3132a8a3</p><p>> 00000fd0: 6463 3336 6463 3131 6235 6365 3837 6236 dc36dc11b5ce87b6</p><p>> 00000fe0: 3437 3930 3736 6339 612f 636f 6e73 6f6c 479076c9a/consol</p><p>> 00000ff0: 652e 736f 636b 0000 0104 0000 0000 0000 e.sock..........</p><p>> </p><p>> However, any attempt to read the container memory from the host</p><p>> produces only ciphertext:</p><p>> </p><p>> amd@pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem</p><p>> bs=4096 count=1 skip=34165702144 | xxd | head</p><p>> dd: /proc/38572/mem: cannot skip to specified offset</p><p>> 1+0 records in</p><p>> 1+0 records out</p><p>> 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.9039e-05 s, 46.0 MB/s</p><p>> 00000000: e9b8 e14d c063 ee18 fd85 5ecc 4d1f c1a2 ...M.c....^.M...</p><p>> 00000010: d681 cdf2 259b a97e c43b 5cde bf9e 695b ....%..~.;\...i[</p><p>> 00000020: db3c 778b 8e77 89f4 f795 e5a6 9ebb 765b .<w..w........v[</p><p>> 00000030: 0905 e1d3 c7ec 6f2b bada ed15 b2e0 db7f ......o+........</p><p>> 00000040: d5e9 6d15 cf28 0ca1 4a45 3b9a 1779 e3ff ..m..(..JE;..y..</p><p>> 00000050: 9ee0 b562 2311 6e5a e972 4c06 3f6a 6ebf ...b#.nZ.rL.?jn.</p><p>> 00000060: 909a 88ea 737a 6226 5d87 8968 b31b d096 ....szb&]..h....</p><p>> 00000070: 9360 cbb0 4f34 d811 89a7 048f 01e8 d19e .`..O4..........</p><p>> 00000080: 5429 995a 4de0 6fba 3360 8bb4 a2dc 17e4 T).ZM.o.3`......</p><p>> 00000090: 80f5 6657 9fd7 0347 e78d 4d13 6b6c c649 ..fW...G..M.kl.I</p><p>> </p><p>> Our threat model is to allow container workloads to reduce their risk</p><p>> exposure to security vulnerabilities in the hosting environment, which seems</p><p>> to overlap nicely with the threat model of Kata Containers. Is this a feature</p><p>> that the Kata community would find useful? If so, we would be very</p><p>> interested to work with the community to enable SEV memory encryption</p><p>> for Kata Containers. Any and all feedback is welcome!</p><p>> </p><p>> Thanks!</p><p>> </p><p>> References:</p><p>> [1] AMD Memory Encryption Whitepaper:</p><p>> http://amd-dev.wpengine.netdna-</p><p>> cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepape</p><p>> r_v7-Public.pdf</p><p>> [2] AMD64 Architecture Programmer's Manual Volume 2: System</p><p>> Programming:</p><p>> http://developer.amd.com/wordpress/media/2012/10/24593_APM_v21.pdf</p><p>> [3] AMD SEV github repo:</p><p>> https://github.com/AMDESE/AMDSEV</p><p>> </p><p>> Sincerely,</p><p>> </p><p>> Jesse Larrew</p><p>> MTS Software Security Architect</p><p>> AMD Security Architecture R&D</p><p>> jesse.larrew@amd.com</p><p>> O: +(1) 512-602-0092 (x50092)</p><p>> M: +(1) 512-791-4852</p><p>> </p><p>> </p><p>> _______________________________________________</p><p>> kata-dev mailing list</p><p>> kata-dev@lists.katacontainers.io</p><p>> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev</p><p><br></p><p>_______________________________________________</p><p>kata-dev mailing list</p><p>kata-dev@lists.katacontainers.io</p><p>http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev</p><p><br></p> </blockquote> </div> </div> </div> </div> <br><br><div style="font-size:12.8px">--</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Xu Wang</div><div style="font-size:12.8px">CTO & Cofounder, Hyper</div><div style="font-size:12.8px">github/twitter/wechat: @gnawux</div><div style="font-size:12.8px"><a href="http://hyper.sh/" target="_blank" style="color:rgb(17,85,204)">http://hyper.sh</a></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Hyper_: Make VM run like container</div> </body> </html>