<div>Hi Carlos,</div><div><br></div>Thank you for the information. This is really helpful! <div><br></div><div>I will share my ideas and let u guys know so that this can help make kata better for everyone.</div><div><br></div><div>Regards </div><div><br></div><div>Wilson</div><div><br>On Thursday, May 7, 2020, Venegas Munoz, Jose Carlos <<a href="mailto:jose.carlos.venegas.munoz@intel.com">jose.carlos.venegas.munoz@intel.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="en-MX" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Wilson,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">It is great to know you are working on some POC based in kata and K8s. The new features Bo mentioned will land in Kata 1.11 (about to be released this week) for that release Kata will use the latest release for Cloud-hypervisor
0.7.0.<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="EN-US">Today for Kata in master we already have testing for K8s and containerd as CRI runtime + Cloud-hypervisor.
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">For our current CI testing we are just missing support for hot plug of storage devices. We hope in next release of Ka ta alpha 1.12-alpha0 this could be ready for testing. Additionally we are working to enable End-to-End
Kubernetes testing to increase the coverage of use cases. There are still a few API options that Cloud-hypervisor support and we have not evaluated. But I’d say that after blk/pmem hotplug enabling , we will have parity with Qemu implementation and the use-cases
we have in our CI today. <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Please open any issue you find, we want to make sure cloud-hypervisor and kata has the best experience.
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">You can get some useful information about future fixes, issues, and missing items in the following github project:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://github.com/orgs/kata-containers/projects/21" target="_blank">https://github.com/orgs/kata-<wbr>containers/projects/21</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">A summary of the current state also is documented here:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://github.com/kata-containers/runtime/issues/2545" target="_blank">https://github.com/kata-<wbr>containers/runtime/issues/2545</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Summary of used cloud-hypervisor APIs (and comments why some are not used)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://github.com/kata-containers/runtime/issues/2541" target="_blank">https://github.com/kata-<wbr>containers/runtime/issues/2541</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">To summarize the more relevant missing items are related with hotplug are:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<ul style="margin-top:0cm" type="disc">
<li style="margin-left:0cm"><span lang="EN-US">Pmem/Block device hotplug
<u></u><u></u></span></li></ul>
<p class="MsoNormal"><span lang="EN-US"><a href="https://github.com/kata-containers/runtime/issues/2541" target="_blank">https://github.com/kata-<wbr>containers/runtime/issues/2541</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://github.com/kata-containers/runtime/issues/2575" target="_blank">https://github.com/kata-<wbr>containers/runtime/issues/2575</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<ul style="margin-top:0cm" type="disc">
<li style="margin-left:0cm"><span lang="EN-US">Net hotplug : First we need to add some testing (at least with qemu) to know where could be used in the k8s workflow, then we could enable it.<u></u><u></u></span></li></ul>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">You can track the CI stability of master and the current set of jobs that we use for test cloud-hypervisor integration.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://github.com/kata-containers/ci/wiki/cloud-hypervisor" target="_blank">https://github.com/kata-<wbr>containers/ci/wiki/cloud-<wbr>hypervisor</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Cheers!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">-<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Carlos<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">"<a href="mailto:wilsonny371@gmail.com" target="_blank">wilsonny371@gmail.com</a>" <<a href="mailto:wilsonny371@gmail.com" target="_blank">wilsonny371@gmail.com</a>><br>
<b>Date: </b>Wednesday, 6 May 2020, 18:25<br>
<b>To: </b>"Chen, Bo2" <<a href="mailto:chen.bo@intel.com" target="_blank">chen.bo@intel.com</a>><br>
<b>Cc: </b>"Venegas Munoz, Jose Carlos" <<a href="mailto:jose.carlos.venegas.munoz@intel.com" target="_blank">jose.carlos.venegas.munoz@<wbr>intel.com</a>>, "<a href="mailto:kata-dev@lists.katacontainers.io" target="_blank">kata-dev@lists.<wbr>katacontainers.io</a>" <<a href="mailto:kata-dev@lists.katacontainers.io" target="_blank">kata-dev@lists.<wbr>katacontainers.io</a>><br>
<b>Subject: </b>Re: [kata-dev] hot plug/unplug for cloud hypervisor support<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">Bo,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">Thanks for your reply. <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">My colleagues and I are working on some POC and it is based on Kata & K8s. The new features you mentioned will definitely help us. However, we are still at the very early stage
and trying to prove that cloud hypervisor is truly what we are looking for compared to firecracker which does not have these features yet.<br clear="all">
<u></u><u></u></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Regards<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">Wilson<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, May 6, 2020 at 4:00 PM Chen, Bo2 <<a href="mailto:chen.bo@intel.com" target="_blank">chen.bo@intel.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Wilson,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for promoting this question. Yes, that is our plan. Along with the supported device hot-plug we have in Kata with cloud-hypervisor (e.g. vcpu/memory/vfio),
we are enabling block (‘disk ‘and ‘pmem’) device hot-plug very soon. For few other HTTP APIs related to device hotplug that cloud-hypervisor recently added (e.g. ‘/vm.add-fs’, ‘/vm.add-net’, and ‘/vm.add-vsock’), we will need to look into them and add support
if there are use cases in Kata.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Carlos, would you please comment on the device hot-unplug side? Base on my limited understanding, we don’t have much use-case for device hot-unplug in Kata, right?
Also, please correct me and add more for the device hot-plug side. Thanks a lot.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I hope it helps and makes sense to you. Any more questions or comments, please don’t hesitate to share.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Best,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Bo<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US">
<a href="mailto:wilsonny371@gmail.com" target="_blank">wilsonny371@gmail.com</a> <<a href="mailto:wilsonny371@gmail.com" target="_blank">wilsonny371@gmail.com</a>>
<br>
<b>Sent:</b> Wednesday, May 6, 2020 3:35 PM<br>
<b>To:</b> <a href="mailto:kata-dev@lists.katacontainers.io" target="_blank">kata-dev@lists.katacontainers.<wbr>io</a><br>
<b>Subject:</b> [kata-dev] hot plug/unplug for cloud hypervisor support<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif">Hi guys,</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif">I have a question regarding Kata Containers hotplug support for cloud hypervisor.</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif">I saw recently cloud hypervisor added support for several different virtio devices hotplug. But the
support for this is not yet there in Kata Containers.</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif">Do we have a plan to add this support soon?</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif">Thanks</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Verdana",sans-serif">Wilson</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div><br><br>-- <br>Sent from my iPhone 6 Plus <br>