<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:DengXian;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:宋体;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:DengXian;
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:DengXian;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="ZH-CN" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">Hi, guys<span class="apple-converted-space"> </span><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">I have Some doubts about virtiofs cache and dax . In the past two days, I have asked a few questions on WeChat and slack</span><span style="font-size:13.5pt;color:black">：</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><a href="https://katacontainers.slack.com/archives/C86U7NZND/p1621580960033100"><span style="color:#954F72">https://katacontainers.slack.com/archives/C86U7NZND/p1621580960033100</span></a><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">, and some experts have answered some of them, which is very useful. But I still hope to get more detailed knowledge and information. @fidencio suggested that I post the question
to the mailing list, so I posted it<o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<b><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">Question 1<span class="apple-converted-space"> </span></span></b><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black">On the official website of virtio-fs, in this article <a href="https://virtio-fs.gitlab.io/howto-qemu.html" target="_blank"><span style="color:#954F72">https://virtio-fs.gitlab.io/howto-qemu.html</span></a>,
it seems that $TESTDIR can set an existing directory on the host , but if I set the source parameter in the /usr/share/defaults/kata-containers/configuration.toml configuration file and specify a directory like </span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#E01E5A">/opt/kata-instance</span></code><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black"> on
the host<br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:black;background:silver"># Format example:</span></code><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black;background:silver"><br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:black;background:silver"># ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]</span></code><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black;background:silver"><br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:black;background:silver">#</span></code><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black;background:silver"><br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:black;background:silver"># see `virtiofsd -h` for possible options.</span></code><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black;background:silver"><br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:black;background:silver">virtio_fs_extra_args = ["-o","--thread-pool-size=1","-o","/opt/kata-instance"]</span></code><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black"> </span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black">kubelet will report error<br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040"> <span style="background:silver">Warning FailedCreatePodSandBox 1s (x14 over 15s) kubelet, k8s05 Failed to create pod sandbox: rpc error: code = Unknown desc =
failed to create containerd task: failed to launch qemu: exit status 1, error messages from qemu log: qemu-system- x86_64: cannot create PID file: Cannot open pid file: No such file or directory</span> </span></code><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040"> </span></code><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<b><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black">Question 2</span></b><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black"><br>
I saw a picture that said that DAX uses the nvdimm device simulated by qemu. It seems to be a device for each VM, which is mapped to the host filesystem buffer cache. For example, there are 100 kata VMs on a single host. I set virtio_fs_cache_size=1024M, Then
the host will use up to 1024M memory, or 1024M*100 memory?Also, can this shmem be writable or read-only? Is it only for kata pod images or all data read and write, including kubernetes persistence volume?</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black">From the official website of qemu, I found that nvdimm is used like this</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#E01E5A;background:silver">-</span></code><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver">machine pc, nvdimm</span></code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver"><br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver">-m $RAM_SIZE, slots=$N,maxmem=$MAX_SIZE</span></code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver"><br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver">-object memory-backend-file,id=mem1,share=on,mem-path=$PATH,size=$NVDIMM_SIZE</span></code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver"><br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver">-device nvdimm,id=nvdimm1,memdev=mem1</span></code><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black"> </span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black">When the kata instance is created, the qemu process can see that it contains a parameter memory-backend-file,<br>
</span><code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040;background:silver">-object memory-backend-file,id=dimm1,size=3072M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1</span></code><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New";color:#404040"><br>
</span><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black">Is this cache dedicated to virtiofs? Where does dimm1 come from? It has nothing to do with nvdimm, right?</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;box-sizing: inherit;counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0;word-spacing:0px">
<span lang="EN-US" style="font-family:"Courier New";color:black"><o:p> </o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black"> </span><span lang="EN-US" style="font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<b><span lang="EN-US" style="font-size:11.5pt;font-family:"Courier New";color:black">Qeustion 3</span></b><span lang="EN-US" style="font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">Now my understanding is that kata has two caches<o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">3.1 Data transmission between guest and virtiofsd requires a cache. The corresponding qemu parameter is<span class="apple-converted-space"> </span><span style="background:silver">-object
memory-backend-file,id=dimm1,size=XXX</span>. I guess the function of this cache may be the memory space of the simulated PCI device? It seems unlikely that it is the vring queue size used by the vhost-user protocol. Sorry, I'm not very proficient in qemu<o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">There is another question. Is the virtiofs cache specially allocated for each VM? If a single host starts many VMs, won't the memory consumption be very large?<o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">3.2 dax, the corresponding parameter is<span class="apple-converted-space"> </span><span style="background:silver">virtio_fs_cache_size,</span><span class="apple-converted-space"> </span>there
is no cache policy control in the kata configuration file.<o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black;background:silver"># Cache mode: virtio_fs_cache</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">It controls the virtiofs cache, not DAX windows, right?<o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">Sorry</span><span style="font-size:13.5pt;color:black">，</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">asked a lot of questions, it looks
a bit scary<span class="apple-converted-space"> </span></span><span style="font-size:13.5pt;color:black">：）</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black">ryo from shanghai 2021-05-21<o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span lang="EN-US" style="font-size:13.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><o:p> </o:p></span></p>
</div>
</body>
</html>