Hi, guys
I have Some doubts about virtiofs cache and dax . In the past two days, I have asked a few questions on WeChat and slack:https://katacontainers.slack.com/archives/C86U7NZND/p1621580960033100
, and some experts have answered some of them, which is very useful. But I still hope to get more detailed knowledge and information. @fidencio suggested that I post the question to the mailing list, so I posted it
*Question 1*
On the official website of virtio-fs, in this article https://virtio-fs.gitlab.io/howto-qemu.html, it seems that $TESTDIR can set an existing directory on the host , but if I set the source parameter in the /usr/share/defaults/kata-containers/configuration.toml configuration file and specify a directory like |/opt/kata-instance| on the host |# Format example:| |# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]| |#| |# see `virtiofsd -h` for possible options.| |virtio_fs_extra_args = ["-o","--thread-pool-size=1","-o","/opt/kata-instance"]|
kubelet will report error |Warning FailedCreatePodSandBox 1s (x14 over 15s) kubelet, k8s05 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to launch qemu: exit status 1, error messages from qemu log: qemu-system- x86_64: cannot create PID file: Cannot open pid file: No such file or directory | The $TESTDIR in the virtio-fs example is to show to users how to test virtio-fs directly. With Kata Containers, users do not have to set it
||
*Question 2* I saw a picture that said that DAX uses the nvdimm device simulated by qemu. It seems to be a device for each VM, which is mapped to the host filesystem buffer cache. For example, there are 100 kata VMs on a single host. I set virtio_fs_cache_size=1024M, Then the host will use up to 1024M memory, or 1024M*100 memory? 1024*100. There is no sharing between dax windows unless when they map
On 2021/5/25 16:18, 李瑞友 wrote: directly and in fact Kata runtime would need the `source=xxx` parameter and if users specify it directly in the config file, virtiofsd might fail to start and it results in the error you see above. the same file on the host.
Also, can this shmem be writable or read-only? It is writable for the guest.
Is it only for kata pod images or all data read and write, including kubernetes persistence volume? Anything that goes through virtio-fs. So it is for images and volumes.
From the official website of qemu, I found that nvdimm is used like this
|-||machine pc, nvdimm| |-m $RAM_SIZE, slots=$N,maxmem=$MAX_SIZE| |-object memory-backend-file,id=mem1,share=on,mem-path=$PATH,size=$NVDIMM_SIZE| |-device nvdimm,id=nvdimm1,memdev=mem1|
When the kata instance is created, the qemu process can see that it contains a parameter memory-backend-file, |-object memory-backend-file,id=dimm1,size=3072M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1| Is this cache dedicated to virtiofs? Where does dimm1 come from? It has nothing to do with nvdimm, right?
Yup. It is *NOT* an nvdimm device but just a normal file backed memory dim for the guest to use. And the dimm is not dedicated to virtiofs either. It is the guest's memory dimm.
*Qeustion 3*
Now my understanding is that kata has two caches
3.1 Data transmission between guest and virtiofsd requires a cache. The corresponding qemu parameter is-object memory-backend-file,id=dimm1,size=XXX. I guess the function of this cache may be the memory space of the simulated PCI device? It seems unlikely that it is the vring queue size used by the vhost-user protocol. Sorry, I'm not very proficient in qemu
Nop, it is guest's memory. Not *JUST* vrings.
There is another question. Is the virtiofs cache specially allocated for each VM? If a single host starts many VMs, won't the memory consumption be very large? It could be.
3.2 dax, the corresponding parameter isvirtio_fs_cache_size,there is no cache policy control in the kata configuration file.
# Cache mode: virtio_fs_cache
It controls the virtiofs cache, not DAX windows, right?
DAX is just a page mapping mechanism. It doesn't have any cache policy. It is guest's page cache file virtiofs point of view. And the file system cache policy is controlled by guest virtiofs file system. Hope I answered your questions ;) Cheers, Tao
Sorry,asked a lot of questions, it looks a bit scary:)
ryo from shanghai 2021-05-21
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev