Hi, guys

I have Some doubts about virtiofs cache and dax . In the past two days, I have asked a few questions on WeChat and slack：https://katacontainers.slack.com/archives/C86U7NZND/p1621580960033100

, and some experts have answered some of them, which is very useful. But I still hope to get more detailed knowledge and information. @fidencio suggested that I post the question to the mailing list, so I posted it

Question 1

On the official website of virtio-fs, in this article https://virtio-fs.gitlab.io/howto-qemu.html, it seems that $TESTDIR can  set an existing directory on the host , but if I set the source parameter in the /usr/share/defaults/kata-containers/configuration.toml  configuration file and specify a directory like /opt/kata-instance on the host
# Format example:
# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]
#
# see `virtiofsd -h` for possible options.
virtio_fs_extra_args = ["-o","--thread-pool-size=1","-o","/opt/kata-instance"]

kubelet will report error
Warning FailedCreatePodSandBox 1s (x14 over 15s) kubelet, k8s05 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to launch qemu: exit status 1, error messages from qemu log: qemu-system- x86_64: cannot create PID file: Cannot open pid file: No such file or directory

Question 2
I saw a picture that said that DAX uses the nvdimm device simulated by qemu. It seems to be a device for each VM, which is mapped to the host filesystem buffer cache. For example, there are 100 kata VMs on a single host. I set virtio_fs_cache_size=1024M, Then the host will use up to 1024M memory, or 1024M*100 memory?Also, can this shmem be writable or read-only? Is it only for kata pod images or all data read and write, including kubernetes persistence volume?

From the official website of qemu, I found that nvdimm is used like this

-machine pc, nvdimm
-m $RAM_SIZE, slots=$N,maxmem=$MAX_SIZE
-object memory-backend-file,id=mem1,share=on,mem-path=$PATH,size=$NVDIMM_SIZE
-device nvdimm,id=nvdimm1,memdev=mem1

When the kata instance is created, the qemu process can see that it contains a parameter memory-backend-file,
-object memory-backend-file,id=dimm1,size=3072M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1
Is this cache dedicated to virtiofs? Where does dimm1 come from? It has nothing to do with nvdimm, right?

Qeustion 3

Now my understanding is that kata has two caches

3.1 Data transmission between guest and virtiofsd requires a cache. The corresponding qemu parameter is -object memory-backend-file,id=dimm1,size=XXX. I guess the function of this cache may be the memory space of the simulated PCI device? It seems unlikely that it is the vring queue size used by the vhost-user protocol. Sorry, I'm not very proficient in qemu

There is another question. Is the virtiofs cache specially allocated for each VM? If a single host starts many VMs, won't the memory consumption be very large?

3.2 dax, the corresponding parameter is virtio_fs_cache_size, there is no cache policy control in the kata configuration file.

# Cache mode: virtio_fs_cache

It controls the virtiofs cache, not DAX windows, right?

Sorry，asked a lot of questions, it looks a bit scary ：）

ryo from shanghai  2021-05-21