Open Stack

On 2021/5/25 16:18, 李瑞友 wrote:
> Hi, guys
> 
> I have Some doubts about virtiofs cache and dax . In the past two days, 
> I have asked a few questions on WeChat and 
> slack：https://katacontainers.slack.com/archives/C86U7NZND/p1621580960033100
> 
> , and some experts have answered some of them, which is very useful. But 
> I still hope to get more detailed knowledge and information. @fidencio 
> suggested that I post the question to the mailing list, so I posted it
> 
> *Question 1*
> 
> On the official website of virtio-fs, in this article 
> https://virtio-fs.gitlab.io/howto-qemu.html, it seems that $TESTDIR can  
> set an existing directory on the host , but if I set the source 
> parameter in the /usr/share/defaults/kata-containers/configuration.toml  
> configuration file and specify a directory like |/opt/kata-instance| on 
> the host
> |# Format example:|
> |# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]|
> |#|
> |# see `virtiofsd -h` for possible options.|
> |virtio_fs_extra_args = 
> ["-o","--thread-pool-size=1","-o","/opt/kata-instance"]|
> 
> kubelet will report error
> |Warning FailedCreatePodSandBox 1s (x14 over 15s) kubelet, k8s05 Failed 
> to create pod sandbox: rpc error: code = Unknown desc = failed to create 
> containerd task: failed to launch qemu: exit status 1, error messages 
> from qemu log: qemu-system- x86_64: cannot create PID file: Cannot open 
> pid file: No such file or directory |
> The $TESTDIR in the virtio-fs example is to show to users how to test 
virtio-fs directly. With Kata Containers, users do not have to set it 
directly and in fact Kata runtime would need the `source=xxx` parameter 
and if users specify it directly in the config file, virtiofsd might 
fail to start and it results in the error you see above.

> ||
> 
> *Question 2*
> I saw a picture that said that DAX uses the nvdimm device simulated by 
> qemu. It seems to be a device for each VM, which is mapped to the host 
> filesystem buffer cache. For example, there are 100 kata VMs on a single 
> host. I set virtio_fs_cache_size=1024M, Then the host will use up to 
> 1024M memory, or 1024M*100 memory?
1024*100. There is no sharing between dax windows unless when they map 
the same file on the host.

> Also, can this shmem be writable or 
> read-only?
It is writable for the guest.

> Is it only for kata pod images or all data read and write, 
> including kubernetes persistence volume?
Anything that goes through virtio-fs. So it is for images and volumes.

> 
>  From the official website of qemu, I found that nvdimm is used like this
> 
> |-||machine pc, nvdimm|
> |-m $RAM_SIZE, slots=$N,maxmem=$MAX_SIZE|
> |-object 
> memory-backend-file,id=mem1,share=on,mem-path=$PATH,size=$NVDIMM_SIZE|
> |-device nvdimm,id=nvdimm1,memdev=mem1|
> 
> When the kata instance is created, the qemu process can see that it 
> contains a parameter memory-backend-file,
> |-object 
> memory-backend-file,id=dimm1,size=3072M,mem-path=/dev/shm,share=on -numa 
> node,memdev=dimm1|
> Is this cache dedicated to virtiofs? Where does dimm1 come from? It has 
> nothing to do with nvdimm, right?
Yup. It is *NOT* an nvdimm device but just a normal file backed memory 
dim for the guest to use. And the dimm is not dedicated to virtiofs 
either. It is the guest's memory dimm.

> 
> *Qeustion 3*
> 
> Now my understanding is that kata has two caches
> 
> 3.1 Data transmission between guest and virtiofsd requires a cache. The 
> corresponding qemu parameter is-object 
> memory-backend-file,id=dimm1,size=XXX. I guess the function of this 
> cache may be the memory space of the simulated PCI device? It seems 
> unlikely that it is the vring queue size used by the vhost-user 
> protocol. Sorry, I'm not very proficient in qemu
> 
Nop, it is guest's memory. Not *JUST* vrings.

> There is another question. Is the virtiofs cache specially allocated for 
> each VM? If a single host starts many VMs, won't the memory consumption 
> be very large?
It could be.

> 
> 3.2 dax, the corresponding parameter isvirtio_fs_cache_size,there is no 
> cache policy control in the kata configuration file.
> 
> # Cache mode: virtio_fs_cache
> 
> It controls the virtiofs cache, not DAX windows, right?
> 
DAX is just a page mapping mechanism. It doesn't have any cache policy. 
It is guest's page cache file virtiofs point of view. And the file 
system cache policy is controlled by guest virtiofs file system.

Hope I answered your questions ;)

Cheers,
Tao

> Sorry，asked a lot of questions, it looks a bit scary：）
> 
> ryo from shanghai  2021-05-21
> 
> 
> _______________________________________________
> kata-dev mailing list
> kata-dev at lists.katacontainers.io
> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
>