Open Stack

* Qixuan Wu (qixuan.wu at linux.alibaba.com) wrote:
> 
> 
> 于 2018/9/25 上午12:14, Dr. David Alan Gilbert 写道:
> > * Qixuan Wu (qixuan.wu at linux.alibaba.com) wrote:
> > > 
> > > 
> > > 于 2018/9/24 下午11:53, Dr. David Alan Gilbert 写道:
> > > > * Qixuan Wu (qixuan.wu at linux.alibaba.com) wrote:
> > > > > > We're currently experimenting with something a bit different;
> > > > > > we've got a setup that uses a modified version of the FUSE protocol
> > > > > > running over vhost-user;  it's:
> > > > > >      a) Got the filesystem access split out of qemu into a separate daemon
> > > > > >          - that's just a modified version of a normal FUSE filesystem daemon
> > > > > >          with the nice bit being that since it's a separate process you
> > > > > >          can do whatever isolation on it you want.
> > > > > >      b) But the latency is low because vhost-user means the daemon can read
> > > > > >         the request queue straight out of the guest memory
> > > > > >      c) We've got a setup with DAX so that the files are mapped straight
> > > > > >         into guest address space, so the overhead is very low for large
> > > > > >         files.
> > > > > That's so cool. I guess it will not use virtio. And this way maybe the new
> > > > > para-virtualization method, it's specific about the file system for the data
> > > > > shared between guest and host.
> > > > 
> > > > It does use virtio!  It's basically just the existing FUSE protocol
> > > > carried over virtio; it's got some tweaks to allow the direct mappings
> > > > and to deal with some difference sin the setup.
> > > > It uses the existing vhost-user implementation of virtio (just like
> > > > vhost-user for network does virtio for dpdk).
> > > > 
> > > > > >      d) We've got a caching scheme for metadata, which again removes a lot
> > > > > >         of latency.
> > > > > >      e) We've got some patches to use it in KATA; I can start a basic KATA
> > > > > >         guest with it.
> > > > > > 
> > > > > > This is the first public mention of it because I didn't want you waiting
> > > > > > for a reply; but our code is still rather messy and experimental; give
> > > > > > us a few weeks and as soon as it survives some smoke tests we'll make
> > > > > > the code public.
> > > > > > 
> > > > > > Because we're reusing both FUSE and vhost-user the kernel changes are
> > > > > > quite small, as are the qemu changes.
> > > > > > 
> > > > > > I realise that's not much detail yet; we're starting to write some of it
> > > > > > up; feel free to ask any specifics.
> > > > > > 
> > > > > 
> > > > > Thank for the replay. Seems that the file data are mmapping direct, but the
> > > > > control plane, like metadata are still using some other simple protocal,
> > > > > maybe new protocal ?
> > > > 
> > > > The control plane again is basically just the existing FUSE protocol;
> > > > but we've got a shared mmap'd region for a fast lookup for some of the
> > > > metadata.
> > > > 
> > > > > Because 9p and nfs are very complex, they are not
> > > > > developed for the file sharing between guest and host. I always hope there
> > > > > is a simple file sharing protocal. I am very looking forward for the code.
> > > > > :-).
> > > > 
> > > > Glad you like the sound of it;  we'll try and get it out ASAP.
> > > > 
> > > 
> > > Though has some doubts about it, anyway, seems like it's faster and simpler
> > > than 9pfs and nfs+vsock. It's a good news for kata user.
> > 
> > Please ask about your doubts; I'd like to make sure we have good
> > answers to them.
> 
> My doubts are:
> 
> 1. Is the Filesystem in Userspace (FUSE) used in Guest os or Host os?

It's between the guest OS and the host qemu+daemon. The host OS doesn't
see it.

> 2. As my understand, FUSE is the mechanism used between user space and
> kernel space, not a protocal. So I cannot understand how create or
> unlink command be transfered from guest to host over virtio. So i did
> not understant "FUSE protocol over virtio".

Ignoring this work; the way FUSE works is that:

    1) application -> syscalls to kernel

    2) kernel translates those to a message stream over an fd

    3) A daemon running as a normal process under the same kernel
    reads commands from that fd and passes data back to the kernel

now we swivel this around a bit:

    a) Guest application -> syscalls to guest kernel

    b) guest kernel translates those to a message stream - this time
       over a virtio command stream.

    c) A daemon connected to qemu via vhost-user reads that
       command stream.

so it's actually pretty much the same; but we've replaced
the fd used between the kernel and the daemon by a virtio transport.

> 3. Did you test the performance compared to 9P?

Only a little; at the moment our code is full of debug and we're
just trying to get it hang together to run benchmarks solidly.
It's looking promising though; there's a couple of things we need
to fix but it's getting there.

> Thanks for replay and clarification very much.

No problem.

Dave

> Regards
> Qixuan.
--
Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK