Open Stack

On Thu, Jun 13, 2019 at 8:17 PM Peng Tao <bergwolf at hyper.sh> wrote:
>
> On Thu, Jun 13, 2019 at 7:09 PM Dr. David Alan Gilbert
> <dgilbert at redhat.com> wrote:
> >
> > * Peng Tao (bergwolf at hyper.sh) wrote:
> > > On Thu, Jun 6, 2019 at 1:33 AM Dr. David Alan Gilbert
> > > <dgilbert at redhat.com> wrote:
> > > >
> > > > * Castelino, Manohar R (manohar.r.castelino at intel.com) wrote:
> > > > > > It's very similar to the trick that NEMU uses for templating.
> > > > > > With the x-ignore-shared-ram migration capability enabled, migration will
> > > > > > not write to the migration stream any RAM block that had the
> > > > > > shared=on flag on the qemu commandline.   So you should then be able
> > > > > > to restart from the migration stream and existing RAM image.
> > > > >
> > > > > So does it mean we can drop our vm-templating patches and move to using " x-ignore-shared-ram" on QEMU 4.0.
> > > > >
> > > > > Today we only need two patches, which will come down to a single patch which would bring up closer to upstream QEMU 4.0 which would be ideal.
> > > > > https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/0001-9p-removing-coroutines-of-9p-to-increase-the-I-O-per.patch
> > > > > https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/0002-migration-add-capability-to-bypass-the-shared-memory.patch
> > > >
> > > > Yes, I'm hoping that with 4.0 you can avoid 0002 - but I've not tried
> > > > it; I'd be interested in your results.
> > > >
> > > Hi Dave,
> > >
> > > I gave it a try and failed to do vm templating with x-ignore-shared.
> > > One key difference between x-ignore-shared and Lai's
> > > migrate-bypass-shared patch is that Lai doesn't verify shared memory
> > > block upon ram load. OTOH x-ignore-shared rely on ram share property
> > > to reconstruct the destination guest memory. In kata, we make use of
> > > the fact that destination ram can be private to implement vm
> > > templating feature so that multiple new guests can share map the same
> > > template VM memory privately (memory-backend-file share=off).
> >
> > (Copying in Yury who wrote those patches).
> >
> > > The way we implement vm templating in kata is:
> > > 1. Start the template VM:
> > >    qemu-system-x86 -m 2G \
> > >      -object memory-backend-file,id=mem0,size=2G,share=on,mem-path=/tmpfs/template-memory
> > > \
> > >      -numa node,memdev=mem0
> > > 2. Stop the template VM, set migration bypass-shared-memory
> > > capability, migrate exec:cat>/tmpfs/state, quit it
> > > 3. Start target VM:
> > >    qemu-system-x86 -m 2G \
> > >      -object memory-backend-file,id=mem0,size=2G,share=off,mem-path=/tmpfs/template-memory
> > > \
> > >      -numa node,memdev=mem0 \
> > >      -incoming "exec:cat /tmpfs/template-state"
> > > 4. Start more target VMs like 3
> > >
> > > There are some major differences between above and the example given
> > > by x-ignore-shared patchset[1]:
> > > 1. on target vm, memory-backend-file is mapped privately (share=off)
> > > 2. no migration capability is set on the target vm
> > > 3. it is possible to create multiple target VMs based on the same template VM
> > > I have made some hacky change to make x-ignore-shared work for vm
> > > templating, mostly by reverting the commit "migration: Add
> > > capabilities validation" and reconstructing destination VM ram without
> > > using migration share capability or ram share property[2].
> > >
> > > So I want to ask about your opinion on how to make vm templating work.
> > > Shall we change qemu to load ram without the x-ignore-shared
> > > capability? Or add another capability to implement similar feature
> > > along side x-ignore-shared?
> >
> > Can we first try a different hack; if you set the ignore shared on the
> > destination as well as the source, what happens?  I guess it'll hit one
> > of the errors in  ram_load?  Which one?
> >
> With vanilla 4.0? I got:
>  qemu-system-x86_64: RAM block mem should  be migrated
>  qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
>  qemu-system-x86_64: load of migration failed: Invalid argument
>
> which matches:
> 4341                     if (migrate_ignore_shared()) {
> 4342                         hwaddr addr = qemu_get_be64(f);
> 4343                         bool ignored = qemu_get_byte(f);
> 4344                         if (ignored != ramblock_is_ignored(block)) {
> 4345                             error_report("RAM block %s should %s
> be migrated",
> 4346                                          id, ignored ? "" : "not");
> 4347                             ret = -EINVAL;
>
> If I remove the check it succeeds.
By success, I meant the following steps:
1. Start the template VM:
   qemu-system-x86 -m 2G \
     -object memory-backend-file,id=mem0,size=2G,share=on,mem-path=/tmpfs/template-memory
\
     -numa node,memdev=mem0
2. Stop the template VM, set migration x-ignore-shared capability,
migrate "exec:cat>/tmpfs/state", quit it
3. Start target VM:
   qemu-system-x86 -m 2G \
     -object memory-backend-file,id=mem0,size=2G,share=off,mem-path=/tmpfs/template-memory
\
     -numa node,memdev=mem0 \
     -incoming defer
4. connect to target VM qmp, set migration x-ignore-shared capability,
migrate_incoming "exec:cat /tmpfs/state"

-Tao
-- 
Into something rich and strange.