Open Stack

13.06.2019, 16:46, "Dr. David Alan Gilbert" <dgilbert at redhat.com>:
> * Yury Kotov (yury-kotov at yandex-team.ru) wrote:
>>  Hi,
>>
>>  13.06.2019, 15:44, "Dr. David Alan Gilbert" <dgilbert at redhat.com>:
>>  > * Peng Tao (bergwolf at hyper.sh) wrote:
>>  >>  On Thu, Jun 13, 2019 at 7:09 PM Dr. David Alan Gilbert
>>  >>  <dgilbert at redhat.com> wrote:
>>  >>  >
>>  >>  > * Peng Tao (bergwolf at hyper.sh) wrote:
>>  >>  > > On Thu, Jun 6, 2019 at 1:33 AM Dr. David Alan Gilbert
>>  >>  > > <dgilbert at redhat.com> wrote:
>>  >>  > > >
>>  >>  > > > * Castelino, Manohar R (manohar.r.castelino at intel.com) wrote:
>>  >>  > > > > > It's very similar to the trick that NEMU uses for templating.
>>  >>  > > > > > With the x-ignore-shared-ram migration capability enabled, migration will
>>  >>  > > > > > not write to the migration stream any RAM block that had the
>>  >>  > > > > > shared=on flag on the qemu commandline. So you should then be able
>>  >>  > > > > > to restart from the migration stream and existing RAM image.
>>  >>  > > > >
>>  >>  > > > > So does it mean we can drop our vm-templating patches and move to using " x-ignore-shared-ram" on QEMU 4.0.
>>  >>  > > > >
>>  >>  > > > > Today we only need two patches, which will come down to a single patch which would bring up closer to upstream QEMU 4.0 which would be ideal.
>>  >>  > > > > https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/0001-9p-removing-coroutines-of-9p-to-increase-the-I-O-per.patch
>>  >>  > > > > https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/0002-migration-add-capability-to-bypass-the-shared-memory.patch
>>  >>  > > >
>>  >>  > > > Yes, I'm hoping that with 4.0 you can avoid 0002 - but I've not tried
>>  >>  > > > it; I'd be interested in your results.
>>  >>  > > >
>>  >>  > > Hi Dave,
>>  >>  > >
>>  >>  > > I gave it a try and failed to do vm templating with x-ignore-shared.
>>  >>  > > One key difference between x-ignore-shared and Lai's
>>  >>  > > migrate-bypass-shared patch is that Lai doesn't verify shared memory
>>  >>  > > block upon ram load. OTOH x-ignore-shared rely on ram share property
>>  >>  > > to reconstruct the destination guest memory. In kata, we make use of
>>  >>  > > the fact that destination ram can be private to implement vm
>>  >>  > > templating feature so that multiple new guests can share map the same
>>  >>  > > template VM memory privately (memory-backend-file share=off).
>>  >>  >
>>  >>  > (Copying in Yury who wrote those patches).
>>  >>  >
>>  >>  > > The way we implement vm templating in kata is:
>>  >>  > > 1. Start the template VM:
>>  >>  > > qemu-system-x86 -m 2G \
>>  >>  > > -object memory-backend-file,id=mem0,size=2G,share=on,mem-path=/tmpfs/template-memory
>>  >>  > > \
>>  >>  > > -numa node,memdev=mem0
>>  >>  > > 2. Stop the template VM, set migration bypass-shared-memory
>>  >>  > > capability, migrate exec:cat>/tmpfs/state, quit it
>>  >>  > > 3. Start target VM:
>>  >>  > > qemu-system-x86 -m 2G \
>>  >>  > > -object memory-backend-file,id=mem0,size=2G,share=off,mem-path=/tmpfs/template-memory
>>  >>  > > \
>>  >>  > > -numa node,memdev=mem0 \
>>  >>  > > -incoming "exec:cat /tmpfs/template-state"
>>  >>  > > 4. Start more target VMs like 3
>>  >>  > >
>>  >>  > > There are some major differences between above and the example given
>>  >>  > > by x-ignore-shared patchset[1]:
>>  >>  > > 1. on target vm, memory-backend-file is mapped privately (share=off)
>>  >>  > > 2. no migration capability is set on the target vm
>>  >>  > > 3. it is possible to create multiple target VMs based on the same template VM
>>  >>  > > I have made some hacky change to make x-ignore-shared work for vm
>>  >>  > > templating, mostly by reverting the commit "migration: Add
>>  >>  > > capabilities validation" and reconstructing destination VM ram without
>>  >>  > > using migration share capability or ram share property[2].
>>  >>  > >
>>  >>  > > So I want to ask about your opinion on how to make vm templating work.
>>  >>  > > Shall we change qemu to load ram without the x-ignore-shared
>>  >>  > > capability? Or add another capability to implement similar feature
>>  >>  > > along side x-ignore-shared?
>>  >>  >
>>  >>  > Can we first try a different hack; if you set the ignore shared on the
>>  >>  > destination as well as the source, what happens? I guess it'll hit one
>>  >>  > of the errors in ram_load? Which one?
>>  >>  >
>>  >>  With vanilla 4.0? I got:
>>  >>   qemu-system-x86_64: RAM block mem should be migrated
>>  >>   qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
>>  >>   qemu-system-x86_64: load of migration failed: Invalid argument
>>  >>
>>  >>  which matches:
>>  >>  4341 if (migrate_ignore_shared()) {
>>  >>  4342 hwaddr addr = qemu_get_be64(f);
>>  >>  4343 bool ignored = qemu_get_byte(f);
>>  >>  4344 if (ignored != ramblock_is_ignored(block)) {
>>  >>  4345 error_report("RAM block %s should %s
>>  >>  be migrated",
>>  >>  4346 id, ignored ? "" : "not");
>>  >>  4347 ret = -EINVAL;
>>  >>
>>  >>  If I remove the check it succeeds.
>>  >
>>  > Great.
>>  >
>>  >>  If we go this way, there is no need
>>  >>  to pass shared ram states during migration since this is the only
>>  >>  place they are used.
>>  >
>>  > Well, we could remove them (since the flag is still an x- we're allowed
>>  > to break compatibility).
>>  >
>>  > Yury: What do you think?
>>  >       In this use case they don't run shared on the destination.
>>  >
>>
>>  I'm not sure I understand this use case correctly. If you don't send memory and
>>  don't share it, so how does it migrate? If memory is not shared then target
>>  will use obsolete RAM from disk, right?
>
> The trick here is that they set it shareable initially when they create
> a template, then they do the migrate with the capability set; so now
> you've got a migration image without the RAM, and you've got a template
> RAM file.
> Now you start your VM from the template but you DONT set the shared
> flag - so you get the old RAM as expected and the VM starts quickly, but
> it doesn't write it back to the file; so now you can start another
> VM using hte same template quickly as well.
>

Oh, I get it, thanks. So, as I said, one of the possible solutions is to add an
option for the memory-backend-file to mark the area of RAM that should not be
migrated, instead of checking whether it is shared or not.

> Dave
>
>>  I thought that ignoring all the shared RAM blocks was enough, but in this case
>>  it might be better to mark the migratable memory backends explicitly.
>>
>>  > Dave
>>  >
>>  >>  Cheers,
>>  >>  Tao
>>  >>  --
>>  >>  Into something rich and strange.
>>  > --
>>  > Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK
>>
>>  Regards,
>>  Yury
> --
> Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK

Regards,
Yury