On Fri, Jun 14, 2019 at 2:46 AM Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
13.06.2019, 16:46, "Dr. David Alan Gilbert" <dgilbert@redhat.com>:
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
Hi,
13.06.2019, 15:44, "Dr. David Alan Gilbert" <dgilbert@redhat.com>:
* Peng Tao (bergwolf@hyper.sh) wrote:
On Thu, Jun 13, 2019 at 7:09 PM Dr. David Alan Gilbert <dgilbert@redhat.com> wrote: > > * Peng Tao (bergwolf@hyper.sh) wrote: > > On Thu, Jun 6, 2019 at 1:33 AM Dr. David Alan Gilbert > > <dgilbert@redhat.com> wrote: > > > > > > * Castelino, Manohar R (manohar.r.castelino@intel.com) wrote: > > > > > It's very similar to the trick that NEMU uses for templating. > > > > > With the x-ignore-shared-ram migration capability enabled, migration will > > > > > not write to the migration stream any RAM block that had the > > > > > shared=on flag on the qemu commandline. So you should then be able > > > > > to restart from the migration stream and existing RAM image. > > > > > > > > So does it mean we can drop our vm-templating patches and move to using " x-ignore-shared-ram" on QEMU 4.0. > > > > > > > > Today we only need two patches, which will come down to a single patch which would bring up closer to upstream QEMU 4.0 which would be ideal. > > > > https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/... > > > > https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/... > > > > > > Yes, I'm hoping that with 4.0 you can avoid 0002 - but I've not tried > > > it; I'd be interested in your results. > > > > > Hi Dave, > > > > I gave it a try and failed to do vm templating with x-ignore-shared. > > One key difference between x-ignore-shared and Lai's > > migrate-bypass-shared patch is that Lai doesn't verify shared memory > > block upon ram load. OTOH x-ignore-shared rely on ram share property > > to reconstruct the destination guest memory. In kata, we make use of > > the fact that destination ram can be private to implement vm > > templating feature so that multiple new guests can share map the same > > template VM memory privately (memory-backend-file share=off). > > (Copying in Yury who wrote those patches). > > > The way we implement vm templating in kata is: > > 1. Start the template VM: > > qemu-system-x86 -m 2G \ > > -object memory-backend-file,id=mem0,size=2G,share=on,mem-path=/tmpfs/template-memory > > \ > > -numa node,memdev=mem0 > > 2. Stop the template VM, set migration bypass-shared-memory > > capability, migrate exec:cat>/tmpfs/state, quit it > > 3. Start target VM: > > qemu-system-x86 -m 2G \ > > -object memory-backend-file,id=mem0,size=2G,share=off,mem-path=/tmpfs/template-memory > > \ > > -numa node,memdev=mem0 \ > > -incoming "exec:cat /tmpfs/template-state" > > 4. Start more target VMs like 3 > > > > There are some major differences between above and the example given > > by x-ignore-shared patchset[1]: > > 1. on target vm, memory-backend-file is mapped privately (share=off) > > 2. no migration capability is set on the target vm > > 3. it is possible to create multiple target VMs based on the same template VM > > I have made some hacky change to make x-ignore-shared work for vm > > templating, mostly by reverting the commit "migration: Add > > capabilities validation" and reconstructing destination VM ram without > > using migration share capability or ram share property[2]. > > > > So I want to ask about your opinion on how to make vm templating work. > > Shall we change qemu to load ram without the x-ignore-shared > > capability? Or add another capability to implement similar feature > > along side x-ignore-shared? > > Can we first try a different hack; if you set the ignore shared on the > destination as well as the source, what happens? I guess it'll hit one > of the errors in ram_load? Which one? > With vanilla 4.0? I got: qemu-system-x86_64: RAM block mem should be migrated qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram' qemu-system-x86_64: load of migration failed: Invalid argument
which matches: 4341 if (migrate_ignore_shared()) { 4342 hwaddr addr = qemu_get_be64(f); 4343 bool ignored = qemu_get_byte(f); 4344 if (ignored != ramblock_is_ignored(block)) { 4345 error_report("RAM block %s should %s be migrated", 4346 id, ignored ? "" : "not"); 4347 ret = -EINVAL;
If I remove the check it succeeds.
Great.
If we go this way, there is no need to pass shared ram states during migration since this is the only place they are used.
Well, we could remove them (since the flag is still an x- we're allowed to break compatibility).
Yury: What do you think? In this use case they don't run shared on the destination.
I'm not sure I understand this use case correctly. If you don't send memory and don't share it, so how does it migrate? If memory is not shared then target will use obsolete RAM from disk, right?
The trick here is that they set it shareable initially when they create a template, then they do the migrate with the capability set; so now you've got a migration image without the RAM, and you've got a template RAM file. Now you start your VM from the template but you DONT set the shared flag - so you get the old RAM as expected and the VM starts quickly, but it doesn't write it back to the file; so now you can start another VM using hte same template quickly as well.
Oh, I get it, thanks. So, as I said, one of the possible solutions is to add an option for the memory-backend-file to mark the area of RAM that should not be migrated, instead of checking whether it is shared or not.
I think I'd rather just remove the check - the check is there to stop someone doing something silly; this case shows it's valid.
Thank you both for the suggestions. Let me cook up a patch and make sure it works for kata. Cheers, Tao -- Into something rich and strange.