On Thu, Jun 13, 2019 at 8:17 PM Peng Tao <bergwolf@hyper.sh> wrote:
On Thu, Jun 13, 2019 at 7:09 PM Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
* Peng Tao (bergwolf@hyper.sh) wrote:
On Thu, Jun 6, 2019 at 1:33 AM Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
* Castelino, Manohar R (manohar.r.castelino@intel.com) wrote:
It's very similar to the trick that NEMU uses for templating. With the x-ignore-shared-ram migration capability enabled, migration will not write to the migration stream any RAM block that had the shared=on flag on the qemu commandline. So you should then be able to restart from the migration stream and existing RAM image.
So does it mean we can drop our vm-templating patches and move to using " x-ignore-shared-ram" on QEMU 4.0.
Today we only need two patches, which will come down to a single patch which would bring up closer to upstream QEMU 4.0 which would be ideal. https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/... https://github.com/kata-containers/packaging/blob/master/qemu/patches/4.0.x/...
Yes, I'm hoping that with 4.0 you can avoid 0002 - but I've not tried it; I'd be interested in your results.
Hi Dave,
I gave it a try and failed to do vm templating with x-ignore-shared. One key difference between x-ignore-shared and Lai's migrate-bypass-shared patch is that Lai doesn't verify shared memory block upon ram load. OTOH x-ignore-shared rely on ram share property to reconstruct the destination guest memory. In kata, we make use of the fact that destination ram can be private to implement vm templating feature so that multiple new guests can share map the same template VM memory privately (memory-backend-file share=off).
(Copying in Yury who wrote those patches).
The way we implement vm templating in kata is: 1. Start the template VM: qemu-system-x86 -m 2G \ -object memory-backend-file,id=mem0,size=2G,share=on,mem-path=/tmpfs/template-memory \ -numa node,memdev=mem0 2. Stop the template VM, set migration bypass-shared-memory capability, migrate exec:cat>/tmpfs/state, quit it 3. Start target VM: qemu-system-x86 -m 2G \ -object memory-backend-file,id=mem0,size=2G,share=off,mem-path=/tmpfs/template-memory \ -numa node,memdev=mem0 \ -incoming "exec:cat /tmpfs/template-state" 4. Start more target VMs like 3
There are some major differences between above and the example given by x-ignore-shared patchset[1]: 1. on target vm, memory-backend-file is mapped privately (share=off) 2. no migration capability is set on the target vm 3. it is possible to create multiple target VMs based on the same template VM I have made some hacky change to make x-ignore-shared work for vm templating, mostly by reverting the commit "migration: Add capabilities validation" and reconstructing destination VM ram without using migration share capability or ram share property[2].
So I want to ask about your opinion on how to make vm templating work. Shall we change qemu to load ram without the x-ignore-shared capability? Or add another capability to implement similar feature along side x-ignore-shared?
Can we first try a different hack; if you set the ignore shared on the destination as well as the source, what happens? I guess it'll hit one of the errors in ram_load? Which one?
With vanilla 4.0? I got: qemu-system-x86_64: RAM block mem should be migrated qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram' qemu-system-x86_64: load of migration failed: Invalid argument
which matches: 4341 if (migrate_ignore_shared()) { 4342 hwaddr addr = qemu_get_be64(f); 4343 bool ignored = qemu_get_byte(f); 4344 if (ignored != ramblock_is_ignored(block)) { 4345 error_report("RAM block %s should %s be migrated", 4346 id, ignored ? "" : "not"); 4347 ret = -EINVAL;
If I remove the check it succeeds.
By success, I meant the following steps: 1. Start the template VM: qemu-system-x86 -m 2G \ -object memory-backend-file,id=mem0,size=2G,share=on,mem-path=/tmpfs/template-memory \ -numa node,memdev=mem0 2. Stop the template VM, set migration x-ignore-shared capability, migrate "exec:cat>/tmpfs/state", quit it 3. Start target VM: qemu-system-x86 -m 2G \ -object memory-backend-file,id=mem0,size=2G,share=off,mem-path=/tmpfs/template-memory \ -numa node,memdev=mem0 \ -incoming defer 4. connect to target VM qmp, set migration x-ignore-shared capability, migrate_incoming "exec:cat /tmpfs/state" -Tao -- Into something rich and strange.