Hi all, One of the features currently available in 2.0.0 is dax support, in an internal discussion virtiofs Dax stability was mentioned as not mature enough yet. For example, some of the weakspots that were mentioned are: * Guest can crash when accessing a DAX page of a truncated file * The stability of the QEMU interface for the mapping is still not there IIUC it's currently enabled by default (qemu), can it be disabled? if not, is it planned to be configurable? Are there currently any upstream issues that can be linked to Dax's instability? Any other insights / thoughts would be highly appreciated. Thank you.
Snir, Firstly, let me reply to one specific bit of your email and also add a question. [...]
IIUC it's currently enabled by default (qemu), can it be disabled? if not, is it planned to be configurable?
The current govmm codes only sets the "cache-size" when it's different from zero: https://github.com/intel/govmm/blob/master/qemu/qemu.go#L1256-L1258 So, setting virtio_fs_cache_size here https://github.com/kata-containers/kata-containers/blob/2.0-dev/src/runtime/... disables DAX. A question I'd like to add to Snir's question is ... Wouldn't it be safer to have `virtio_fs_cache_size = 0` as a default value when using virtio-fs, at least till DAX gets solid? [...] Best Regards, -- Fabiano Fidêncio
* Fabiano Fidêncio (fidencio@redhat.com) wrote:
Snir,
Firstly, let me reply to one specific bit of your email and also add a question.
[...]
IIUC it's currently enabled by default (qemu), can it be disabled? if not, is it planned to be configurable?
The current govmm codes only sets the "cache-size" when it's different from zero: https://github.com/intel/govmm/blob/master/qemu/qemu.go#L1256-L1258
So, setting virtio_fs_cache_size here https://github.com/kata-containers/kata-containers/blob/2.0-dev/src/runtime/... disables DAX.
A question I'd like to add to Snir's question is ... Wouldn't it be safer to have `virtio_fs_cache_size = 0` as a default value when using virtio-fs, at least till DAX gets solid?
I think that should work; the current qemu vhost-user-fs.c code has: DEFINE_PROP_SIZE("cache-size", VHostUserFS, conf.cache_size, 0), so ommitting the property should give us 0 and cache disabled and ommitting the property also means you work with upstream qemu's. I do remember in older dev branches we had the default none-0 but finally found the error of our ways. Dave
[...]
Best Regards, -- Fabiano Fidêncio
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On 2 Oct 2020, at 11:24, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
* Fabiano Fidêncio (fidencio@redhat.com <mailto:fidencio@redhat.com>) wrote:
Snir,
Firstly, let me reply to one specific bit of your email and also add a question.
[...]
IIUC it's currently enabled by default (qemu), can it be disabled? if not, is it planned to be configurable?
The current govmm codes only sets the "cache-size" when it's different from zero: https://github.com/intel/govmm/blob/master/qemu/qemu.go#L1256-L1258
So, setting virtio_fs_cache_size here https://github.com/kata-containers/kata-containers/blob/2.0-dev/src/runtime/... disables DAX.
A question I'd like to add to Snir's question is ... Wouldn't it be safer to have `virtio_fs_cache_size = 0` as a default value when using virtio-fs, at least till DAX gets solid?
I think that should work; the current qemu vhost-user-fs.c code has:
DEFINE_PROP_SIZE("cache-size", VHostUserFS, conf.cache_size, 0),
so ommitting the property should give us 0 and cache disabled and ommitting the property also means you work with upstream qemu's.
I do remember in older dev branches we had the default none-0 but finally found the error of our ways.
Do you have any idea of the timeline for the DAX issues to be resolved? 2.0 is a long-lived branch, presumably, so we want to have something in place that works when DAX is here. You may want to look at https://bugzilla.redhat.com/show_bug.cgi?id=1876915#c30
Dave
[...]
Best Regards, -- Fabiano Fidêncio
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev> -- Dr. David Alan Gilbert / dgilbert@redhat.com <mailto:dgilbert@redhat.com> / Manchester, UK
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev>
Christophe, looks like the bug you mentioned is not public yet, alteast I am not able to access it. Agree with disabling DAX, although as Snir mentioned would like to see any upstream bugs opened to track this. In addition, we are using a 5.6 kernel with Dax patches by default for virtiofs for both qemu-virtiofs and cloud-hypervisor. I propose we instead switch to the LTS 5.4 upstream kernel that has all the bits for virtiofs except Dax. If some user wants to use dax, they can switch to the experimental 5.6 kernel instead. -Archana From: Christophe de Dinechin <cdupontd@redhat.com> Date: Friday, October 2, 2020 at 2:43 AM To: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Cc: kata-dev <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] Dax enablement in 2.0 On 2 Oct 2020, at 11:24, Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>> wrote: * Fabiano Fidêncio (fidencio@redhat.com<mailto:fidencio@redhat.com>) wrote: Snir, Firstly, let me reply to one specific bit of your email and also add a question. [...] IIUC it's currently enabled by default (qemu), can it be disabled? if not, is it planned to be configurable? The current govmm codes only sets the "cache-size" when it's different from zero: https://github.com/intel/govmm/blob/master/qemu/qemu.go#L1256-L1258 So, setting virtio_fs_cache_size here https://github.com/kata-containers/kata-containers/blob/2.0-dev/src/runtime/... disables DAX. A question I'd like to add to Snir's question is ... Wouldn't it be safer to have `virtio_fs_cache_size = 0` as a default value when using virtio-fs, at least till DAX gets solid? I think that should work; the current qemu vhost-user-fs.c code has: DEFINE_PROP_SIZE("cache-size", VHostUserFS, conf.cache_size, 0), so ommitting the property should give us 0 and cache disabled and ommitting the property also means you work with upstream qemu's. I do remember in older dev branches we had the default none-0 but finally found the error of our ways. Do you have any idea of the timeline for the DAX issues to be resolved? 2.0 is a long-lived branch, presumably, so we want to have something in place that works when DAX is here. You may want to look at https://bugzilla.redhat.com/show_bug.cgi?id=1876915#c30 Dave [...] Best Regards, -- Fabiano Fidêncio _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev -- Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On 5 Oct 2020, at 20:21, Shinde, Archana M <archana.m.shinde@intel.com> wrote:
Christophe, looks like the bug you mentioned is not public yet, alteast I am not able to access it.
Sorry about that.
Agree with disabling DAX, although as Snir mentioned would like to see any upstream bugs opened to track this.
In addition, we are using a 5.6 kernel with Dax patches by default for virtiofs for both qemu-virtiofs and cloud-hypervisor. I propose we instead switch to the LTS 5.4 upstream kernel that has all the bits for virtiofs except Dax. If some user wants to use dax, they can switch to the experimental 5.6 kernel instead.
Do you know what is the failure mode if the kernel does not have DAX support and you try to configure it?
-Archana
From: Christophe de Dinechin <cdupontd@redhat.com> Date: Friday, October 2, 2020 at 2:43 AM To: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Cc: kata-dev <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] Dax enablement in 2.0
On 2 Oct 2020, at 11:24, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
* Fabiano Fidêncio (fidencio@redhat.com) wrote:
Snir,
Firstly, let me reply to one specific bit of your email and also add a question.
[...]
IIUC it's currently enabled by default (qemu), can it be disabled? if not, is it planned to be configurable?
The current govmm codes only sets the "cache-size" when it's different from zero: https://github.com/intel/govmm/blob/master/qemu/qemu.go#L1256-L1258
So, setting virtio_fs_cache_size here https://github.com/kata-containers/kata-containers/blob/2.0-dev/src/runtime/... disables DAX.
A question I'd like to add to Snir's question is ... Wouldn't it be safer to have `virtio_fs_cache_size = 0` as a default value when using virtio-fs, at least till DAX gets solid?
I think that should work; the current qemu vhost-user-fs.c code has:
DEFINE_PROP_SIZE("cache-size", VHostUserFS, conf.cache_size, 0),
so ommitting the property should give us 0 and cache disabled and ommitting the property also means you work with upstream qemu's.
I do remember in older dev branches we had the default none-0 but finally found the error of our ways.
Do you have any idea of the timeline for the DAX issues to be resolved? 2.0 is a long-lived branch, presumably, so we want to have something in place that works when DAX is here.
You may want to look at https://bugzilla.redhat.com/show_bug.cgi?id=1876915#c30
Dave
[...]
Best Regards, -- Fabiano Fidêncio
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
Archana, [...]
In addition, we are using a 5.6 kernel with Dax patches by default for virtiofs for both qemu-virtiofs and cloud-hypervisor. I propose we instead switch to the LTS 5.4 upstream kernel that has all the bits for virtiofs except Dax. If some user wants to use dax, they can switch to the experimental 5.6 kernel instead.
I agree with the suggestion. However, I'd like to know whether it'd be possible to have a jenkins job, running weekly or so, with a 5.6 kernel with all the new bits enabled there, as with this we could: 1. Ensure we don't break anything on the Kata side related to DAX; 2. Ensure we do at least some test to early report issues to virtio-fs folks; Is this a reasonable and feasible approach? [...] Best Regards, -- Fabiano Fidêncio
Yes, I can create both jobs to keep testing what we consider experimental. Today experimental mean virtiofs kernel. I can rename it to be virtiofs experimental. And move that version freely even for more recent releases as will not break anything we depend on. Sounds good ? Experimental -> virtifos-experimental - Carlos On 06/10/20 3:46, "Fabiano Fidêncio" <fidencio@redhat.com> wrote: Archana, [...] > In addition, we are using a 5.6 kernel with Dax patches by default for virtiofs for both qemu-virtiofs and cloud-hypervisor. I propose we instead switch to the LTS 5.4 upstream kernel that has all the bits for virtiofs except Dax. If some user wants to use dax, they can switch to the experimental 5.6 kernel instead. I agree with the suggestion. However, I'd like to know whether it'd be possible to have a jenkins job, running weekly or so, with a 5.6 kernel with all the new bits enabled there, as with this we could: 1. Ensure we don't break anything on the Kata side related to DAX; 2. Ensure we do at least some test to early report issues to virtio-fs folks; Is this a reasonable and feasible approach? [...] Best Regards, -- Fabiano Fidêncio _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
Sorry we hadn't test DAX much before because we have not enabled it for container storages in our environment. I agree with the suggestion disable it by default. However, I don't think we will officially support non-LTS kernels like 5.6, and I wonder whether the patches could be back ported to 5.4 LTS if they could fix the crash. Xu ------------------------------------------------------------------ From:Fabiano Fidêncio <fidencio@redhat.com> Send Time:2020年10月6日(星期二) 16:46 To:"Shinde, Archana M" <archana.m.shinde@intel.com> Cc:kata-dev <kata-dev@lists.katacontainers.io> Subject:Re: [kata-dev] Dax enablement in 2.0 Archana, [...]
In addition, we are using a 5.6 kernel with Dax patches by default for virtiofs for both qemu-virtiofs and cloud-hypervisor. I propose we instead switch to the LTS 5.4 upstream kernel that has all the bits for virtiofs except Dax. If some user wants to use dax, they can switch to the experimental 5.6 kernel instead.
I agree with the suggestion. However, I'd like to know whether it'd be possible to have a jenkins job, running weekly or so, with a 5.6 kernel with all the new bits enabled there, as with this we could: 1. Ensure we don't break anything on the Kata side related to DAX; 2. Ensure we do at least some test to early report issues to virtio-fs folks; Is this a reasonable and feasible approach? [...] Best Regards, -- Fabiano Fidêncio _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
* Christophe de Dinechin (cdupontd@redhat.com) wrote:
On 2 Oct 2020, at 11:24, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
* Fabiano Fidêncio (fidencio@redhat.com <mailto:fidencio@redhat.com>) wrote:
Snir,
Firstly, let me reply to one specific bit of your email and also add a question.
[...]
IIUC it's currently enabled by default (qemu), can it be disabled? if not, is it planned to be configurable?
The current govmm codes only sets the "cache-size" when it's different from zero: https://github.com/intel/govmm/blob/master/qemu/qemu.go#L1256-L1258
So, setting virtio_fs_cache_size here https://github.com/kata-containers/kata-containers/blob/2.0-dev/src/runtime/... disables DAX.
A question I'd like to add to Snir's question is ... Wouldn't it be safer to have `virtio_fs_cache_size = 0` as a default value when using virtio-fs, at least till DAX gets solid?
I think that should work; the current qemu vhost-user-fs.c code has:
DEFINE_PROP_SIZE("cache-size", VHostUserFS, conf.cache_size, 0),
so ommitting the property should give us 0 and cache disabled and ommitting the property also means you work with upstream qemu's.
I do remember in older dev branches we had the default none-0 but finally found the error of our ways.
Do you have any idea of the timeline for the DAX issues to be resolved? 2.0 is a long-lived branch, presumably, so we want to have something in place that works when DAX is here.
I think full Dax might take a while; the kernel code is moving upstream, but there are still a couple of nasty corner cases where we need to figure out good solutions. Dave
You may want to look at https://bugzilla.redhat.com/show_bug.cgi?id=1876915#c30
Dave
[...]
Best Regards, -- Fabiano Fidêncio
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev> -- Dr. David Alan Gilbert / dgilbert@redhat.com <mailto:dgilbert@redhat.com> / Manchester, UK
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev>
-- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
participants (7)
-
Christophe de Dinechin
-
Dr. David Alan Gilbert
-
Fabiano Fidêncio
-
Shinde, Archana M
-
Snir
-
Venegas Munoz, Jose Carlos
-
Xu Wang