error occurs when kernel is 4.19.110-300.el7.x86_64/centos7.7.1908
Hi, all: if pod enables "securityContext: privileged: true", containerd faild to start kata. host kernel:centos7 , 4.19.110-300.el7.x86_64 (kernel 3.10.0-1062.4.3.el7.x86_64 and 5.10.12-1.el7.elrepo.x86_64 are both ok), it seems kernel and qemu-vanilla-bin have confilcts. ENV: kata-runtime-1.11.5-10.2.x86_64/kata-linux-container-5.4.32.76-11.2.x86_64/kata-containers-image-1.11.5-10.2.x86_64/containerd.io-1.4.11/k8s 1.20 time="2021-10-20T19:33:28.842023733+08:00" level=error msg="StartContainer for \"7c3ba5c846c91dd45d48c9ed4796d19a71998f82b783abea7e148086e38a6e19\" failed" error="failed to create containerd task: QMP command failed: Failed to connect socket /dev/vport2p1: Connection refused: unknown" Oct 20 19:33:28 node6 kubelet: E1020 19:33:28.842259 1791 remote_runtime.go:251] StartContainer "7c3ba5c846c91dd45d48c9ed4796d19a71998f82b783abea7e148086e38a6e19" from runtime service failed: rpc error: code = Unknown desc = failed to create containerd task: QMP command failed: Failed to connect socket /dev/vport2p1: Connection refused: unknown Oct 20 19:33:28 node6 kubelet: E1020 19:33:28.842414 1791 kuberuntime_manager.go:829] container &Container{Name:centos7-kata,Image:myregistry.com/library/centos:latest,Command:[/bin/sh],Args:[-c while true; do sleep 99999999999; done],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:LEVEL,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:myconfig,},Key:special.level,Optional:nil,},SecretKeyRef:nil,},},EnvVar{Name:TYPE,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:myconfig,},Key:special.type,Optional:nil,},SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:myconfig,ReadOnly:false,MountPath:/etc/config,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:config,ReadOnly:true,MountPath:/config,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:test-volume,ReadOnly:false,MountPath:/host1,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:default-token-7gg9q,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod centos7-kata-5bcffffd46-j2knt_default(bea2c995-75b4-4614-90d1-6185187ec2a8): RunContainerError: failed to create containerd task: QMP command failed: Failed to connect socket /dev/vport2p1: Connection refused: unknown Oct 20 19:33:28 node6 kubelet: E1020 19:33:28.842451 1791 pod_workers.go:191] Error syncing pod bea2c995-75b4-4614-90d1-6185187ec2a8 ("centos7-kata-5bcffffd46-j2knt_default(bea2c995-75b4-4614-90d1-6185187ec2a8)"), skipping: failed to "StartContainer" for "centos7-kata" with RunContainerError: "failed to create containerd task: QMP command failed: Failed to connect socket /dev/vport2p1: Connection refused: unknown" Oct 20 19:33:29 node6 kubelet: I1020 19:33:29.544984 1791 scope.go:95] [topologymanager] RemoveContainer - Container ID: 98912179f80f5d996069977f455739b757870f3e3d8ecf79bae2b5d5037850dc Oct 20 19:33:29 node6 kubelet: I1020 19:33:29.545256 1791 scope.go:95] [topologymanager] RemoveContainer - Container ID: 7c3ba5c846c91dd45d48c9ed4796d19a71998f82b783abea7e148086e38a6e19 Oct 20 19:33:29 node6 kubelet: E1020 19:33:29.545652 1791 pod_workers.go:191] Error syncing pod bea2c995-75b4-4614-90d1-6185187ec2a8 ("centos7-kata-5bcffffd46-j2knt_default(bea2c995-75b4-4614-90d1-6185187ec2a8)"), skipping: failed to "StartContainer" for "centos7-kata" with CrashLoopBackOff: "back-off 10s restarting failed container=centos7-kata pod=centos7-kata-5bcffffd46-j2knt_default(bea2c995-75b4-4614-90d1-6185187ec2a8)" Oct 20 19:33:29 node6 containerd: time="2021-10-20T19:33:29.546001819+08:00" level=info msg="RemoveContainer for \"98912179f80f5d996069977f455739b757870f3e3d8ecf79bae2b5d5037850dc\""
On Wed, Oct 20, 2021 at 07:46:41PM +0800, ylg wrote:
Hi, all:
if pod enables "securityContext: privileged: true", containerd faild to start kata.
"privileged" is ill-defined in the context of kata containers. One of the main uses of privileged containers is to manage things on the host, which is fundamentally incompatible with Kata's approach of running the container in a VM. There are some edge cases where privileged kata containers are used, but in general it's really not clear what permissions and abilities such a container should have. As a consequence, I'd recommend not attempting to use Kata for privileged containers. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On 2021/10/20 19:46, ylg wrote:
Hi, all:
if pod enables "securityContext: privileged: true", containerd faild to start kata.
host kernel:centos7 , 4.19.110-300.el7.x86_64 (kernel 3.10.0-1062.4.3.el7.x86_64 and 5.10.12-1.el7.elrepo.x86_64 are both ok), it seems kernel and qemu-vanilla-bin have confilcts. ENV: kata-runtime-1.11.5-10.2.x86_64/kata-linux-container-5.4.32.76-11.2.x86_64/kata-containers-image-1.11.5-10.2.x86_64/containerd.io-1.4.11/k8s 1.20
Does your host kernel have vsock support? Also can you run qemu without kata in your setup? Cheers, Tao
On 2021/10/23 13:12, David Gibson wrote:
On Wed, Oct 20, 2021 at 07:46:41PM +0800, ylg wrote:
Hi, all:
if pod enables "securityContext: privileged: true", containerd faild to start kata.
"privileged" is ill-defined in the context of kata containers. One of the main uses of privileged containers is to manage things on the host, which is fundamentally incompatible with Kata's approach of running the container in a VM.
There are some edge cases where privileged kata containers are used, but in general it's really not clear what permissions and abilities such a container should have.
As a consequence, I'd recommend not attempting to use Kata for privileged containers.
It is just one use case of privileged containers. We still have other cases like running FUSE daemons or nested containers (like dind or kind), or anything that requires SYS_ADMIN capability in general. I don't think we should just ask users to move away from privileged containers with Kata. If you think that `privileged` is not clearly defined with Kata, we can make it clear together. Dropping it completely is not a wise choice imho. Cheers, Tao -- Into something rich and strange.
participants (3)
-
David Gibson
-
Peng Tao
-
ylg