I am chasing a bug that "crictl rm" returns ESRCH which ends up tracking down in the rust agent code:
https://github.com/kata-containers/kata-containers/blob/a12772c6010f6886be7a...
I don't yet understand how to get some debugging messages outside of the agent, so I can see why it return ESRCH. I tried added things like,
info!(sl!(), "KKK in do_signal_process()");
but I can't see that message anywhere. I even setup a debug console which allows me to connect to a shell, but that is. The only way I can get some message out (showed up in the CRI daemon logs) so far is,
return Err(anyhow!("KKK in do_signal_process()"));
which has to terminate the function. Do I have to use things like agent-ctl to capture those "info!" output?
Anyway, it is even better if anyone has some ideas on the bug itself which is that if we do a "crictl rm" without "crictl start" first, it will error out.
# crictl runp --runtime=kata pod-config.json # crictl create <sid> container-pod.json pod-config.json # crictl rm <cid>
ERRO[0010] removing container "144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621" failed: rpc error: code = Unknown desc = unable to stop container 144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621: failed to stop container 144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621: ESRCH: No such process: unknown FATA[0010] unable to remove container(s)
# cat pod-config.json { "metadata": { "name": "alpine-sandbox", "namespace": "default", "attempt": 1, "uid": "hdishd83djaidwnduwk28bcsb" }, "log_directory": "/tmp", "linux": { } }
# cat container-pod.json { "metadata": { "name": "alpine" }, "image":{ "image": "alpine" }, "command": [ "sleep", "3600" ], "log_path":"alpine.0.log", "linux": { } }
On 2 Mar 2021, at 22:02, Qian Cai qcai@redhat.com wrote:
I am chasing a bug that "crictl rm" returns ESRCH which ends up tracking down in the rust agent code:
https://github.com/kata-containers/kata-containers/blob/a12772c6010f6886be7a...
I don't yet understand how to get some debugging messages outside of the agent, so I can see why it return ESRCH. I tried added things like,
info!(sl!(), "KKK in do_signal_process()");
Have you tried replacing that with a warn! ? I remember having a similar issue where info! was filtered by default.
Did you change the logging level, see https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-...
but I can't see that message anywhere. I even setup a debug console which allows me to connect to a shell, but that is. The only way I can get some message out (showed up in the CRI daemon logs) so far is,
return Err(anyhow!("KKK in do_signal_process()"));
which has to terminate the function. Do I have to use things like agent-ctl to capture those "info!" output?
Anyway, it is even better if anyone has some ideas on the bug itself which is that if we do a "crictl rm" without "crictl start" first, it will error out.
# crictl runp --runtime=kata pod-config.json # crictl create <sid> container-pod.json pod-config.json # crictl rm <cid>
ERRO[0010] removing container "144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621" failed: rpc error: code = Unknown desc = unable to stop container 144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621: failed to stop container 144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621: ESRCH: No such process: unknown FATA[0010] unable to remove container(s)
# cat pod-config.json { "metadata": { "name": "alpine-sandbox", "namespace": "default", "attempt": 1, "uid": "hdishd83djaidwnduwk28bcsb" }, "log_directory": "/tmp", "linux": { } }
# cat container-pod.json { "metadata": { "name": "alpine" }, "image":{ "image": "alpine" }, "command": [ "sleep", "3600" ], "log_path":"alpine.0.log", "linux": { } }
kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On Wed, 2021-03-03 at 10:34 +0100, Christophe de Dinechin wrote:
On 2 Mar 2021, at 22:02, Qian Cai qcai@redhat.com wrote:
I am chasing a bug that "crictl rm" returns ESRCH which ends up tracking down in the rust agent code:
https://github.com/kata-containers/kata-containers/blob/a12772c6010f6886be7a...
I don't yet understand how to get some debugging messages outside of the agent, so I can see why it return ESRCH. I tried added things like,
info!(sl!(), "KKK in do_signal_process()");
Have you tried replacing that with a warn! ? I remember having a similar issue where info! was filtered by default.
Did you change the logging level, see https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-... ?
Thanks Christophe, with those in place, I finally received the debug (and even info) output with "crio -l debug" even though previously I used "crio -l trace" which I thought it will override that log_level.
but I can't see that message anywhere. I even setup a debug console which allows me to connect to a shell, but that is. The only way I can get some message out (showed up in the CRI daemon logs) so far is,
return Err(anyhow!("KKK in do_signal_process()"));
which has to terminate the function. Do I have to use things like agent-ctl to capture those "info!" output?
Anyway, it is even better if anyone has some ideas on the bug itself which is that if we do a "crictl rm" without "crictl start" first, it will error out.
# crictl runp --runtime=kata pod-config.json # crictl create <sid> container-pod.json pod-config.json # crictl rm <cid>
ERRO[0010] removing container "144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621" failed: rpc error: code = Unknown desc = unable to stop container 144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621: failed to stop container 144cf0b64d154983da7419a5e8ae5bcb796cc7b054aa7fdcfbb6c44ad8f33621: ESRCH: No such process: unknown FATA[0010] unable to remove container(s)
# cat pod-config.json { "metadata": { "name": "alpine-sandbox", "namespace": "default", "attempt": 1, "uid": "hdishd83djaidwnduwk28bcsb" }, "log_directory": "/tmp", "linux": { } }
# cat container-pod.json { "metadata": { "name": "alpine" }, "image":{ "image": "alpine" }, "command": [ "sleep", "3600" ], "log_path":"alpine.0.log", "linux": { } }
kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
Qian,
On Wed, Mar 3, 2021 at 1:54 PM Qian Cai qcai@redhat.com wrote:
On Wed, 2021-03-03 at 10:34 +0100, Christophe de Dinechin wrote:
On 2 Mar 2021, at 22:02, Qian Cai qcai@redhat.com wrote:
I am chasing a bug that "crictl rm" returns ESRCH which ends up tracking down in the rust agent code:
https://github.com/kata-containers/kata-containers/blob/a12772c6010f6886be7a...
I don't yet understand how to get some debugging messages outside of the agent, so I can see why it return ESRCH. I tried added things like,
info!(sl!(), "KKK in do_signal_process()");
Have you tried replacing that with a warn! ? I remember having a similar issue where info! was filtered by default.
Did you change the logging level, see https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-... ?
Thanks Christophe, with those in place, I finally received the debug (and even info) output with "crio -l debug" even though previously I used "crio -l trace" which I thought it will override that log_level.
https://github.com/kata-containers/kata-containers/issues/1473#issuecomment-... may be of your interest.
Best Regards,
On Wed, 2021-03-03 at 14:14 +0100, Fabiano Fidêncio wrote:
Qian,
On Wed, Mar 3, 2021 at 1:54 PM Qian Cai qcai@redhat.com wrote:
On Wed, 2021-03-03 at 10:34 +0100, Christophe de Dinechin wrote:
On 2 Mar 2021, at 22:02, Qian Cai qcai@redhat.com wrote:
I am chasing a bug that "crictl rm" returns ESRCH which ends up tracking down in the rust agent code:
https://github.com/kata-containers/kata-containers/blob/a12772c6010f6886be7a...
I don't yet understand how to get some debugging messages outside of the agent, so I can see why it return ESRCH. I tried added things like,
info!(sl!(), "KKK in do_signal_process()");
Have you tried replacing that with a warn! ? I remember having a similar issue where info! was filtered by default.
Did you change the logging level, see https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-... ?
Thanks Christophe, with those in place, I finally received the debug (and even info) output with "crio -l debug" even though previously I used "crio -l trace" which I thought it will override that log_level.
https://github.com/kata-containers/kata-containers/issues/1473#issuecomment-... may be of your interest.
Good catch! That definite looks like a silly piece of code.
participants (3)
-
Christophe de Dinechin
-
Fabiano Fidêncio
-
Qian Cai