Fabiano suggested that I share a bit of information ahead of time about the recorder library I was talking about during our last discussion on tracing.
First, let me be very clear that I am not suggesting that we use the library directly for Kata. It is written in C and designed for C and C++ programs. I am merely presenting it to share a few ideas that may be relevant to the current discussions about tracing and debuggability.
In addition to the long-running / short-running and dynamic/static tracing distinctions that we have already talked about, it may also be interesting to talk about usage models for the instrumentation. We should probably consider at least four major scenarios:
1. Operational monitoring, i.e. checking how the system behaves. This is basically where we use metrics.
2. Operational deep-dive, i.e. being able to dive deep into the behaviour of a system. My understanding is that the current tracing is mostly focusing on that use case
3. Post-mortem, i.e. figuring out why something went wrong. This is where we use logs and also where we have concerns about tracing, e..g in case of crash.
4. Interactive debugging, i.e. a developer trying to understand a bug. This is where we enable all debugging, don't care much about performance, and often can repeat an experiment.
Intuitively, there is a lot of overlap between the instrumentation for all these cases. However, there are conflicting requirements notably with respect to performance, amount of data collected and verbosity that lead to the frequent use of totally different subsystems. For example, in many programs, there will be logging that uses 2 and 3, with "levels" such as "info", "warning", "debug"… and when you actually do interactive debugging, you add your own messages that you need to remove before committing anything. This is very similar to the discussions we are currently having in the context of tracing, about dynamic vs. static tracing, and so on.
The recorder library attempts to address all four use-cases with a single instrumentation, that you can configure to:
1. Be able to synthesise metric-style data from low-cost instrumentation probes.
2. Collect topical data that makes it possible to zoom on this or that feature.
3. Record data in a loop, flight-recorder style, to be ready to dump it in case of anomalous event, for post-mortem analysis.
4. Turn individual topics into dynamic traces, so that you have finer control on your debug output than just a log level.
Here is some information to better understand the design:
- GitHub project: https://github.com/c3d/recorder
- A general presentation about it: https://www.youtube.com/watch?v=kEnQY1zFa0Y
- An example of application: https://www.youtube.com/watch?v=kpy4SqpunoY (starting around 12 minutes)
- The evolution (and a few historical applications): https://www.youtube.com/watch?v=qzvW-Hy2rTs.
Once the recent work on tracing has been merged, we may start exploring these ideas like these to solve the following issues that were raised notably by James:
- Synchronization between multiple data sources
- Being able to get some data even without a nice shutdown from the agent
- Having a single data channel (one VSOCK) for console, tracing and metrics
- Being able to get some data while the container is running
Hope this helps,
Christophe
Hi All,
TL;DR
Please review the tracing proposals document PR [1].
# Detail
Hot on the heels of the last tracing update [2], I've just raised a doc PR that summarises liubin's proposals for tracing with Kata [1].
In summary, the proposals simplify the planned architecture still further, but remove "dynamic tracing" from our plans (although this could be considered at a later date if the community deems it worthwhile).
# Feedback required
If you have an interest in tracing, _please_ review this document and comment. We're particularly keen to understand the answers to the questions listed on the PR above (do you care more about short-lived or long-running containers, etc). The answers will help us decide whether to proceed with the proposals.
# Architecture Committee discussion
I've added an agenda item for next weeks AC meeting [3] to discuss these proposals so you have a few days to review the doc before we discuss next Tuesday.
Cheers,
James
[1] - https://github.com/kata-containers/kata-containers/pull/2062
[2] - https://gist.github.com/jodh-intel/0ee54d41d2a803ba761e166136b42277
[3] - https://etherpad.opendev.org/p/Kata_Containers_2021_Architecture_Committee_…
---
https://katacontainers.io/ | https://github.com/kata-containers
Open Source Technology Center
Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ.