Open Stack

Fabiano suggested that I share a bit of information ahead of time about the recorder library I was talking about during our last discussion on tracing.

First, let me be very clear that I am not suggesting that we use the library directly for Kata. It is written in C and designed for C and C++ programs. I am merely presenting it to share a few ideas that may be relevant to the current discussions about tracing and debuggability.

In addition to the long-running / short-running and dynamic/static tracing distinctions that we have already talked about, it may also be interesting to talk about usage models for the instrumentation. We should probably consider at least four major scenarios:

1. Operational monitoring, i.e. checking how the system behaves. This is basically where we use metrics.
2. Operational deep-dive, i.e. being able to dive deep into the behaviour of a system. My understanding is that the current tracing is mostly focusing on that use case
3. Post-mortem, i.e. figuring out why something went wrong. This is where we use logs and also where we have concerns about tracing, e..g in case of crash.
4. Interactive debugging, i.e. a developer trying to understand a bug. This is where we enable all debugging, don't care much about performance, and often can repeat an experiment.

Intuitively, there is a lot of overlap between the instrumentation for all these cases. However, there are conflicting requirements notably with respect to performance, amount of data collected and verbosity that lead to the frequent use of totally different subsystems. For example, in many programs, there will be logging that uses 2 and 3, with "levels" such as "info", "warning", "debug"… and when you actually do interactive debugging, you add your own messages that you need to remove before committing anything. This is very similar to the discussions we are currently having in the context of tracing, about dynamic vs. static tracing, and so on.

The recorder library attempts to address all four use-cases with a single instrumentation, that you can configure to:

1. Be able to synthesise metric-style data from low-cost instrumentation probes. 
2. Collect topical data that makes it possible to zoom on this or that feature.
3. Record data in a loop, flight-recorder style, to be ready to dump it in case of anomalous event, for post-mortem analysis.
4. Turn individual topics into dynamic traces, so that you have finer control on your debug output than just a log level.

Here is some information to better understand the design:

- GitHub project: https://github.com/c3d/recorder
- A general presentation about it: https://www.youtube.com/watch?v=kEnQY1zFa0Y
- An example of application: https://www.youtube.com/watch?v=kpy4SqpunoY (starting around 12 minutes)
- The evolution (and a few historical applications): https://www.youtube.com/watch?v=qzvW-Hy2rTs.

Once the recent work on tracing has been merged, we may start exploring these ideas like these to solve the following issues that were raised notably by James:

- Synchronization between multiple data sources
- Being able to get some data even without a nice shutdown from the agent
- Having a single data channel (one VSOCK) for console, tracing and metrics
- Being able to get some data while the container is running


Hope this helps,
Christophe