Handling large traces
Kieker was originally developed with web-based interactive software and later applied to event-based embedded software. These usually have rather short traces, as the software should respond quickly in a given timeframe. Unfortunately, desktop software and even more simulations, like climate models, run for a long time uninterrupted producing long traces. For example, we have one Kieker log consisting of one trace which is 79 GB in size as a binary log. These can no longer be processed in memory. There is an option to store this data using graph databases, but this might be also an ineffective use of that utility, as these traces are still hard to process.
A simple solution in the past was, to not instrument certain parts of the main program and in consequence get several partial traces. This is a somewhat hacked solution to circumvent the issue. It also has the downside that parts of the program are excluded from the observation.
In this ticket, I propose a generic solution which makes the traces smaller, better usable without the necessity to exclude arbitrary parts of the software.
Software is usually separated in files, packages, modules or other structure. Based on these boundaries, traces can be cut and partial traces can be created. Kieker already has the concept of parent traces. These could be used here too. A trace splitting stage could then receive a large trace and cut it based on a set of boundaries. In Java this could be packages. However, the boundaries should be configurable.
Essentially, the algorithm should work as follows:
Read a TraceMetadata record as trace start
Detect based on its first OperationExecution or BeforeOperationEvent the boundary.
Create a new TraceMetadata record for this trace and send it
For all incoming BeforeOperationEvents/AfterOperationEvents increase/decrease a stack counter and duplicate them and send them out.
If a BeforeOperationEvent indicates a new boundary, create a new TraceMetadata record, initialize a new stack depth counter for the trace and continue.
If the stack counter is 0 and a new AfterOperationEvent appears (this indicates leaving a Boundary) pop the previous stack counter from the counter stack.