[LITMUS^RT] Right size for scheduling trace buffer

[Glenn Elliott]

On Apr 18, 2013, at 11:33 AM, Youcef Amine Benabbas <s9yobena at stud.uni-saarland.de> wrote:

> Hello all,
> 
> I know that the size of the scheduling trace buffer is workload
> dependent, however, I am having problems getting the right size for my
> workload so as to avoid missing scheduling trace information. 
> 
> In particular, I chose the largest possible shift value (13), still, I
> am missing some traces. 
> 
> I am running experiments on a 24 core machine, with tasks whose
> utilization and period are randomly drawn from [0.1,0.3) and
> [10ms,100ms), respectively.
> 
> Did any one try a workload/trace-buffer-configuration  that is known to
> incur no scheduling trace misses(I know it is easy to get this correct
> for less than 4 cpus, but it would be interesting to get this correct
> for >16 cpus)?
> 
> Regards,
> Youcef
> 
> 
> _______________________________________________
> litmus-dev mailing list
> litmus-dev at lists.litmus-rt.org
> https://lists.litmus-rt.org/listinfo/litmus-dev



Hi Youcef,

There are a number of steps that you can take to mitigate this problem (in increasing difficulty):
1) Make sure that you only trace the events that you need to trace.  For example, you don't need to record SWITCH_TO and SWITCH_AWAY events if you only want to discover deadline miss rates or compute response times (I think you only need PARAM, RELEASE, COMPLETION, and SYS_RELEASE to compute these measures).  Reducing the types of events you record will reduce the speed at which the ring buffers are filled.  See "ST_IDS" in feather-trace-tools/st_trace.
2) Write trace logs to shared memory instead of disk.  This incurs lower overheads, so more records have a chance of getting flushed out of the ring buffers.  On Ubuntu, you can write the *.bin record streams to the directory located at /dev/shm.  Copy the trace data out of /dev/shm after each experiment though.  Filling /dev/shm will trigger page swapping.
3) Run the trace tasks as SCHED_FIFO task (check out the "sched_tool" utility).  This will ensure that any idle time from Litmus is spend flushing the ring buffers. (Note: Running tracers as SCHED_FIFO used to crash Litmus, but I believe it has been fixed).
4) Increase the maximum allowable image size of the kernel.  If your ring buffers get too big, you can get the link error "kernel image bigger than KERNEL_IMAGE_SIZE".  However, you can work around this by increasing KERNEL_IMAGE_SIZE.  On x86, this is located in arch/x86/include.  **HOWEVER**, you cannot merely increase this compile-time constant.  You also have to update KERNEL_IMAGE_START to reflect increased image layout size.  There may be some other gotchas, so be careful.
5) Run the trace tasks as Litmus real-time tasks with rt_launch.  I've never tried this one myself, but it should work (theoretically).  Of course, this would result in capacity loss for the real-time tasks you want to trace.  (You would also have to filter out trace events from the tracers themselves during analysis.)

-Glenn