[LITMUS^RT] Extracting TRACE logs from crashed system

[Andrea Bastoni]

Hi Joshua,

On 16/05/2020 21:40, Joshua Bakita wrote:
> Hello all,
> 
> I've been implementing the EDF-SC scheduler (Hobbs, Tong, Anderson RTNS 2019) in
> LITMUS^RT, and occasionally run into kernel panics that prevent pending I/O from
> being flushed to the serial console. The system under test is in a QEMU VM, so
> gdb can easily be attached and the lx-dmesg command can be used to get the
> kernel log messages. However, this obviously doesn't include pending TRACE messages.
> 
> Is anyone aware of a way to get at the pending TRACE message buffer from gdb? It
> seems like you could read the trace buffer off the ftdev_minor struct easily
> enough in gdb, but I can't figure out a way to get a handle to that without
> trying to parse through the VFS state in gdb.

I haven't looked into it in a while, but IIRC I just dumped the memory of the
(possibly per CPU) trace buffers into a file and then read the log or parsed the
binary data structure (depending on whether the trace was plain text or binary
feather trace events).

If you know the kernel VA of the log buffers (possibly per CPU) it shouldn't be
hard to do, because it's constant across tasks, and you should have access to it
when you crash.

IIRC I used something like:

dump binary memory trace.txt log_buffer.buf.kfifo->buffer
log_buffer.buf.kfifo->buffer+log_buffer.buf.kfifo->size

Where the log_buffer symbol was the kernel VA where the ft was storing the data.

Hope it helps,
Andrea

> 
> If anyone can help me figure this out, I'd be happy to contribute the result
> back up to LITMUS^RT as a new helper script for gdb. I can also provide the core
> dump if someone else would like to take a look at it.
> 
> Best,
> 
> Joshua Bakita
> PhD Student - UNC Chapel Hill
> 
> _______________________________________________
> litmus-dev mailing list
> litmus-dev at lists.litmus-rt.org
> https://lists.litmus-rt.org/listinfo/litmus-dev
>