[LITMUS^RT] Extracting TRACE logs from crashed system

[Joshua Bakita]

Hello Andrea,

On Sat, May 16, 2020 at 4:28 PM Andrea Bastoni <bastoni at sprg.uniroma2.it>
wrote:

> Hi Joshua,
>
> On 16/05/2020 21:40, Joshua Bakita wrote:
> > Hello all,
> >
> > I've been implementing the EDF-SC scheduler (Hobbs, Tong, Anderson RTNS
> 2019) in
> > LITMUS^RT, and occasionally run into kernel panics that prevent pending
> I/O from
> > being flushed to the serial console. The system under test is in a QEMU
> VM, so
> > gdb can easily be attached and the lx-dmesg command can be used to get
> the
> > kernel log messages. However, this obviously doesn't include pending
> TRACE messages.
> >
> > Is anyone aware of a way to get at the pending TRACE message buffer from
> gdb? It
> > seems like you could read the trace buffer off the ftdev_minor struct
> easily
> > enough in gdb, but I can't figure out a way to get a handle to that
> without
> > trying to parse through the VFS state in gdb.
>
> I haven't looked into it in a while, but IIRC I just dumped the memory of
> the
> (possibly per CPU) trace buffers into a file and then read the log or
> parsed the
> binary data structure (depending on whether the trace was plain text or
> binary
> feather trace events).
>
> If you know the kernel VA of the log buffers (possibly per CPU) it
> shouldn't be
> hard to do, because it's constant across tasks, and you should have access
> to it
> when you crash.
>
> IIRC I used something like:
>
> dump binary memory trace.txt log_buffer.buf.kfifo->buffer
> log_buffer.buf.kfifo->buffer+log_buffer.buf.kfifo->size
>

This works almost as written! The actual command works out to:
 `dump binary memory trace.txt debug_buffer.buf
debug_buffer.buf+(debug_buffer.kfifo.in-debug_buffer.kfifo.out)`.
Running that seems to work perfectly for dumping all the pending trace
messages from the
global buffer to trace.txt.

Thank you so much for your help!
- Joshua


>
> Where the log_buffer symbol was the kernel VA where the ft was storing the
> data.
>
> Hope it helps,
> Andrea
>
> >
> > If anyone can help me figure this out, I'd be happy to contribute the
> result
> > back up to LITMUS^RT as a new helper script for gdb. I can also provide
> the core
> > dump if someone else would like to take a look at it.
> >
> > Best,
> >
> > Joshua Bakita
> > PhD Student - UNC Chapel Hill
> >
> > _______________________________________________
> > litmus-dev mailing list
> > litmus-dev at lists.litmus-rt.org
> > https://lists.litmus-rt.org/listinfo/litmus-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.litmus-rt.org/pipermail/litmus-dev/attachments/20200516/28f174bd/attachment.html>