<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Apr 28, 2014, at 10:30 AM, Thijs Raets <<a href="mailto:thijsraets@gmail.com">thijsraets@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><br><div class="gmail_extra">No usable information: meaning I do have information but nothing out of the ordinary. I'm running GSN-EDF. I also looked at /var/log/kernel.log but nothing special. I'm new to kernel debugging so I'm having a hard time finding the right information. But firstly I would like to know if it is normal for Litmus-RT to crash the system when overloaded with RT tasks? </div>
<div class="gmail_extra"><br></div><div class="gmail_extra">If I only run the first 7 tasks on 6 VCPUs there's no crash, also if I run all the tasks on all of my 15 VCPUs there's no crash. <br><br><div class="gmail_quote">
2014-04-28 15:02 GMT+02:00 Björn Brandenburg <span dir="ltr"><<a href="mailto:bbb@mpi-sws.org" target="_blank">bbb@mpi-sws.org</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 28 Apr 2014, at 14:56, Thijs Raets <<a href="mailto:thijsraets@gmail.com" target="_blank">thijsraets@gmail.com</a>> wrote:<br>
><br>
> I'm running a virtual machine with 6 VCPU's with the following task set:<br>
> ./rtspin 12 601 120 -w &<br>
> ./rtspin 478 634 120 -w &<br>
> ./rtspin 334 636 120 -w &<br>
> ./rtspin 128 644 120 -w &<br>
> ./rtspin 460 645 120 -w &<br>
> ./rtspin 10 662 120 -w &<br>
> ./rtspin 103 666 120 -w &<br>
> ./rtspin 276 678 120 -w &<br>
> ./rtspin 137 691 120 -w &<br>
><br>
> When I release this task set, my system crashes. I calculated the MPR interface and 6 VCPUs should be enough. Also /dev/litmus/log does not give me any usable information. I would like to test my virtual machine for real time behavior, so it's ok if deadlines are missed, a crash however is unacceptable. Can anyone help (how can I avoid this behavior or what could be the problem)?<br>
<br>
</div>We're happy to help debugging crashes, but you'll need to provide more information. Which scheduler plugin are you using? What do you mean by "crash"? Are you getting a kernel panic? An OOPS? Is a BUG_ON() triggering? Do you have a backtrace? Is it a lock-up?<br>
<br>
Also, what do you mean by "usable information"? Are you getting any info at all?<br>
<span><font color="#888888"><br>
- Björn<br>
<br>
</font></span></blockquote></div></div></div></blockquote></div><div><br></div><br><div>Hi Thijs,</div><div><br></div><div>Litmus should not be crashing under overload. It is possible that you’re not getting any useful information out of the logs because the kernel is crashing before they get written to disk or even printed to the console.</div><div><br></div><div>Would you be able to test in <font face="Courier New">qemu</font>, or is <font face="Courier New">xen</font> fundamental to your research? With <font face="Courier New">qemu</font>, it is possible to attach <font face="Courier New">gdb</font> over a local socket connection. This is extremely helpful because you can:</div><div><br></div><div>1) Break into the system to diagnose a deadlock</div><div>2) Set break points</div><div>3) Dump log buffers</div><div>4) Examine thread/task/scheduler state</div><div>5) And probably most important to you: Get back traces. A back trace is the first step in figuring out what’s going wrong.</div><div><br></div><div>If you’re lucky, the bug is captured immediately in the backtrace. However, crashes are often caused by an earlier state corruption (most often due to a race condition). In this case, armed with a backtrace, you can go to the log from <font face="Courier New">/dev/litmus/log</font> and try to determine event that directly lead to the crash (it helps to know the particular CPU where the crash occurred—this should be in the backtrace). Then you must meticulously trace backwards through the log to understand the sequence of events that led to the crash. You’ll need kernel code to help you piece together the emitted log messages into sequences of executed operations. You’ll start feeling like Cypher from The Matrix after a while.</div><div><br></div><div>Here’s some info that can get you started with qemu: <a href="https://wiki.litmus-rt.org/litmus/VirtualMachineImages">https://wiki.litmus-rt.org/litmus/VirtualMachineImages</a></div><div><br></div><div>I’m not entirely sure the qemu command-line arguments in the above link are complete enough to debug with <font face="Courier New">gdb</font>. Here are the command-line arguments I use:</div><div><br></div><div><font face="Courier New">qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -hda ./<your disk image> -m 1024 -name “blah” -nographic -kernel litmus-rt/arch/x86/boot/bzImage -append “console=ttyS0,115200 root=/dev/hda1 nopat” -gdb tcp::12345 -net nic -net user,hostfwd::2222-:22</font></div><div><br></div><div>Then to attach with gdb:</div><div><font face="Courier New">gdb litmus-rt/vmlinux</font></div><div><br></div><div>(in the gdb console...)</div><div><font face="Courier New">> target remote localhost:12345</font></div><div><font face="Courier New">> <set break points></font></div><div><font face="Courier New">> continue</font></div><div>(once crashed, on break points, or ctrl-c)</div><div>(to dump the printk buffer:)</div><div><font face="Courier New">> dump binary value printk.txt __log_buf</font></div><div>(to dump the litmus TRACE() buffer:)</div><div><font face="Courier New">> dump binary memory trace.txt debug_buffer.buf debug_buffer+<number of bytes in buffer—it’s 2^(CONFIG_SCHED_DEBUG_TRACE_SHIFT)></font></div><div><br></div><div>You may want to run <font face="Courier New">strings</font> on the *.txt output to filter out garbage characters. Note that both are ring buffers, so the “start” of the log may be somewhere in the middle of the file. Scan through the files and find where the timestamp wraps around to determine the start/end of the trace.</div><div><br></div><div>Another useful debug tool is <font face="Courier New">kdump</font>—especially when used with RedHat’s <font face="Courier New">crash</font> tool (it’s a gdb wrapper with some kernel knowledge). I use <font face="Courier New">kdump</font> when I must debug a natively run kernel (versus one within a virtual machine). There are several tutorials on the internet for how to use <font face="Courier New">kdump</font>. I’m not sure if you can use <font face="Courier New">kdump</font> within <font face="Courier New">xen</font>, but it might be worth a try.</div><div><br></div><div>In any case, don’t forget to compile your kernel with debug symbols and frame pointers (<font face="Courier New">CONFIG_DEBUG_INFO</font> and <font face="Courier New">CONFIG_FRAME_POINTER</font>).</div><div><br></div><div>Finally, I’ve attacked the <font face="Courier New">.config</font> that I use for debugging with <font face="Courier New">qemu/gdb</font>. Hopefully it can help you out, but I cannot guarantee that it will work on your system.</div><div><br></div><div>-Glenn</div><div><br></div><div></div></body></html>