<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">This is an issue has hit me before. I think running ftcat as SCHED_FIFO is useful. Is there any way we can fix this? Perhaps we should silently ignore the resched event instead of BUG()'ing?<div><br></div><div>-Glenn<br><div><br></div><div><br><div><div>On Nov 30, 2012, at 5:12 PM, Jonathan Herman <<a href="mailto:hermanjl@cs.unc.edu">hermanjl@cs.unc.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">This is caused by my scripts elevating ftcat for sched_trace to SCHED_FIFO. Apparently the SCHED_FIFO rebalancing timer can trigger a reschedule for two reasons:<div>1. Too much SCHED_FIFO work was done in the last period</div>
<div>2. Too little SCHED_FIFO work was done in the last period (this is what got me)</div><div>My SCHED_FIFO ftcat's were getting starved, causing the rebalancing timer to remotely reschedule a running rtspin in the vain hope that ftcat would be selected to run.</div>
<div><br></div><div>I wouldn't worry about this issue. False alarm.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 30, 2012 at 2:21 PM, Jonathan Herman <span dir="ltr"><<a href="mailto:hermanjl@cs.unc.edu" target="_blank">hermanjl@cs.unc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It does still crash, yes. It looks like this is being caused by the real-time balancing timer in linux (see Documentation/scheduler/sched-rt-group.txt) which forces Linux real-time (ie SCHED_FIFO) tasks to relinquish 5% of the CPU to SCHED_OTHER tasks. It works like so:<div>
1. A timer fires periodically.</div><div>2. The timer scans ALL cpus.</div><div>3. If any CPU has real-time work exceeding some threshold over the last timer period, the currently running task on that CPU is rescheduled.</div>
<div><br></div><div>Step 3 is what caused our BUG. The BUG is hit if a remote processor, in this case the processor on which the rebalancing timer fired, triggers a reschedule of a SCHED_LITMUS task.</div><div><br></div>
<div>
<div>This can be disabled with:<br><div>echo -1 > /proc/sys/kernel/sched_rt_runtime_us</div></div></div><div><br></div><div>I will spend some time figuring out why this timer is considering SCHED_LITMUS work when rebalancing. Somehow our work is counting towards the 95% work which SCHED_FIFO tasks can use. That is why the BUG is only hit when the system is fully utilized as well.</div>
<div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">On Thu, Nov 29, 2012 at 5:39 PM, Björn Brandenburg <span dir="ltr"><<a href="mailto:bbb@mpi-sws.org" target="_blank">bbb@mpi-sws.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
On Nov 29, 2012, at 9:20 PM, Jonathan Herman <<a href="mailto:hermanjl@cs.unc.edu" target="_blank">hermanjl@cs.unc.edu</a>> wrote:<br>
<br>
> I have also confirmed that this only happens when shed-trace is running (not overhead tracing, just scheduling). Every commit I checked back to the 3.0 merge where my sched-trace still ran had this same issue.<br>
<br>
</div>Yikes. Sounds like it could be a bug in the Feather-Trace triggers.<br>
<br>
Could you please try the following: edit arch/x86/Kconfig to remove ARCH_HAS_FEATHER_TRACE. This should disable the asm Feather-Trace hacks and replace it with a tame default implementation. Does it still crash?<br>
<div><br>
Thanks,<br>
Björn<br>
<br>
<br>
_______________________________________________<br>
litmus-dev mailing list<br>
<a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
<a href="https://lists.litmus-rt.org/listinfo/litmus-dev" target="_blank">https://lists.litmus-rt.org/listinfo/litmus-dev</a><br>
</div></blockquote></div><br><br clear="all"><div><br></div></div></div><div class="im">-- <br>Jonathan Herman<br>Department of Computer Science at UNC Chapel Hill<br>
</div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Jonathan Herman<br>Department of Computer Science at UNC Chapel Hill<br>
</div>
_______________________________________________<br>litmus-dev mailing list<br><a href="mailto:litmus-dev@lists.litmus-rt.org">litmus-dev@lists.litmus-rt.org</a><br>https://lists.litmus-rt.org/listinfo/litmus-dev<br></blockquote></div><br></div></div></body></html>