[LITMUS^RT] IPI vs. Cache Affinity

[Daniel Bristot de Oliveira]

Hi Glenn


On Fri, Sep 27, 2013 at 8:23 PM, Glenn Elliott <gelliott at cs.unc.edu> wrote:

>
> On Sep 27, 2013, at 7:17 PM, Andrea Bastoni <bastoni at sprg.uniroma2.it>
> wrote:
>
> > Hi Glenn,
> >
> > On 09/27/2013 10:46 PM, Glenn Elliott wrote:
> >> I've been getting more acquainted with the latest litmus code and I see
> that
> >> GSN-EDF/C-EDF prefer to schedule a task on the local CPU if the CPU is
> >> available.  This avoids an IPI.  However, this may result in higher
> cache
> >> migration costs because this scheduling decision is made before
> checking CPU
> >> affinity.
> >
> > Umm, what is the use case you've in mind? Evaluating a local condition is
> > relatively cheap (as you've already all the locks you need). Preempting
> a remote
> > CPU is more expensive (locks to be reacquired, interrupt to be sent and
> > received, etc.).
> >
> >> Are IPI costs (significantly) higher than cache affinity loss?
> >
> > What's your working set size, what's the load of the system, what's your
> > architecture?
> >
> > Also, an IPI will cause an interrupt on the destination CPU, thus
> disturbing the
> > execution of the task already running there. This perturbation may be
> completely
> > pointless in the case the task is eventually not scheduled on the CPU
> being
> > preempted by the IPI.
> >
> >> Are we making
> >> the assumption that affinity has already been lost due to cache
> polluters?
> >
> > Are you assuming worst-case conditions?
> >
> > Thanks,
> > - Andrea
> >
> >> Do we have empirical data to support the current implementation?  I'm
> not
> >> arguing in favor of one method or the other---I am just interested in
> the
> >> motivations behind the changes.
> >>
> >> Thanks,
> >> Glenn
> >> _______________________________________________
> >> litmus-dev mailing list
> >> litmus-dev at lists.litmus-rt.org
> >> https://lists.litmus-rt.org/listinfo/litmus-dev
> >
> > --
> > Andrea Bastoni, PhD <bastoni at sprg.uniroma2.it>
> > Dept. of Computer Science, Systems, and Industrial Engineering
> > University of Rome "Tor Vergata", Via del Politecnico, 1 - 00133 Rome
> >
> > _______________________________________________
> > litmus-dev mailing list
> > litmus-dev at lists.litmus-rt.org
> > https://lists.litmus-rt.org/listinfo/litmus-dev
>
>
> Hi Andrea,
>
> Thank you for the deeper explanation.  I certainly appreciate the
> simplicity of local scheduling.  I will be investigating response times of
> cache heavy tasks under G-EDF/C-EDF over the next few weeks.  I'll report
> back if I see any noticeable trade-offs between local-first vs.
> affinity-first scheduling.
>
> -Glenn
> _______________________________________________
> litmus-dev mailing list
> litmus-dev at lists.litmus-rt.org
> https://lists.litmus-rt.org/listinfo/litmus-dev
>

I don't  have much experience with the behavior of Litmus RT, but, I've saw
some cases of high latency in the PREEMPT RT, on the real time tasks that
the scheduler chose to migrate to an idle processor before wake it up.

This issue happens on systems that use the mwait_idle for the cpu_idle. The
mwait_idle is the idle function for processors with the X86_FEATURE_MWAIT.

In this cases, when the processor is on idle state (this processors are
waked up by the pm, not by the schedule_ipi), sometimes, it presents a high
latency to wake up.

In the majority of times, the latency between the sched_wakeup tracepoint
and the context_switch  tracepoint is under 40 us, but sometimes, this
latency reach around 120 us, considering only the case where the idle task
do not block between wake up and leave the processor.

I know that it, much probably, happened because of the pm_idle, but it
happens very often. In the system that I saw this, the latency is lower
when all the processors are busy and FIFO scheduler choose to run the task
in the same processor.

I do not spent many time trying to understand this, it is not the main
objective of my work, but it shows a case that, if you know that the
current processor are ready to the context switch, like Andrea said, with
the desired locks etc, is better to choose wake up the task in the same
processor.

-- 
Daniel Bristot de Oliveira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.litmus-rt.org/pipermail/litmus-dev/attachments/20130927/c5af52b2/attachment.html>