[LITMUS^RT] switch_sched_plugin() bug?

[Björn Brandenburg]

On 24.09.2012, at 00:27, Felipe Cerqueira <felipeqcerqueira at gmail.com> wrote:

> I think there is a bug in the function switch_sched_plugin(). A
> deadlock occurs sometimes when the scheduler plugin is changed, while
> I run task sets in sequence.
> 
> I'm running Litmus on a dual-core system, so that may be the reason
> why the deadlock occurs easily, because smp_call_function_many() turns
> into smp_call_function_single().
> Have a look at this behaviour of switch_sched_plugin():
> 
> PROC 0 (the one which is trying to change the scheduler plugin):
> 1) Changes cannot_use_plugin to 1
> 2) Sends IPI for PROC 1 to call synch_on_plugin_switch()
> 
> PROC 1:
> 3) Receives IPI
> 4) Increment cannot_use_plugin to 2
> 5) Wait while cannot_use_plugin > 0
>    ... blocked.
> 
> PROC 0:
> 6) Still inside smp_call_function_single()
> 7) Hangs at csd_lock_wait() ... blocked.
> 
> Deadlock.
> 
> Proc 0 was supposed to free the other processor afterwards, but it's
> blocked trying to obtain csd lock.
> Proc 1, on the other hand, would call csd_unlock() only after the IPI
> function returned.
> 
> It's said in kernel/smp.c that we should use non-blocking functions
> when calling IPIs.
> Perhaps in smp_call_function_many() case (in a system with more
> processors) the csd lock is removed by some other processor and the
> deadlock doesn't happen.
> 
> Isn't there another way to implement this cannot_use_plugin barrier?
> Or the implementation is right and the bug is in my kernel version?

Yes, that's always been an ugly hack, which has been patched multiple times in the past but never worked quite right... I've seen deadlock related to plugin switching in recent kernels as well, but didn't have the time to debug it yet.

Plugin switching doesn't have to be fast, but it needs to ensure that none of the processors is currently executing plugin code. I think a proper solution could be based on the "kernel freezer" that stops all but one CPU, or spawning a bunch of kthreads to execute the delay loop in process context.

Patches welcome… ;-)

Thanks,
Björn