<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hello all,<div class=""><br class=""></div><div class="">I have been working with Litmus-RT inside a Xen guest, but have come across the following issue: the guest crashes with the following</div><div class="">trace output by Xen:</div><div class=""><br class=""></div><div class=""><div class="">[25623.603388] ------------[ cut here ]------------</div><div class="">[25623.603499] kernel BUG at drivers/xen/events/events_base.c:1209!</div><div class="">[25623.603531] invalid opcode: 0000 [#1] PREEMPT SMP </div><div class="">[25623.603564] Modules linked in: x86_pkg_temp_thermal joydev coretemp ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64</div><div class="">[25623.603666] CPU: 0 PID: 31 Comm: xenwatch Tainted: G L 4.1.3+ #2</div><div class="">[25623.603700] task: ffff8801f4abe5c0 ti: ffff8801f4ba0000 task.ti: ffff8801f4ba0000</div><div class="">[25623.603735] RIP: e030:[<ffffffff8147e169>] [<ffffffff8147e169>] xen_send_IPI_one+0x59/0x60</div><div class="">[25623.603784] RSP: e02b:ffff8801f4ba3a78 EFLAGS: 00010086</div><div class="">[25623.603809] RAX: ffff8801f5ec8240 RBX: 0000000000000000 RCX: 0000000000000001</div><div class="">[25623.604058] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 00000000ffffffff</div><div class="">[25623.604058] RBP: ffff8801f4ba3a78 R08: 0000000000000000 R09: fffffffffffffd3d</div><div class="">[25623.604058] R10: 0000000000000001 R11: 0000000000050108 R12: 0000000000000001</div><div class="">[25623.604058] R13: ffff8801f5e92c60 R14: ffff88001bc21970 R15: 0000000000012c60</div><div class="">[25623.604058] FS: 00007f5c92344700(0000) GS:ffff8801f5e00000(0000) knlGS:0000000000000000</div><div class="">[25623.604058] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033</div><div class="">[25623.604058] CR2: 00007f5c92357000 CR3: 00000001d97df000 CR4: 0000000000042660</div><div class="">[25623.604058] Stack:</div><div class="">[25623.604058] ffff8801f4ba3a88 ffffffff81013fb0 ffff8801f4ba3aa8 ffffffff8139ff45</div><div class="">[25623.604058] ffff8801f4ba3b18 ffff880000000001 ffff8801f4ba3af8 ffffffff8139ceb8</div><div class="">[25623.604058] 0000000000000000 ffffffff817b0bc0 ffffffff819f54c2 0000000000000113</div><div class="">[25623.604058] Call Trace:</div><div class="">[25623.604058] [<ffffffff81013fb0>] xen_smp_send_reschedule+0x10/0x20</div><div class="">[25623.604058] [<ffffffff8139ff45>] litmus_reschedule+0x85/0xc0</div><div class="">[25623.604058] [<ffffffff8139ceb8>] preempt_if_preemptable+0xa8/0x150</div><div class="">[25623.604058] [<ffffffff813a5e91>] check_for_preemptions+0x201/0x3b0</div><div class="">[25623.604058] [<ffffffff813a633e>] gsnedf_release_jobs+0x3e/0x60</div><div class="">[25623.604058] [<ffffffff813a2163>] on_release_timer+0x73/0x80</div><div class="">[25623.604058] [<ffffffff810ddb16>] __run_hrtimer+0x76/0x290</div><div class="">[25623.604058] [<ffffffff813a20f0>] ? arm_release_timer_on+0x300/0x300</div><div class="">[25623.604058] [<ffffffff810de6d3>] hrtimer_interrupt+0x113/0x290</div><div class="">[25623.604058] [<ffffffff8176a9f1>] ? _raw_spin_unlock_irqrestore+0x21/0x40</div><div class="">[25623.604058] [<ffffffff810de8a5>] __hrtimer_peek_ahead_timers+0x55/0x60</div><div class="">[25623.604058] [<ffffffff810de9cb>] hrtimer_cpu_notify+0x11b/0x240</div><div class="">[25623.604058] [<ffffffff8176a9ae>] ? _raw_spin_unlock_irq+0x1e/0x40</div><div class="">[25623.604058] [<ffffffff8109693d>] notifier_call_chain+0x4d/0x70</div><div class="">[25623.604058] [<ffffffff810969fe>] __raw_notifier_call_chain+0xe/0x10</div><div class="">[25623.604058] [<ffffffff810774b0>] __cpu_notify+0x20/0x40</div><div class="">[25623.604058] [<ffffffff81077595>] cpu_notify_nofail+0x15/0x20</div><div class="">[25623.604058] [<ffffffff81756615>] _cpu_down+0x155/0x2b0</div><div class="">[25623.604058] [<ffffffff81481d20>] ? xenbus_thread+0x2a0/0x2a0</div><div class="">[25623.604058] [<ffffffff817567a5>] cpu_down+0x35/0x50</div><div class="">[25623.604058] [<ffffffff81479772>] handle_vcpu_hotplug_event+0x72/0xf0</div><div class="">[25623.604058] [<ffffffff81481dc7>] xenwatch_thread+0xa7/0x170</div><div class="">[25623.604058] [<ffffffff810b4d60>] ? prepare_to_wait_event+0x100/0x100</div><div class="">[25623.604058] [<ffffffff81095ff9>] kthread+0xc9/0xe0</div><div class="">[25623.604058] [<ffffffff81095f30>] ? flush_kthread_worker+0x90/0x90</div><div class="">[25623.604058] [<ffffffff8176b622>] ret_from_fork+0x42/0x70</div><div class="">[25623.604058] [<ffffffff81095f30>] ? flush_kthread_worker+0x90/0x90</div><div class="">[25623.604058] Code: ff ff 5d c3 bf 0b 00 00 00 48 63 f1 31 d2 e8 af 31 b8 ff 85 c0 79 eb 89 c2 89 ce 48 c7 c7 c8 d3 a1 81 31 c0 e8 05 15 2e 00 5d c3 <0f> 0b 0f 1f 44 00 00 66 66 66 66 90 55 48 89 e5 53 48 83 ec 08 </div><div class="">[25623.604058] RIP [<ffffffff8147e169>] xen_send_IPI_one+0x59/0x60</div><div class="">[25623.604058] RSP <ffff8801f4ba3a78></div><div class="">[25623.604058] ---[ end trace 702caae929567a1d ]---</div><div class="">[25623.604058] note: xenwatch[31] exited with preempt_count 1</div></div><div class=""><br class=""></div><div class="">I’m not sure whether the issue lies with Xen or Litmus-RT, but thought to try here first. I am adding and removing VCPUs from the VM at runtimes per</div><div class=""><a href="http://backdrift.org/how-to-hot-addremove-vcpus-from-a-xen-domain" class="">http://backdrift.org/how-to-hot-addremove-vcpus-from-a-xen-domain</a>, so that may be a factor here. The issue pops up intermittently, so I have not</div><div class="">been able to nail down steps to reproduce this.</div><div class=""><br class=""></div><div class="">Has anyone observed it or have an idea on how to resolve this? </div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Geoffrey</div></body></html>