<div dir="ltr">figures for the data are attached in case it does not show up.<div><br></div><div>Thanks.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 21 February 2017 at 14:20, Shuai Zhao <span dir="ltr"><<a href="mailto:zs673@york.ac.uk" target="_blank">zs673@york.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Björn<div><br></div><div>Thank you for your respond.<div><br></div><div>Yes, the kmalloc function is now out of the critical section.</div><div><br></div><div>The code I posted in the email is protected by a spin lock to avoid race conditions. The tickets are maintained and obtained by atomic_t variables.</div><div><br></div><div>Using the feather-trace tool we get a csv file, where all the overheads are recorded. I noticed that you processed the data as max, 99.9prec, 99prec and 95prec. I wonder what is the rational behind this? </div><div><br></div><div>Is that the 99,9prec or 99prec result filter out some of the out-layer data which is influenced by the system overheads or interrupts? </div><div><br></div><div>For example: we tried to gather the overheads of the original pfp-scheduler. we did this experiment with a increasing number of processors and expect a constant overheads. However, the data we have is confusing. The samples for each test is above one million.</div><div><br></div><div><img src="cid:ii_izfkbuxw1_15a60d80f3dd6f65" style="margin-right:0px"><br></div><div>We gather this data inside the pfp-scheduler (we put time stamps inside the run queue locks) to get the exact overheads for executing the scheduling code. The result above gives the max overhead we observed in each test.</div><div><br></div><div>As shown in the result, the overheads of the pfp-scheduler is extremely high when using cpu 1,2 and 4. By repeating the same tests, we can often observe such a extreme value, but with different number of cpus.</div><div><br></div><div><br></div><div>A more simple example: the lock function that I post on previous email with the kmalloc removed. This part of code has only one path and is O(1), shich suppose to have a stable overhead.</div><div><img src="cid:ii_izfklgs72_15a60dee4e28f8e2" style="margin-right:0px"></div><div>However, as shown above, the overheads of the code is extremely high with 2,14 and 15 cpus.</div><div><br></div><div>I wonder have you met any sitations like this before? and how you explain such a result or how you solve the problem(if there is).<br></div><div><br></div><div>Do you have any suggestions when facing such a confusing result?</div><span class=""><div><br></div><div>Thank you in advance.</div><div><br></div><div>Best wishes</div><div>Shuai </div></span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 20 February 2017 at 11:14, Shuai Zhao <span dir="ltr"><<a href="mailto:zs673@york.ac.uk" target="_blank">zs673@york.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Meng<div><br></div><div>Thank you so much for this information.</div><div><br></div><div>I will have a try :).</div><div><br></div><div>Best wishes</div><span class="m_-1558795222963305823HOEnZb"><font color="#888888"><div>Shuai</div></font></span></div><div class="m_-1558795222963305823HOEnZb"><div class="m_-1558795222963305823h5"><div class="gmail_extra"><br><div class="gmail_quote">On 20 February 2017 at 11:00, <span dir="ltr"><<a href="mailto:litmus-dev-request@lists.litmus-rt.org" target="_blank">litmus-dev-request@lists.litm<wbr>us-rt.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send litmus-dev mailing list submissions to<br>
<a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.litmus-rt.org/listinfo/litmus-dev" rel="noreferrer" target="_blank">https://lists.litmus-rt.org/li<wbr>stinfo/litmus-dev</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:litmus-dev-request@lists.litmus-rt.org" target="_blank">litmus-dev-request@lists.litmu<wbr>s-rt.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:litmus-dev-owner@lists.litmus-rt.org" target="_blank">litmus-dev-owner@lists.litmus-<wbr>rt.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of litmus-dev digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: litmus-dev Digest, Vol 60, Issue 2 (Meng Xu)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Sun, 19 Feb 2017 17:34:52 -0500<br>
From: Meng Xu <<a href="mailto:xumengpanda@gmail.com" target="_blank">xumengpanda@gmail.com</a>><br>
To: <a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
Subject: Re: [LITMUS^RT] litmus-dev Digest, Vol 60, Issue 2<br>
Message-ID:<br>
<CAENZ-+kzpYR3kHNBZ=<a href="mailto:kDVaJMgvRvbViyQ-j256xCNWwQh9Eo0A@mail.gmail.com" target="_blank">kDVaJMgvRv<wbr>bViyQ-j256xCNWwQh9Eo0A@mail.gm<wbr>ail.com</a>><br>
Content-Type: text/plain; charset=UTF-8<br>
<br>
On Sun, Feb 19, 2017 at 5:31 PM, Shuai Zhao <<a href="mailto:zs673@york.ac.uk" target="_blank">zs673@york.ac.uk</a>> wrote:<br>
> Hi Meng<br>
><br>
> Thank you for your fast respond.<br>
><br>
> Yes, the CPU prefetch is already disabled. But there isn't any other options<br>
> of the hardware prefetching. But I guess its should be OK.<br>
><br>
> The kmalloc() function can be one of the reasons. I will adjust the code.<br>
><br>
> BTW, is there any other features or facilities that could be disabled to<br>
> minimise the system interferences?<br>
<br>
How about the cache prefecthing? Did you disable it?<br>
You also need to keep the memory bus frequency constant.<br>
<br>
But I think such large variance may mainly come from the kmalloc().<br>
<br>
Best,<br>
<br>
Meng<br>
<br>
<br>
><br>
><br>
> On 19 February 2017 at 21:27, <<a href="mailto:litmus-dev-request@lists.litmus-rt.org" target="_blank">litmus-dev-request@lists.litm<wbr>us-rt.org</a>><br>
> wrote:<br>
>><br>
>> Send litmus-dev mailing list submissions to<br>
>> <a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.or<wbr>g</a><br>
>><br>
>> To subscribe or unsubscribe via the World Wide Web, visit<br>
>> <a href="https://lists.litmus-rt.org/listinfo/litmus-dev" rel="noreferrer" target="_blank">https://lists.litmus-rt.org/l<wbr>istinfo/litmus-dev</a><br>
>> or, via email, send a message with subject or body 'help' to<br>
>> <a href="mailto:litmus-dev-request@lists.litmus-rt.org" target="_blank">litmus-dev-request@lists.litm<wbr>us-rt.org</a><br>
>><br>
>> You can reach the person managing the list at<br>
>> <a href="mailto:litmus-dev-owner@lists.litmus-rt.org" target="_blank">litmus-dev-owner@lists.litmus<wbr>-rt.org</a><br>
>><br>
>> When replying, please edit your Subject line so it is more specific<br>
>> than "Re: Contents of litmus-dev digest..."<br>
>><br>
>><br>
>> Today's Topics:<br>
>><br>
>> 1. A question about feather-trace tool (Shuai Zhao)<br>
>> 2. Re: A question about feather-trace tool (Shuai Zhao)<br>
>> 3. Re: A question about feather-trace tool (Meng Xu)<br>
>><br>
>><br>
>> ------------------------------<wbr>------------------------------<wbr>----------<br>
>><br>
>> Message: 1<br>
>> Date: Sun, 19 Feb 2017 20:32:17 +0000<br>
>> From: Shuai Zhao <<a href="mailto:zs673@york.ac.uk" target="_blank">zs673@york.ac.uk</a>><br>
>> To: <a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
>> Subject: [LITMUS^RT] A question about feather-trace tool<br>
>> Message-ID:<br>
>><br>
>> <<a href="mailto:CAA133hO%2B_4Svyx6dJUyUn_rsnY50AfUVxLyOHKcsUN6%2BvG8UiQ@mail.gmail.com" target="_blank">CAA133hO+_4Svyx6dJUyUn_rsnY50<wbr>AfUVxLyOHKcsUN6+vG8UiQ@mail.gm<wbr>ail.com</a>><br>
>> Content-Type: text/plain; charset="utf-8"<br>
>><br>
>> Hi Björn<br>
>><br>
>> I am a student from the University of York working with Alan and Andy to<br>
>> study MrsP nested behaviours now.<br>
>><br>
>> We now have a full implementation of nested MrsP under Litmus P-FP<br>
>> scheduler and we are now trying to do some evaluations of the<br>
>> implementation overheads.<br>
>><br>
>> We use the feather-trace tool to trace the overheads of the scheduler<br>
>> (which includes the P-FP schedule function), context switch (includes<br>
>> finish_switch function), mrsp_lock and mrsp_unlock function.<br>
>><br>
>> During evaluation, we fixed the CPU clock speed, bounds interrupts to cpu<br>
>> 0<br>
>> and isolate other cpus for testing to minimise the interference from the<br>
>> system.<br>
>><br>
>><br>
>> However, the result seems werid. Use mrsp_lock as an example: we evaluated<br>
>> the overhead of the following code using the timestamps "TS_LOCK_START"<br>
>> and<br>
>> "TS_LOCK_END".<br>
>><br>
>> TS_LOCK_START;<br>
>><br>
>> if (t->rt_param.task_params.helpe<wbr>r == NULL) {<br>
>> t->rt_param.task_params.priori<wbr>ty = sem->prio_per_cpu[get_partitio<wbr>n(t)] <<br>
>> get_priority(t) ? sem->prio_per_cpu[get_partitio<wbr>n(t)] : get_priority(t);<br>
>> t->rt_param.task_params.migrat<wbr>ed_time = -1;<br>
>> }<br>
>><br>
>> ticket = atomic_read(&sem->next_ticket)<wbr>;<br>
>> t->rt_param.task_params.ticket = ticket;<br>
>> atomic_inc(&sem->next_ticket);<br>
>><br>
>> add_task(t, &(sem->tasks_queue->next));<br>
>> t->rt_param.task_params.reques<wbr>ting_lock = sem;<br>
>><br>
>> TS_LOCK_END;<br>
>><br>
>> Where function add_task() is as follows:<br>
>><br>
>> void add_task(struct task_struct* task, struct list_head *head) {<br>
>> struct task_list *taskPtr = (struct task_list *) kmalloc(sizeof(struct<br>
>> task_list), GFP_KERNEL);<br>
>> BUG_ON(taskPtr == NULL);<br>
>><br>
>> taskPtr->task = task;<br>
>> INIT_LIST_HEAD(&taskPtr->next)<wbr>;<br>
>> list_add_tail(&taskPtr->next, head);<br>
>> }<br>
>><br>
>> We expect the overheads of the code above to be stable as the time<br>
>> complexity is order 1. However, the testing result gives us a different<br>
>> story, as shown below:<br>
>><br>
>> Overhead Overhead Unit Samples MAX<br>
>> 99.9th perc. 99perc. 95th perc. avg med<br>
>> min std var<br>
>> MRSP LOCK cycles 149985 20089<br>
>> 2220.016 1584 1287 853.9508 939<br>
>> 319 367.314 134918.7<br>
>><br>
>> As we can see, the max overheads we have is 20089 cycles, which is far<br>
>> bigger than the med/avg value and even the 99.9the perc value.<br>
>><br>
>> I am confused about this result. I wonder have you meet this situation<br>
>> before? Is there any explanation for the result like this? or is there any<br>
>> way to avoid this?<br>
>><br>
>> Thank you in advance.<br>
>><br>
>> Best wishes<br>
>> Shuai<br>
>> -------------- next part --------------<br>
>> An HTML attachment was scrubbed...<br>
>> URL:<br>
>> <<a href="http://lists.litmus-rt.org/pipermail/litmus-dev/attachments/20170219/d6859058/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.litmus-rt.org/pi<wbr>permail/litmus-dev/attachments<wbr>/20170219/d6859058/attachment-<wbr>0001.html</a>><br>
>><br>
>> ------------------------------<br>
>><br>
>> Message: 2<br>
>> Date: Sun, 19 Feb 2017 20:37:52 +0000<br>
>> From: Shuai Zhao <<a href="mailto:zs673@york.ac.uk" target="_blank">zs673@york.ac.uk</a>><br>
>> To: <a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
>> Subject: Re: [LITMUS^RT] A question about feather-trace tool<br>
>> Message-ID:<br>
>><br>
>> <<a href="mailto:CAA133hM4boX5fLX1%2BPERwbxtvymOFOonn0wwV2r3q9s3hehBog@mail.gmail.com" target="_blank">CAA133hM4boX5fLX1+PERwbxtvymO<wbr>FOonn0wwV2r3q9s3hehBog@mail.gm<wbr>ail.com</a>><br>
>> Content-Type: text/plain; charset="utf-8"<br>
>><br>
>> PS: I commented out the original "TS_LOCK_START" and "TS_LOCK_END" stamps<br>
>> in sys_lock function. thanks.<br>
>><br>
>> On 19 February 2017 at 20:32, Shuai Zhao <<a href="mailto:zs673@york.ac.uk" target="_blank">zs673@york.ac.uk</a>> wrote:<br>
>><br>
>> > Hi Björn<br>
>> ><br>
>> > I am a student from the University of York working with Alan and Andy to<br>
>> > study MrsP nested behaviours now.<br>
>> ><br>
>> > We now have a full implementation of nested MrsP under Litmus P-FP<br>
>> > scheduler and we are now trying to do some evaluations of the<br>
>> > implementation overheads.<br>
>> ><br>
>> > We use the feather-trace tool to trace the overheads of the scheduler<br>
>> > (which includes the P-FP schedule function), context switch (includes<br>
>> > finish_switch function), mrsp_lock and mrsp_unlock function.<br>
>> ><br>
>> > During evaluation, we fixed the CPU clock speed, bounds interrupts to<br>
>> > cpu<br>
>> > 0 and isolate other cpus for testing to minimise the interference from<br>
>> > the<br>
>> > system.<br>
>> ><br>
>> ><br>
>> > However, the result seems werid. Use mrsp_lock as an example: we<br>
>> > evaluated<br>
>> > the overhead of the following code using the timestamps "TS_LOCK_START"<br>
>> > and<br>
>> > "TS_LOCK_END".<br>
>> ><br>
>> > TS_LOCK_START;<br>
>> ><br>
>> > if (t->rt_param.task_params.helpe<wbr>r == NULL) {<br>
>> > t->rt_param.task_params.priori<wbr>ty = sem->prio_per_cpu[get_partitio<wbr>n(t)] <<br>
>> > get_priority(t) ? sem->prio_per_cpu[get_partitio<wbr>n(t)] : get_priority(t);<br>
>> > t->rt_param.task_params.migrat<wbr>ed_time = -1;<br>
>> > }<br>
>> ><br>
>> > ticket = atomic_read(&sem->next_ticket)<wbr>;<br>
>> > t->rt_param.task_params.ticket = ticket;<br>
>> > atomic_inc(&sem->next_ticket);<br>
>> ><br>
>> > add_task(t, &(sem->tasks_queue->next));<br>
>> > t->rt_param.task_params.reques<wbr>ting_lock = sem;<br>
>> ><br>
>> > TS_LOCK_END;<br>
>> ><br>
>> > Where function add_task() is as follows:<br>
>> ><br>
>> > void add_task(struct task_struct* task, struct list_head *head) {<br>
>> > struct task_list *taskPtr = (struct task_list *) kmalloc(sizeof(struct<br>
>> > task_list), GFP_KERNEL);<br>
>> > BUG_ON(taskPtr == NULL);<br>
>> ><br>
>> > taskPtr->task = task;<br>
>> > INIT_LIST_HEAD(&taskPtr->next)<wbr>;<br>
>> > list_add_tail(&taskPtr->next, head);<br>
>> > }<br>
>> ><br>
>> > We expect the overheads of the code above to be stable as the time<br>
>> > complexity is order 1. However, the testing result gives us a different<br>
>> > story, as shown below:<br>
>> ><br>
>> > Overhead Overhead Unit Samples MAX<br>
>> > 99.9th perc. 99perc. 95th perc. avg med<br>
>> > min std var<br>
>> > MRSP LOCK cycles 149985 20089<br>
>> > 2220.016 1584 1287 853.9508 939<br>
>> > 319 367.314 134918.7<br>
>> ><br>
>> > As we can see, the max overheads we have is 20089 cycles, which is far<br>
>> > bigger than the med/avg value and even the 99.9the perc value.<br>
>> ><br>
>> > I am confused about this result. I wonder have you meet this situation<br>
>> > before? Is there any explanation for the result like this? or is there<br>
>> > any<br>
>> > way to avoid this?<br>
>> ><br>
>> > Thank you in advance.<br>
>> ><br>
>> > Best wishes<br>
>> > Shuai<br>
>> ><br>
>> -------------- next part --------------<br>
>> An HTML attachment was scrubbed...<br>
>> URL:<br>
>> <<a href="http://lists.litmus-rt.org/pipermail/litmus-dev/attachments/20170219/824d81ab/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.litmus-rt.org/pi<wbr>permail/litmus-dev/attachments<wbr>/20170219/824d81ab/attachment-<wbr>0001.html</a>><br>
>><br>
>> ------------------------------<br>
>><br>
>> Message: 3<br>
>> Date: Sun, 19 Feb 2017 16:27:26 -0500<br>
>> From: Meng Xu <<a href="mailto:xumengpanda@gmail.com" target="_blank">xumengpanda@gmail.com</a>><br>
>> To: <a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
>> Subject: Re: [LITMUS^RT] A question about feather-trace tool<br>
>> Message-ID:<br>
>><br>
>> <CAENZ-+mfQQ+-5qxbJOyh2UEPFrhv<wbr>Pp2OzsB4dYXUG_r=<a href="mailto:tNj95A@mail.gmail.com" target="_blank">tNj95A@mail.gm<wbr>ail.com</a>><br>
>> Content-Type: text/plain; charset=UTF-8<br>
>><br>
>> On Sun, Feb 19, 2017 at 3:32 PM, Shuai Zhao <<a href="mailto:zs673@york.ac.uk" target="_blank">zs673@york.ac.uk</a>> wrote:<br>
>> ><br>
>> > Hi Björn<br>
>><br>
>><br>
>> Hi,<br>
>><br>
>> Can I hijack the question? ;-)<br>
>><br>
>> ><br>
>> ><br>
>> > I am a student from the University of York working with Alan and Andy to<br>
>> > study MrsP nested behaviours now.<br>
>> ><br>
>> > We now have a full implementation of nested MrsP under Litmus P-FP<br>
>> > scheduler and we are now trying to do some evaluations of the implementation<br>
>> > overheads.<br>
>> ><br>
>> > We use the feather-trace tool to trace the overheads of the scheduler<br>
>> > (which includes the P-FP schedule function), context switch (includes<br>
>> > finish_switch function), mrsp_lock and mrsp_unlock function.<br>
>> ><br>
>> > During evaluation, we fixed the CPU clock speed, bounds interrupts to<br>
>> > cpu 0 and isolate other cpus for testing to minimise the interference from<br>
>> > the system.<br>
>><br>
>><br>
>> Did you disable the hardware prefetching mechanisms in BIOS?<br>
>> Maybe you want to disable them as well.<br>
>><br>
>><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > However, the result seems werid. Use mrsp_lock as an example: we<br>
>> > evaluated the overhead of the following code using the timestamps<br>
>> > "TS_LOCK_START" and "TS_LOCK_END".<br>
>> ><br>
>> > TS_LOCK_START;<br>
>> ><br>
>> > if (t->rt_param.task_params.helpe<wbr>r == NULL) {<br>
>> > t->rt_param.task_params.priori<wbr>ty = sem->prio_per_cpu[get_partitio<wbr>n(t)] <<br>
>> > get_priority(t) ? sem->prio_per_cpu[get_partitio<wbr>n(t)] : get_priority(t);<br>
>> > t->rt_param.task_params.migrat<wbr>ed_time = -1;<br>
>> > }<br>
>> ><br>
>> > ticket = atomic_read(&sem->next_ticket)<wbr>;<br>
>> > t->rt_param.task_params.ticket = ticket;<br>
>> > atomic_inc(&sem->next_ticket);<br>
>> ><br>
>> > add_task(t, &(sem->tasks_queue->next));<br>
>> > t->rt_param.task_params.reques<wbr>ting_lock = sem;<br>
>> ><br>
>> > TS_LOCK_END;<br>
>> ><br>
>> > Where function add_task() is as follows:<br>
>> ><br>
>> > void add_task(struct task_struct* task, struct list_head *head) {<br>
>> > struct task_list *taskPtr = (struct task_list *) kmalloc(sizeof(struct<br>
>> > task_list),<br>
>> ><br>
>> > GFP_KERNEL);<br>
>><br>
>><br>
>> I guess the variance comes from the kmalloc().<br>
>> kmalloc() latency varies a lot.<br>
>><br>
>> You should avoid kmalloc() in the critical section.<br>
>> One approach is pre-allocating the space, re-initializing the value<br>
>> every time when you want to use it.<br>
>><br>
>> You can also measure the overhead value spent in this function to<br>
>> validate my speculation.<br>
>><br>
>><br>
>> ><br>
>> > BUG_ON(taskPtr == NULL);<br>
>> ><br>
>> > taskPtr->task = task;<br>
>> > INIT_LIST_HEAD(&taskPtr->next)<wbr>;<br>
>> > list_add_tail(&taskPtr->next, head);<br>
>> > }<br>
>> ><br>
>> > We expect the overheads of the code above to be stable as the time<br>
>> > complexity is order 1. However, the testing result gives us a different<br>
>> > story, as shown below:<br>
>> ><br>
>> > Overhead Overhead Unit Samples MAX<br>
>> > 99.9th perc. 99perc. 95th perc. avg med<br>
>> > min std var<br>
>> > MRSP LOCK cycles 149985 20089<br>
>> > 2220.016 1584 1287 853.9508 939<br>
>> > 319 367.314 134918.7<br>
>><br>
>><br>
>> This is a bit difficulty to read. Next time when you send the new<br>
>> result, could you please send it in column. ;-)<br>
>><br>
>><br>
>> ><br>
>> ><br>
>> > As we can see, the max overheads we have is 20089 cycles, which is far<br>
>> > bigger than the med/avg value and even the 99.9the perc value.<br>
>> ><br>
>> > I am confused about this result. I wonder have you meet this situation<br>
>> > before? Is there any explanation for the result like this? or is there any<br>
>> > way to avoid this?<br>
>><br>
>><br>
>> You should be able to avoid this, as I mentioned above.<br>
>><br>
>> Best,<br>
>><br>
>> Meng<br>
>><br>
>><br>
>> --<br>
>> -----------<br>
>> Meng Xu<br>
>> PhD Student in Computer and Information Science<br>
>> University of Pennsylvania<br>
>> <a href="http://www.cis.upenn.edu/~mengxu/" rel="noreferrer" target="_blank">http://www.cis.upenn.edu/~meng<wbr>xu/</a><br>
>><br>
>><br>
>><br>
>> ------------------------------<br>
>><br>
>> Subject: Digest Footer<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> litmus-dev mailing list<br>
>> <a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
>> <a href="https://lists.litmus-rt.org/listinfo/litmus-dev" rel="noreferrer" target="_blank">https://lists.litmus-rt.org/li<wbr>stinfo/litmus-dev</a><br>
>><br>
>><br>
>> ------------------------------<br>
>><br>
>> End of litmus-dev Digest, Vol 60, Issue 2<br>
>> ******************************<wbr>***********<br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> litmus-dev mailing list<br>
> <a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
> <a href="https://lists.litmus-rt.org/listinfo/litmus-dev" rel="noreferrer" target="_blank">https://lists.litmus-rt.org/li<wbr>stinfo/litmus-dev</a><br>
><br>
<br>
<br>
<br>
--<br>
-----------<br>
Meng Xu<br>
PhD Student in Computer and Information Science<br>
University of Pennsylvania<br>
<a href="http://www.cis.upenn.edu/~mengxu/" rel="noreferrer" target="_blank">http://www.cis.upenn.edu/~meng<wbr>xu/</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
______________________________<wbr>_________________<br>
litmus-dev mailing list<br>
<a href="mailto:litmus-dev@lists.litmus-rt.org" target="_blank">litmus-dev@lists.litmus-rt.org</a><br>
<a href="https://lists.litmus-rt.org/listinfo/litmus-dev" rel="noreferrer" target="_blank">https://lists.litmus-rt.org/li<wbr>stinfo/litmus-dev</a><br>
<br>
<br>
------------------------------<br>
<br>
End of litmus-dev Digest, Vol 60, Issue 4<br>
******************************<wbr>***********<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>