<blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Followup question: Do these changes allow for backwards-compatibility with existing Litmus plugins? That is, core data structures are only changed to an extent that existing non-container plugins would only need small (or no) updates.</span></blockquote>
<div> </div>Ack, forgot to mention that. I believe minor changes to existing plugins will be required, but nothing major. This work would mostly be related to using struct rt_task instead of struct task_struct to make scheduling decisions. For example, all plugins would be returning struct rt_tasks from their _schedule() methods, and some code would have to change to reflect the migration of data to task->rt_param.server (the task's struct rt_task) from struct rt_params.<div>
<div><br></div><div>I want to keep changes minor, but I don't think it will be possible to cause zero changes.</div><div><br><div class="gmail_quote">On Mon, Feb 27, 2012 at 3:04 PM, Glenn Elliott <span dir="ltr"><<a href="mailto:gelliott@cs.unc.edu">gelliott@cs.unc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div></div><div class="h5"><br><div><div>On Feb 27, 2012, at 2:54 PM, Jonathan Herman wrote:</div>
<br><blockquote type="cite"><br><br><div class="gmail_quote">On Mon, Feb 27, 2012 at 1:23 PM, Glenn Elliott <span dir="ltr"><<a href="mailto:gelliott@cs.unc.edu" target="_blank">gelliott@cs.unc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div><div><div></div><div><div>On Feb 26, 2012, at 9:40 PM, Jonathan Herman wrote:</div><br><blockquote type="cite"><div><div>I will be adding container support to litmus over the next</div>
<div>year. Exciting. My goals are to support a hierarchical container</div><div>scheme where each container can have its own scheduling policy and a</div>
<div>single container can be run on multiple CPUs simultaneously.</div></div><div><br></div><div>This is my idea and I would love it to be critiqued.</div><div><br></div><div><b>-- Interface --</b></div><div>Tasks should be able to join and leave containers dynamically, but I</div>
<div>don't intend to support (initially) dynamic creation of a container</div><div>hierarchy. Instead, plugins will configure their own containers.</div><div><br></div><div>I intend to completely copy the container proc interface from</div>
<div>Linux. The interface looks something like this:</div><div> - <plugin>/<container-id>*/tasks</div><div>Tasks are added and removed from containers by echoing their IDs into</div><div>the tasks file of each container.</div>
<div><br></div><div>For example, a CEDF plugin with containers would work as follows:</div><div>1. The scheduling plugin CODE would create a container for each</div><div>cluster, and the container framework would automatically expose them</div>
<div>under proc:</div><div> - CEDF/0/tasks</div><div> - CEDF/1/tasks</div><div> - CEDF/2/tasks</div><div> - CEDF/3/tasks</div><div>2. The scheduling plugin USER would create tasks, and echo their PIDs</div><div>into these containers as he/she sees fit.</div>
</blockquote><div><br></div></div></div><div>Having to use fwrite() to join a container seems a bit heavy handed. Why the departure from the mechanism used to specify CPU partitioning? Perhaps a system call could return to the user a description of the container hierarchy. A task could traverse this hierarchy and join the container with a given identifier. I would also appreciate an interface, to be used from within the kernel, for migrating tasks between containers.</div>
<div><br></div><div>What happens when a scheduled task changes containers?</div></div></div></blockquote><br>The following interface description might address these issues better:<br>A Kernel interface will be provided for adding tasks to and moving tasks between containers. Kernelspace (e.g. plugins) may use this interface directly to migrate tasks between containers. The interface is exposed to userspace through the proc interface mentioned above.</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">The proc interface is designed for simplicity when scripting and running task sets. Using a method similar to -p to access hierarchies of containers could get confusing (e.g., how do I remove a task from a container?). Additionally, the interface is expandable later if we want to add additional per-container features, such as memory allocation. However, should tasks need to dynamically leave and join containers at run time from -userspace- at performance-critical times, the overhead of fwrite() may be an issue. Calls would have to be added to liblitmus for container membership for these cases.</div>
<div class="gmail_quote"><br>Thanks for pointing that out, I was only picturing tasks entering and leaving containers during initialization.<br><br>><br>><br>> What happens when a scheduled task changes containers?<br>
<br>I imagine the result would be:</div><div class="gmail_quote"><br><font face="'courier new', monospace">source->policy->ops->remove(source, task);<br>/* As that container was previously running, it now selects </font></div>
<div class="gmail_quote"><font face="'courier new', monospace"> * another task to run after task is unlinked.<br></font><span style="font-family:'courier new',monospace"> */</span></div><div class="gmail_quote">
<font face="'courier new', monospace">dest->policy->ops->add(dest, task);<br>/* If this container is running and task is of sufficiently </font></div><div class="gmail_quote"><font face="'courier new', monospace"> * high priority, it will now run on one of dest's rt_cpus.</font></div>
<div class="gmail_quote"><font face="'courier new', monospace"> */</font><br><br><div> So unless the other container can run the task, the scheduled task will have to stop.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div><div><div></div><div><br><blockquote type="cite"><div><b>-- The framework --</b></div><div>The framework I'm showing is designed with the following thoughts:</div><div>
* Eventual support for less ad-hoc slack stealing</div><div>* Not breaking locking code too much</div>
<div>* Trying to minimize amount of extra data being passed around</div><div>* Keeping as much container code out of the plugins as possible</div><div><br></div><div>The basic idea is to have rt_task's become the fundamental schedulable</div>
<div>entity in Litmus. An rt_task can correspond to a struct task_struct,</div><div>as they do now, or an rt_cpu. I want to use these rt_tasks to abstract</div><div>out the code needed to manage container hierarchies. Per-plugin</div>
<div>scheduling code should not have to manage containers at all after initialization.</div><div><br></div><div>An rt_cpu is a virtual processor which can run other tasks. It can</div><div>have a task which is @linked to it, and it optionally enforces budget</div>
<div>with an @timer. The virtual processor is run when its corresponding</div><div>rt_task, or @server, is selected to run. When the rt_cpu is selected</div><div>to run, it chooses a task to execute by asking its corresponding</div>
<div>@container.</div><div><br></div><div><font face="'courier new', monospace">struct rt_cpu {</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>unsigned int cpu; /* Perhaps an RT_GLOBAL macro corresponds to</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * a wandering global virtual processor?</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> */</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct rt_task *server; /* 0xdeadbeef for no server maybe?</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * I'm thinking of doing something</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * special for servers which have</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * full utilization of a processor,</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * as servers in the BASE container</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * will.</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> */</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct rt_task *linked; /* What is logically running */</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct enforcement_timer timer;</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct bheap_node *hn; /* For membership in heaps */</font></div><div><font face="'courier new', monospace"><br>
</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct rt_container *container; /* Clearly necessary */</font></div><div><font face="'courier new', monospace">};</font></div>
<div><br></div><div>An rt_container struct is a group of tasks scheduled together. The container</div><div>can run tasks when one or more @procs are selected to run. When a</div><div>container can run a task, it selects the next task to run using a</div>
<div>@policy.</div><div><br></div><div><font face="'courier new', monospace">struct rt_container {</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>/* Potentially one of these for each CPU */</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct rt_cpu *procs;</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>cpumask_t cpus; /* Or perhaps num_cpus? I want O(1) access to</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * partitioned CPUs, but a container may also</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * have multiple global rt_cpus. I am not sure</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * how to accomplish O(1) access with global</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * rt_cpus. Ideas? I'll try to think of something.</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> */</font></div><div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>/* To create the container hieararchy */</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct rt_container *parent;</font></div><div><font face="'courier new', monospace"><br></font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>/* The per-container method for scheduling container task */</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct *rt_policy policy;</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>/* Metadata kept seperate from the rest of the container</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * because it is not used often. E.g. a task list, proc</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> * entries, or a container name would be stored in this.</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span> */</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>struct rt_cont_data *data;</font></div>
<div><font face="'courier new', monospace">};</font></div></blockquote><div><br></div></div></div><div>Is a given policy instance static? That is, is a single G-EDF policy instance shared between containers, or are there several distinct G-EDF policy instances, one per container?</div>
</div></div></blockquote><div><br></div><div>The policy->ops are static, in that all G-EDF policies will point to the same ops struct. The data wrapped around the rt_policy struct, like rt_domains, is distinct per container.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><br></div><div>In general, I like this interface. It appears clean to me.</div>
</div></div></blockquote><div>Thanks!</div></div><div><br></div>-- <br>Jonathan Herman<br>Department of Computer Science at UNC Chapel Hill<br></blockquote></div><div><br></div><br></div></div><div>Followup question: Do these changes allow for backwards-compatibility with existing Litmus plugins? That is, core data structures are only changed to an extent that existing non-container plugins would only need small (or no) updates.</div>
<div><br></div><font color="#888888"><div>-Glenn</div><div><br></div></font></div><br>_______________________________________________<br>
litmus-dev mailing list<br>
<a href="mailto:litmus-dev@lists.litmus-rt.org">litmus-dev@lists.litmus-rt.org</a><br>
<a href="https://lists.litmus-rt.org/listinfo/litmus-dev" target="_blank">https://lists.litmus-rt.org/listinfo/litmus-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Jonathan Herman<br>Department of Computer Science at UNC Chapel Hill<br>
</div></div>