<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hello All,</div><div><br></div><div>I've pushed a branch to github named "prop/uncachedev" I think its inclusion in mainline would be a great help in our continued work with caches.</div><div><br></div><div>From the commit (434eef21cb):</div><div><br></div><div>This patch creates a new character device, uncachedev.</div><div>Pages of RAM allocated by this device are not cached by</div><div>CPUs.</div><div><br></div><div>Uses for such pages:</div><div>1) Determining *very* pessimistic emperical worst-</div><div> case execution times.</div><div>2) Compare against performance with caches (quantify</div><div> the avg. case benefit).</div><div>3) Deterministic memory accesses (access cannot cause a</div><div> cache eviction.)</div><div>4) Theoretically, increased performance can be achieved</div><div> by storing infrequently accessed data in uncache pages.</div><div><br></div><div>uncachedev allocates pages with the pgprot_noncached() page</div><div>attribute for user applications. Since pages allocated by</div><div>uncachedev are not locked in memory by default, applications</div><div>with any access level may mmap pages with uncachedev.</div><div><br></div><div>Limitations:</div><div>1) Uncache pages must be MAP_PRIVATE.</div><div>2) Remapping not supported.</div><div><br></div><div>Usage (user level):</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>int size = NR_PAGES*PAGE_SIZE;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>int fd = open("/dev/litmus/uncache", O_RDWR);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>char *data = mmap(NULL, size, PROT_READ | PROT_WRITE,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>MAP_PRIVATE, fd, 0); /* must be MAP_PRIVATE */</div><div><span class="Apple-tab-span" style="white-space:pre"> </span><...do stuff...></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>munmap(data, size);</div><div><br></div><div>I'll also be pushing a companion patch to liblitmus that adds a test tool for uncachedev. It can be used test uncachedev as well as profile the cost of uncacheable memory. With unoptimized code on Bonham (Intel Xeon X5650), execution times with uncached memory were 30x greater than normal memory. However, when my test code was compiled with -O2, this gap grew to be over 100x! This huge difference is due to the very small working set size of my test application (just the first byte of a uncache page). I verified that the system behaved properly by profiling with tools/perf-stat. perf reports the expected number of cache misses for uncache data.</div><div><br></div><div>Here's a sample of perf's output (I forced all computation and memory to reside on a single NUMA node).</div><div><br></div><div>Uncached memory. Nearly 100% cache misses.</div><div><div></div><blockquote type="cite"><div>root@bonham:/home/gelliott/LitmusGPU/liblitmus# numactl -m 0 -N 0 perf stat -B -e cache-misses:u,cache-references:u ./uncache -u</div><div>Page Size: 4096</div><div>Running data access test.</div><div>Allocating 16 uncacheable pages.</div><div>Data allocated at 0x7f8d83197000.</div><div>Beginning tests...</div><div>Uncache Time: 1617009us</div><div><br></div><div> Performance counter stats for './uncache -u':</div><div><br></div><div> 10,000,182 cache-misses:u # 99.971 % of all cache refs </div><div> 10,003,093 cache-references:u </div><div><br></div><div> 1.623216983 seconds time elapsed</div></blockquote><div><br></div><div>Same code, with regular cacheable memory. I believe that the total number of cache references is low because application data is "cached" in registers, or perf doesn't profile the L1.</div><blockquote type="cite"><div>root@bonham:/home/gelliott/LitmusGPU/liblitmus# numactl -m 0 -N 0 perf stat -B -e cache-misses:u,cache-references:u ./uncache -c</div><div>Page Size: 4096</div><div>Running data access test.</div><div>Allocating 16 cacheable pages.</div><div>Data allocated at 0x7f8e38ac3000.</div><div>Beginning tests...</div><div>Cache Time: 15040us</div><div><br></div><div> Performance counter stats for './uncache -c':</div><div><br></div><div> 14 cache-misses:u # 0.445 % of all cache refs </div><div> 3,148 cache-references:u </div><div><br></div><div> 0.016859298 seconds time elapsed</div></blockquote></div><div><br></div><div><br></div><div>And the output comparing runtimes (optimized):</div><div><div></div><blockquote type="cite"><div>Page Size: 4096</div><div>Timing cached pages...</div><div>Running data access test.</div><div>Allocating 16 cacheable pages.</div><div>Data allocated at 0x7f53c292e000.</div><div>Beginning tests...</div><div>Cache Time: 15030us</div><div>Timing uncached pages...</div><div>Running data access test.</div><div>Allocating 16 uncacheable pages.</div><div>Data allocated at 0x7f53c292e000.</div><div>Beginning tests...</div><div>Uncache Time: 1616430us</div><div>Uncached/Cached Ratio: <b>107.546906</b></div></blockquote></div><div><br></div><div>Thanks to Chris for helping me with early issues on this!</div><div><br></div></body></html>