[LITMUS^RT] [PATCH] EXPLANATION: Architecture dependent uncachable control page.

[Manohar Vanga]

Hey Chris!

This seems like an interesting issue and I'm going to try reproducing it tonight or tomorrow. I will let you know if I succeed :-)

> Kernel system call S():
>    1) Read C->val and print it using TRACE().
>    2) C->val := 12345
> 
> User code:
>    1) C->val := 98765
>    2) S()
>    3) printf(C->val)
> 
> ...
> 
> On ARM (tested on an ODROID-X / Samsung Exynos4412), the result is
> non-deterministic. Usually, each address space prints only the value that it
> wrote (12345 in the kernel and 98765 in the userland), but adding sleep()
> between steps 1 and 2 of the userland code results in the kernel occasionally
> printing the "correct" value.

So it was printing 12345 in the kernel before you assigned it?? Or did you just write the sequence of steps in the system call backwards by mistake?

> I explored a few solutions to the problem. One was the restrict the task to run
> on a single CPU. That did not work. I also tried inserting general memory
> barriers in various places, but this also had no effect.

I would have assumed it was some sort of issue with the write buffer not being flushed out. It can apparently be forced with the 'dsb' instruction. However, looking at the source code, it seems like mb() falls down to a dsb instruction followed by a sync in the case that CONFIG_SMP has been set. That makes this issue less probable :P

If this had been the issue, it would actually explain the problem as cacheable memory seems to be buffered while using pgprot_noncached() seems to set it to non-buffered. This can be seen in arch/arm/include/asm/pgtable.h. In fact from what I see you can also use DMA coherent memory (using dma_alloc_coherent()) for allocating your control page. This might also explain why adding a sleep was making it work correctly; the write buffer was being flushed in that time. However, none of this applies as you say you already tried to use memory barriers (which flush the write buffer when CONFIG_SMP is set, according to arch/arm/include/asm/system.h), and it still didn't work :P

> What did work is to make the mapping of the control page uncachable in both the
> user- and kernelspace. One theory that could explain why this fixes the problem
> is due to caches indexed or tagged with virtual addresses. Shared memory
> accessed via different virtual addresses may need to be made uncached, because
> the cache would contain different entries for each virtual address range. If
> any of these mappings are writable, it causes a coherency problem because
> modifications made through one mapping aren't visible through the cache entries
> for other mappings. However, the documentation I found for the Cortex-A9 says
> that the data cache is Physically Index and Physically Tagged (PIPT), and
> that the instruction cache is Virtually Indexed and Physically Tagged
> (VIPT). Since the control page is data, I thought it should be using the
> PIPT cache, and the "aliasing" of virtual addresses should not be an issue.

So could it be that the cache is not PIPT (even though the documentation clearly states it :D)? Perhaps we could try flushing the data cache before returning from the syscall and check? (I believe it can be done with the flush_dcache_page() function).

Keep me updated! I'll let you know about my results once I set things up :-)

Thanks and best regards,
/manohar