nontemporal_store: make sure that the intrinsic is truly just a hint

[RalfJung]

The !nontemporal flag for stores in LLVM sounds like it is just a hint, but actually, it is not -- at least on x86, non-temporal stores need very special treatment by the programmer or else the Rust memory model breaks down. LLVM still treats these stores as-if they were normal stores for optimizations, which is highly dubious. Let's avoid all that dubiousness by making our own non-temporal stores be truly just a hint, which is possible on some targets (e.g. ARM). On all other targets, non-temporal stores become regular stores.

Blocked on rust-lang/stdarch#1541 propagating to the rustc repo, to make sure the _mm_stream intrinsics are unaffected by this change.

Fixes #114582
Cc @Amanieu @workingjubilee

[rustbot]

r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

[Kobzol]

It's a bit sad if this becomes an actual no-op on x86, nontemporal stores can be (very rarely) very useful. Is this usable through anything else than mm_stream_* and friends? These functions already contain a safety note that a fence has to be used after them.

[workingjubilee]

It's a bit sad if this becomes an actual no-op on x86, nontemporal stores can be (very rarely) very useful. Is this usable through anything else than mm_stream_* and friends? These functions already contain a safety note that a fence has to be used after them.

Those intrinsic functions are, in the latest commits to std::arch, internally assembly. They should behave identically to how they did before, please file a bug if they emit notably worse codegen. The distinction is that, in general, LLVM does not try to perform store deduplication into assembly code.

[Kobzol]

So mm_stream will still emit nontemporal instructions on x86? Thanks for clarifying!

[RalfJung]

Yes it will, that's why rust-lang/stdarch#1541 needs to propagate first.

[Kobzol]

Ok, great, then I misunderstood this change, sorry!

[workingjubilee]

Thank you, @RalfJung.

[RalfJung]

rust-lang/stdarch#1541 has propagated, so this should be good to go. We haven't heard from @fmease though so let's re-roll the reviewer dice.

r? compiler

[jieyouxu]

Thanks! The changes themselves look good based on the prior discussions. Just a question about the codegen test with regards to target coverage.

[jieyouxu]

Question [REV1 (2/2)]: If I understand it correctly, the codegen test is modified such that it only codegens for aarch64-unknown-linux-gnu target, but here we have 4 "well-behaved" architectures that we do emit !nontemporal for.

[jieyouxu]

r? jieyouxu

[jieyouxu]

Thanks, feel free to r=me,Amanieu,Jubilee after maybe fixing the nit :3

[RalfJung]

@bors r=jieyouxu,Amanieu,Jubilee

[bors]

📌 Commit 5be6c7d has been approved by jieyouxu,Amanieu,Jubilee

It is now in the queue for this repository.

[RalfJung]

@bors r=jieyouxu,Amanieu,Jubilee

[bors]

📌 Commit 75743dc has been approved by jieyouxu,Amanieu,Jubilee

It is now in the queue for this repository.