[idea] Allow .rmeta to be translated in .rlib files

[bjorn3]

We could then teach cargo to make .rmeta files and concurrently convert them to .rlib files. If you only need .rmeta files for dependencies cargo could convert the .rmeta files when some cores are idle. This would be more efficient, as a crate can be compiled before llvm is run for its dependencies.

[hanna-kruppe]

Interesting idea, though it has significant overlap with MIR-only rlibs (#38913) which would achieve the same effect (by not involving LLVM in the creation of rlibs) without putting more complexity in cargo. There are also some shared technical challenges between this and MIR-only rlibs: You have to be able to actually generate all the machine code from the metadata, which requires some compiler refactorings around statics.

[retep998]

So ideally we would have a three phased compilation. In the first phase cargo would get each crate to emit metadata so all the metadata is available as rapidly as possible. In the second phase, which would happen in parallel to the first phase, cargo would them get each crate to emit MIR based off the metadata. In the final phase we would rely on the future ideal of LLVM's ThinLTO, which would be fully parallel and incremental code generation on par with link.exe's incremental LTCG.

[hanna-kruppe]

emit MIR based off the metadata

That doesn't make sense to me. MIR is an essential part of metadata. AFAIK metadata currently doesn't contain anything even close to the full HIR, let alone earlier representations, so we'd have to start doing that, which invites all sorts of headaches besides pointlessly increasing the amount of data encoded and stored on disk. Furthermore, in the near future MIR is necessary for borrow checking and other analyses that can impact whether code should even compile.

[retep998]

@rkruppe If the only thing differentiating .meta and .rlib is the presence of machine code generated by LLVM, then wouldn't this issue become invalid as a result of MIR only rlibs? I'd much rather have MIR only rlibs combined with fully incremental code generation, which would work with existing build systems. Having a separate .rmeta -> .rlib phase would mean build systems would have to be updated to take advantage of that, which is more complexity for no gain over MIR only rlibs.

[hanna-kruppe]

It's true that the ideal MIR only rlibs workflow would fully obsolete this idea. Since that probably requires ThinLTO and (more robust) incremental, the rmeta -> rlib transformation could conceivably be implemented sooner. However, I agree that the impact on build systems is a big downside, and I don't think rolling out a sub-optimal solution that affects build systems is worthwhile when a more general solution is just a little further off.

[ishitatsuyuki]

I just came up with the exactly same idea. How much impact on build systems is estimated? I'm not sure if there's a build system that handles Rust build without calling Cargo.

Also, this idea is quite useful in the current model. Currently you need to run the compiler separately for check and build; the duplication is obsoleted with this model.

[hanna-kruppe]

Using build systems other than/in addition to Cargo is explicitly a goal in the 2018 roadmap. Even if it weren't, changing Cargo and rolling out the change is still a big step. And since the previous comments on this issue, we've gained ThinLTO and an actual prototype implementation of MIR-only rlibs -- and although this revealed that internal parallelization of rustc is another blocker for good results, that's already actively being worked on.

[ishitatsuyuki]

MIR only rlibs are not a silver bullet; separating compile stages still has its own benefits.

MIR only rlibs only applies to rlibs, and that's all. For instance, it doesn't benefit rustc much because most of the huge components are dylib.

Also, breaking the artifacts up with stages allows cargo check to be a part of compilation, speeding up the development experience.

At last, to incrementally migrate, I have an idea: we can take variable type of inputs (source, metadata, MIR, object), and we run fallback to transform serially if the caller doesn't support multiple pass compilation. This way, we can incrementally implement this as feature additions.

[hanna-kruppe]

Whatever the merits of MIR-only rlibs (I think your points underestimate or misunderstand them), what's been described in the OP is subsumed by MIR-only rlibs.

At last, to incrementally migrate, I have an idea: we can take variable type of inputs (source, metadata, MIR, object), and we run fallback to transform serially if the caller doesn't support multiple pass compilation. This way, we can incrementally implement this as feature additions.

That still requires build systems, including Cargo, to be updated to get any benefit at all. I don't think anyone proposed to remove the way to go "source files -> object files" in one rustc invocation, but a new way to drive the compilation still has to be implemented and used and can still become redundant. It's not a matter of whether we can implement this without breaking rustc's CLI interface, it's how much churn is needed to actually bring tangible benefits to users (and how large those benefits are) compared to other solutions.

[bjorn3]

With build pipelining and -Zno-link/-Zlink-only I think this doesn't provide much benefit anymore.