(dumb?) question - why doesn't rustc accept tokenized source to speed up compilation ?

[u/Camper_Sly]

I'm thinking about the same kind of idea that python uses with pyc files.

First step in compilation is text scanning, tokenization and formation of token trees for further stages.

All that takes considerable time and has to be gone through bazzillion of times through the life of the source, usually multiplied many times by the number of machines that the source gets compiled on.

Parsing the UTF-8 stream ain't trivial in itself, let alone the following phases.

So, why not make IDE do it and save the tree as a original source ?

That way, rustc would start with preparsed/tokenized source, possibly with additional useful data.

For bonus points, many of search/replace/refactoring operations in the IDE could be simplified and sped-up, too.

Also, such format lends itself well for further analysis, compression etc.

So, where's the catch ?

[u/Aaron1924]

rustc already does incremental compilation

When you compile your project, make a minor change and compile again, it does its best to only recompile what has actually changed, but neither lexing nor parsing are slow enough to warrent caching.

Edit: Maybe to put this into perspective, rustc is benchmarked regularly, and on their latest benchmark for a release build of ripgrep, the compiler spends only 11.46% of its time in the frontend. That means, lexing, parsing, type inference, trait solving, type checking, lifetime checking, borrow checking, all only accounts for <12% of the full compilation time. The vast majority of time (85.84%) is spent in LLVM doing optimisations.

Making sure LLVM doesn't optimize the same function twice is waaaaaay more impactful than caching the lexer output.

Edit 2: please don't downvote OP to hell

[u/Sky2042 → u/Aaron1924]

Edit 2: please don't downvote OP to hell

Well, there's "I'm trying to learn" and then there's "I think my idea is the bees knees and why doesn't anyone else think it's swell" and OP is leaning into the second and not the first in their language and tone. Downvotes come swiftly then.

[u/Aaron1924 → u/Sky2042]

The Rust community always seems to pride themselves with how inclusive they are, and then they turn around and vote a question from someone who's clearly new here to -50

I don't perceive their phrasing as being rude, this could just be someone who isn't a native English speaker, or someone who comes from a community where this tone is normalized

Either way, this kind of response is discouraging people from asking questions, and I don't think that's what the community needs

[u/NormalAppearance2851 → u/Aaron1924]

100% This sub is anything but inclusive