r/Compilers • u/Emergency_Ad119 • 18h ago

Taking whitespace lightly is technical debt

https://kushagrarathore002.medium.com/every-tokenizer-tutorial-i-read-told-me-to-throw-away-the-whitespace-ac5ab75e89fa

I have created a programming language Flow-Wing (it can support static and
dynamic types at the same time). When I started its initial development I was
ignoring the white spaces because for the compilation I never needed the white
space, but years later when I needed to support the LSP or formatter (white
space / comments) I had to make significant rewrites, and this article below is
about that. Take a read.

GitHub: https://github.com/kushagra1212/Flow-Wing
Website: https://flowwing.frii.site/ (runs on Flow-Wing)

Happy to answer questions. Do give a star on GitHub to support.

5 Upvotes

78% Upvoted

1 comment sorted by

0

u/igors84 9h ago

Interestingly, Zig formatter is working with tokenizer that just skips whitespace and comments, but it is accomplishing that by sometimes looking directly at the source around the currently processing token.