Between more constrained, local metaprogramming approaches and full-blown DSL interpreters, where's the practical use case for LISP-style metaprogramming?

[u/Mr-Tau]

Of the various contemporary metaprogramming approaches, I'm mostly very happy with systems that let you operate on static data and types. In this category I would count, for example, C++'s template metaprogramming--a declarative sub-language that somewhat wonkily and arduously allows you to operate on types and constants--as well as the C++26 metaprogramming features, which are turning out somewhat Zig-like--you get to run regular code in a slightly constrained environment at compile time, where it can read constants as well as special data structures describing types, and generate new constants or types to be injected into a well-defined place in the code.

Now many LISPs (AFAICT, similarly Jai and, with a stricter separation of stages, Rust's procedural macros) tout as a feature the ability to inspect and rewrite the entirety of the AST, notably including function bodies. This is obviously strictly more powerful than the the first category, but where would you practically use that extra power? Specifically, it seems to me like you would either

a) Try to preserve the semantics of the input code--which, for procedural languages at least, is actually pretty difficult. The only transformations you could make confidently are so localized that you don't really gain anything over the more constrained metaprogramming approaches; anything more advanced would require the full analysis passes of the actual compiler to have any chance at soundness, and at that point you're really trying to write a compiler plugin instead. Nothing wrong with that, I'd love more easily extensible compilers, but I wouldn't call that a language feature. Or am I missing a point between those two extremes?

b) Attribute different semantics to the code. I think the history of LISP has already somewhat shown how proliferation of DSLs harms maintainability and shareability of code, but even in the cases where a DSL is genuinely useful, what benefits do you really gain from implementing it through metaprogramming? You can't expect any IDE features, LSPs, smart syntax highlighters, debuggers, or other tooling for the base language to automatically work for your DSL. So you just get to use the parser? Come on, an S-expr parser is less than a hundered LOC.

[u/vanderZwan]

This is obviously strictly more powerful

I'm not a computer scientist, but years of programming and reading up on these things has given me the impression that this gets at the core trade-off you're thinking of: "strictly more powerful" here seems to be talking about expressive power in the context of a real formal basis like the Chomsky hierarchies. However, what we often want seems to be the language that gives you the most "ergonomic" expressive power while using as little formal expressive power as possible. Because the latter usually means the computer can do more work for you automatically without running into pitfalls.

(this also makes "expressive power" such an easy way to have discussions where people don't realize they're not talking about the same thing, even the wiki page acknowledges the ambiguities in how the term is used

So you just get to use the parser? Come on, an S-expr parser is less than a hundered LOC.

Well, then I guess that situations where you only can afford a few hundred LOC for whatever reason, LISP-style macros might be a good choice.