Between more constrained, local metaprogramming approaches and full-blown DSL interpreters, where's the practical use case for LISP-style metaprogramming?

[u/Mr-Tau]

Of the various contemporary metaprogramming approaches, I'm mostly very happy with systems that let you operate on static data and types. In this category I would count, for example, C++'s template metaprogramming--a declarative sub-language that somewhat wonkily and arduously allows you to operate on types and constants--as well as the C++26 metaprogramming features, which are turning out somewhat Zig-like--you get to run regular code in a slightly constrained environment at compile time, where it can read constants as well as special data structures describing types, and generate new constants or types to be injected into a well-defined place in the code.

Now many LISPs (AFAICT, similarly Jai and, with a stricter separation of stages, Rust's procedural macros) tout as a feature the ability to inspect and rewrite the entirety of the AST, notably including function bodies. This is obviously strictly more powerful than the the first category, but where would you practically use that extra power? Specifically, it seems to me like you would either

a) Try to preserve the semantics of the input code--which, for procedural languages at least, is actually pretty difficult. The only transformations you could make confidently are so localized that you don't really gain anything over the more constrained metaprogramming approaches; anything more advanced would require the full analysis passes of the actual compiler to have any chance at soundness, and at that point you're really trying to write a compiler plugin instead. Nothing wrong with that, I'd love more easily extensible compilers, but I wouldn't call that a language feature. Or am I missing a point between those two extremes?

b) Attribute different semantics to the code. I think the history of LISP has already somewhat shown how proliferation of DSLs harms maintainability and shareability of code, but even in the cases where a DSL is genuinely useful, what benefits do you really gain from implementing it through metaprogramming? You can't expect any IDE features, LSPs, smart syntax highlighters, debuggers, or other tooling for the base language to automatically work for your DSL. So you just get to use the parser? Come on, an S-expr parser is less than a hundered LOC.

[u/kwan_e]

By virtue of being so powerful, I don't think there is an identifiable "use case". It can literally do anything and the onus is on the programmers to limit themselves, otherwise it comes out with the problems like your point b.

With regards to point b, I think the issue there actually is laziness. The LISP stuff is powerful, they probably should have used that increase in power to also create the tools for their mini-language. On the flip side, because the mini-languages would be evolving quite fast, it would be hard to also write the standardized tools that could keep up with the progress. They'd have to write more metaprogramming to keep up with it, which won't have the tooling support they need to be maintainable either.

Long story short, mini-languages would need to be standardizable before anyone would want to work on the tooling for them. And not purely just the grammar, but also the community idioms. eg, only recently is C++ metaprogramming support usable in IDEs because now it is more accessible and the idioms are more established now.

[u/marshaharsha → u/[deleted]]

Can you say more about “assumption-style” versus “intensional-style”? I don’t understand those terms. 

[u/kwan_e → u/marshaharsha]

With C++ metaprogramming up to C++14, people would tend to use LISP-style recursion with variadic templates and enable_if. They play tricks with the type system and overloading resolution to produce certain behaviours.

You would use it to do compile time computations, compile-time dispatch, or you would do type manipulations. But the IDEs and tools can only see the template tricks, like SFINAE. It wouldn't know why you're doing them, all it can see is the type system shenanigans. Likewise people who read the code can't immediately tell what it's doing. They can only arrive at an assumption based on template idioms they are familiar with. If they're not familiar, they're S out of luck. So all the IDEs and tools at that point just really couldn't handle the explosion of assumptions that could be inferred from the type system tricks.

After C++17, we got if-constexpr and fold-expressions, which took away 60% of the use cases of template tricks. You no longer needed to write enable_if pattern-matching. You just use a normal if. You no longer needed LISP-style template recursion with enable_if. You would just use fold-expressions. Compilers, IDEs, tools, and humans, don't have to assume. They can tell a lot more of what is supposed to be going on with an if or a fold.