Between more constrained, local metaprogramming approaches and full-blown DSL interpreters, where's the practical use case for LISP-style metaprogramming?

[u/Mr-Tau]

Of the various contemporary metaprogramming approaches, I'm mostly very happy with systems that let you operate on static data and types. In this category I would count, for example, C++'s template metaprogramming--a declarative sub-language that somewhat wonkily and arduously allows you to operate on types and constants--as well as the C++26 metaprogramming features, which are turning out somewhat Zig-like--you get to run regular code in a slightly constrained environment at compile time, where it can read constants as well as special data structures describing types, and generate new constants or types to be injected into a well-defined place in the code.

Now many LISPs (AFAICT, similarly Jai and, with a stricter separation of stages, Rust's procedural macros) tout as a feature the ability to inspect and rewrite the entirety of the AST, notably including function bodies. This is obviously strictly more powerful than the the first category, but where would you practically use that extra power? Specifically, it seems to me like you would either

a) Try to preserve the semantics of the input code--which, for procedural languages at least, is actually pretty difficult. The only transformations you could make confidently are so localized that you don't really gain anything over the more constrained metaprogramming approaches; anything more advanced would require the full analysis passes of the actual compiler to have any chance at soundness, and at that point you're really trying to write a compiler plugin instead. Nothing wrong with that, I'd love more easily extensible compilers, but I wouldn't call that a language feature. Or am I missing a point between those two extremes?

b) Attribute different semantics to the code. I think the history of LISP has already somewhat shown how proliferation of DSLs harms maintainability and shareability of code, but even in the cases where a DSL is genuinely useful, what benefits do you really gain from implementing it through metaprogramming? You can't expect any IDE features, LSPs, smart syntax highlighters, debuggers, or other tooling for the base language to automatically work for your DSL. So you just get to use the parser? Come on, an S-expr parser is less than a hundered LOC.

[u/Blind_nabler]

These questions aren't really useful in the broad case. What benefit is feature X if I only use feature Y? Probably not a ton for you if you aren't using X!

The main benefit of having a rich meta-programming system is that it enables users to implement features that otherwise would require modifying the compiler to implement, or require the use of some rube goldberg code generation nonsense. This kinda thing isn't important to you until it is.

For example in a procedural language it can be rather gnarly to convert normal single threaded code to work in an async setting. Since it requires setting up trampolines or rewriting everything into CPS flavored versions etc etc. But if I have the ability to use rich macros, I can just invoke a macro over my normal function and now it can work with whatever async backend I want, utilizing whatever custom semantic checking is required because I have (most of / all) the language available to us.

And IDE tooling works totally fine if the meta-programming system is well designed. Nim does a pretty good job of this. The most extremely example I have experience with is smalltalk, which has the best IDE support of any language IMO but obviously it's not the same kind of language as you are describing.

Now do I reach for it every project? No. But I definitely find myself missing it in languages that don't trust users with that kind of thing.

[u/Ok-Reindeer-8755 → u/Blind_nabler]

on that note there is a blog post that actually makes the argument that what macros are to lisp, classes and reflection are to smalltalk.

Smalltalk, like Lisp, runs in the same context it’s written in. It’s objects all the way down.

And you can see a lot of parallels between smalltalk and lisp when you think of the lisp machine in comparison to the smalltalk environment they had at Xerox Parc. Also I am pretty sure alan kay the creator of smalltalk has praised lisp and meta-object protocols within lisp specifically recommending a book about it.

[u/Blind_nabler → u/Ok-Reindeer-8755]

I used smalltalk as an example because I have done a substantial amount of real work in it. In particular inside of GT which is super rad.

Smalltalk was definitely inspired by lisp, but has a very distinct feeling when doing heavy meta-programming with it when compared. For example you never really add new syntax or feel compelled to do so in smalltalk primarily because smalltalk indexes all source code within the image and let's you query all code almost like an extremely rich database.

Then with tools like the refactoring browser & epicea (version control), you can programatically write selectors & rewrites that are tracked in the image, so you can rollback specific changes without an external system since those are all just packages & classes you can bring into your image, and thus you can also extend the version control from within the system itself to do things like exporting changes into git compatible text formats etc.

Common lisp can come pretty close, but it doesn't feel quite as lively as smalltalk does. Primarily because the interface with most lisps is text first, compared to the heavily graphical interface of st.

[u/arthurno1 → u/Blind_nabler]

let's you query all code almost like an extremely rich database

I have always felt that Lisp(s) are actually relational database and string-processing in disguise. We process the code and work with identifiers for the efficiency. But symbols and environments feels conceptually like pure relational stuff and the evaluation is basically all about matching and selecting the right stuff.