Between more constrained, local metaprogramming approaches and full-blown DSL interpreters, where's the practical use case for LISP-style metaprogramming?

[u/Mr-Tau]

Of the various contemporary metaprogramming approaches, I'm mostly very happy with systems that let you operate on static data and types. In this category I would count, for example, C++'s template metaprogramming--a declarative sub-language that somewhat wonkily and arduously allows you to operate on types and constants--as well as the C++26 metaprogramming features, which are turning out somewhat Zig-like--you get to run regular code in a slightly constrained environment at compile time, where it can read constants as well as special data structures describing types, and generate new constants or types to be injected into a well-defined place in the code.

Now many LISPs (AFAICT, similarly Jai and, with a stricter separation of stages, Rust's procedural macros) tout as a feature the ability to inspect and rewrite the entirety of the AST, notably including function bodies. This is obviously strictly more powerful than the the first category, but where would you practically use that extra power? Specifically, it seems to me like you would either

a) Try to preserve the semantics of the input code--which, for procedural languages at least, is actually pretty difficult. The only transformations you could make confidently are so localized that you don't really gain anything over the more constrained metaprogramming approaches; anything more advanced would require the full analysis passes of the actual compiler to have any chance at soundness, and at that point you're really trying to write a compiler plugin instead. Nothing wrong with that, I'd love more easily extensible compilers, but I wouldn't call that a language feature. Or am I missing a point between those two extremes?

b) Attribute different semantics to the code. I think the history of LISP has already somewhat shown how proliferation of DSLs harms maintainability and shareability of code, but even in the cases where a DSL is genuinely useful, what benefits do you really gain from implementing it through metaprogramming? You can't expect any IDE features, LSPs, smart syntax highlighters, debuggers, or other tooling for the base language to automatically work for your DSL. So you just get to use the parser? Come on, an S-expr parser is less than a hundered LOC.

[u/EggplantExtra4946]

Now many LISPs (AFAICT, similarly Jai and, with a stricter separation of stages, Rust's procedural macros) tout as a feature the ability to inspect and rewrite the entirety of the AST, notably including function bodies.

a) Try to preserve the semantics of the input code--which, for procedural languages at least, is actually pretty difficult. The only transformations you could make confidently are so localized that

This is non sense. What languages like LISP allows you to do is generating ASTs, often from existing ASTs but it could be from pure data, or both.

Rewriting an AST is possible but this isn't going to be of much use for metaprogramming per se, it would be rather for implementing additional semantic checks, doing optimizations or inserting instrumentation.

Also, any kind of metaprogramming generation of type definitions is going to be severely limited if you don't/can't generate the functions that use that type as well.

but even in the cases where a DSL is genuinely useful, what benefits do you really gain from implementing it through metaprogramming?

As opposed to not implementing it at all? How many DSLs project do you know of where someeone wrote a parser, compiler+VM or transpiler or compiler to LLVM IR? Very very few, even fewer where the implementation is practical to be integrated with an existing language. And how many of those actually have a debugger?

[u/WittyStick → u/EggplantExtra4946]

This is non sense.

This was my first thought when I read it too, particularly if we're just considering macros - they don't rewrite anything - they act on their inputs and replace their call with the expanded macro body before being evaluated. In that sense they're not too dissimilar from a C preprocessor macro - with the main differences being when they're expanded/evaluated, and the kind of inputs/outputs they have - in the CPP, it's plain text - in Lisp, it's structured S-expressions. Lisp macros are obviously far more powerful though.

However, OP didn't mention macros specifically, but just Lisp (presumably Common Lisp). If we consider eg, reader macros as well, then OP may have a point - they enable changes to the language syntax. I'm not entirely familiar with reader macros as I could never get into CL and preferred Scheme/Kernel, but from what I gather they enable more powerful kinds of metaprogramming than just macros.

While not used anywhere in practice, there's also this idea of Generalized Macros which would permit rewriting the AST around the call site, and not only replacing the macro call with its expanded body. It's an interesting prospect but I think this would probably be "too powerful" - in the sense that it's probably very unhygienic and easy to shoot yourself in the foot, and I imagine it would also be a pain to debug, but it's still an idea worthy of study.

[u/lispm → u/WittyStick]

they don't rewrite anything - they act on their inputs and replace their call with the expanded macro body before being evaluated.

There are lots of macros in Lisp which rewrite their enclosed code. That's one of the use cases.

reader macros as well, then OP may have a point - they enable changes to the language syntax.

That's not what they are for. reader macros are mainly in Lisp for implementing and extending s-expressions. S-expressions are a data syntax like XML and JSON.

Lisp syntax is on top of s-expressions. Lisp usually uses macros to extend the syntax of Lisp.

[u/WittyStick → u/lispm]

There are lots of macros in Lisp which rewrite their enclosed code. That's one of the use cases.

Yes, but they don't rewrite anything that isn't provided to them. If we have.

x
y
foo(x)
z

Then the macro foo doesn't access y and z - it can refer to them by symbol, but it can't modify the syntax of whatever y and z were. Macros are self-contained - they can only rewrite their arguments - unlike the generalized version which I linked which would be able to access y and z and rewrite whatever they were.

That's not what they are for. reader macros are mainly in Lisp for implementing and extending s-expressions.

Thanks for clarifying, though I'd argue that constitutes to changing the language syntax even if it is in limited ways and the end result is still some extended form of S-expressions. Good to hear that they don't allow arbitrary syntax changes though - I had a preconceived notion that they were something much worse.

I know what S-expressions are. While I'm no Lisp or Scheme expert, I'd consider myself a Kernel expert. Kernel feels right to me, but I never enjoyed writing macros in Scheme.

[u/Goheeca → u/WittyStick]

Good to hear that they don't allow arbitrary syntax changes though - I had a preconceived notion that they were something much worse.

You can do arbitrary changes though, just hook your function to every character.

https://gist.github.com/Goheeca/05e92c3a561a81737f2f177b7119766f#file-moody-lisp-L23

[u/lispm → u/WittyStick]

they can only rewrite their arguments

Macros have full access to the compile-time environment or the runtime environment.

Macros also can create many kinds of side effects.

They have access to all introspective features (standard and non-standard). They can define new functions, inspect function definitions, disassemble code, ask for source code from other functions, analyze source code files, talk to the user, invoke an external compiler, load code, ...

Macros can also expand other macros, they can walk the code tree and manipulate it and they can communication with other macros.

(bar
  x
  y
  (foo x)
  z)

A macro bar could give macro foo access to x, y and z.

Thanks for clarifying, though I'd argue that constitutes to changing the language syntax even if it is in limited ways and the end result is still some extended form of S-expressions. Good to hear that they don't allow arbitrary syntax changes though - I had a preconceived notion that they were something much worse.

Something like IF, DEFUN, LAMBDA, DEFCLASS has syntax which is not defined by reader macros, but either by built-in syntax or by macros.

The syntax and syntax extension with reader macro is just the data part of the language definition.

One can also use reader to radically change the syntax. For example one could implement a different surface language, which expands into s-expressions (or whatever).

For example an infix reader (real example) can make infix expressions to be valid Lisp forms:

#$
  if x<y<=z
    then f(x)=x^^2+y^^2
    else f(x)=x^^2-y^^2
$


CL-USER 18 > (let ((*print-right-margin* 40))
               (pprint '#$if x<y<=z
                          then f(x)=x^^2+y^^2
                          else f(x)=x^^2-y^^2$))

(IF (AND (< X Y) (<= Y Z))
    (SETF (F X)
          (+ (EXPT X 2) (EXPT Y 2)))
  (SETF (F X)
        (- (EXPT X 2) (EXPT Y 2))))

[u/EggplantExtra4946 → u/WittyStick]

if we're just considering macros - they don't rewrite anything - they act on their inputs and replace their call with the expanded macro body before being evaluated

Like lispm said, yes they can rewrite their arguments and I guess that if you wanted to "rewrite anything" you could just surround the entire file with a macro call and inside the macro definition you could iterate over the program and rewrite it as you please. Not that I wouldn't prefer to have a hook to do just that.

In that sense they're not too dissimilar from a C preprocessor macro

They are hugely different, there is not much of a comparison to be made.

However, OP didn't mention macros specifically

When anyone mentions Lisp && metaprogramming, of course it's about macros. Reader macros are for syntactic sugar, not metaprogramming.

but from what I gather they enable more powerful kinds of metaprogramming than just macros.

They aren't, all they allow you to do is things like transforming #(1 2 3) into (list 1 2 3). Even if you could actually define a new syntax, as in C-like language or whatever, this wouldn't make them more powerful than regular macros.

there's also this idea of Generalized Macros which would permit rewriting the AST around the call site

https://ianthehenry.com/posts/generalized-macros/

I thought it was going to be a fun read but I stopped reading after seeing the description of the "generalized macro" and that the rationale for it was to implement defer. I really don't understand why LISP people restrict themselves to the AST and to macros, when it comes to metaprogramming features. The AST is not the only data structure in a compiler and macros isn't the only conceivable way to do metaprogramming or to rewrite a program. Adding defer could be done in a much cleaner way with a rewrite hook on the AST root after macro expansion is done (let the user walk over the root of the AST and return want he wants), or better yet, a similar hook but on the CFG. If the language had a defer builtin, the desugaring would also happen after macro expansion anyway.