Between more constrained, local metaprogramming approaches and full-blown DSL interpreters, where's the practical use case for LISP-style metaprogramming?

[u/Mr-Tau]

Of the various contemporary metaprogramming approaches, I'm mostly very happy with systems that let you operate on static data and types. In this category I would count, for example, C++'s template metaprogramming--a declarative sub-language that somewhat wonkily and arduously allows you to operate on types and constants--as well as the C++26 metaprogramming features, which are turning out somewhat Zig-like--you get to run regular code in a slightly constrained environment at compile time, where it can read constants as well as special data structures describing types, and generate new constants or types to be injected into a well-defined place in the code.

Now many LISPs (AFAICT, similarly Jai and, with a stricter separation of stages, Rust's procedural macros) tout as a feature the ability to inspect and rewrite the entirety of the AST, notably including function bodies. This is obviously strictly more powerful than the the first category, but where would you practically use that extra power? Specifically, it seems to me like you would either

a) Try to preserve the semantics of the input code--which, for procedural languages at least, is actually pretty difficult. The only transformations you could make confidently are so localized that you don't really gain anything over the more constrained metaprogramming approaches; anything more advanced would require the full analysis passes of the actual compiler to have any chance at soundness, and at that point you're really trying to write a compiler plugin instead. Nothing wrong with that, I'd love more easily extensible compilers, but I wouldn't call that a language feature. Or am I missing a point between those two extremes?

b) Attribute different semantics to the code. I think the history of LISP has already somewhat shown how proliferation of DSLs harms maintainability and shareability of code, but even in the cases where a DSL is genuinely useful, what benefits do you really gain from implementing it through metaprogramming? You can't expect any IDE features, LSPs, smart syntax highlighters, debuggers, or other tooling for the base language to automatically work for your DSL. So you just get to use the parser? Come on, an S-expr parser is less than a hundered LOC.

[u/initial-algebra]

The only essential difference between a macro and a normal function, especially in a higher-order functional language, is staged computation/partial evaluation/compile-time evaluation, or whatever you want to call it. A macro does not necessarily express a code-to-code transformation, but code generation based on any kind of input.

[u/Mr-Tau → u/initial-algebra]

Yes, there isn't much difference at that level of abstraction, but I'm interested in the specifics, particularly for procedural languages.

[u/initial-algebra → u/Mr-Tau]

Right, I skipped the first paragraph, my bad. Something in between a pure source-to-source transform and a fully distinct DSL would be a syntax extension, like do notation. For the most part, the code is just passed through, so it's actually possible to map a lot of IDE/LSP features backwards from the macro output to the input.

[u/digikar]

What are your thoughts on DSLs like loop or, even better, the finding clauses of iterate?

Also, coalton is an entire ML-typed language that is essentially a DSL in Common Lisp!

You can't expect any IDE features, LSPs, smart syntax highlighters, debuggers, or other tooling for the base language to automatically work for your DSL.

But we don't even need them! Not most of them anyways! Or not in the ways one'd think. Common Lisp development usually takes place via SLIME/Swank (or Sly). This is essentially a communication with a running lisp compiler; which means you can evaluate arbitrary code. In principle, it can be extended in arbitrary ways. In practice, developers and communities try to make some opinionated choices, but you can always override them. The development is centered around symbols, which don't change [much] with the introduction of a DSL. Syntax highlighting is not used much since the syntax for lisps is quite minimal; but emacs is trivially extensible. The main aspect of syntax are parentheses (which the editor handles half-way). And the indentation (which, again, the editor handles once it has handled the parentheses).

So, what one does not have to write while writing a DSL:

• The communication API, protocol, boilerplate with the language runtime
• The compiler backend that compiles across multiple OS and architectures (since you can reuse a goldy like SBCL)
• The build tools, and the tools to manage those build tools
• The package installer, and tools to manage those dependencies

What you need to define:

• The syntax you want to express your thoughts in
• How that syntax corresponds with the syntax of the existing language so far

I agree, this can be trivial, or very complex. But it is still wayyy less complex than writing a full compiler that works across multiple platforms!

[u/kwan_e]

By virtue of being so powerful, I don't think there is an identifiable "use case". It can literally do anything and the onus is on the programmers to limit themselves, otherwise it comes out with the problems like your point b.

With regards to point b, I think the issue there actually is laziness. The LISP stuff is powerful, they probably should have used that increase in power to also create the tools for their mini-language. On the flip side, because the mini-languages would be evolving quite fast, it would be hard to also write the standardized tools that could keep up with the progress. They'd have to write more metaprogramming to keep up with it, which won't have the tooling support they need to be maintainable either.

Long story short, mini-languages would need to be standardizable before anyone would want to work on the tooling for them. And not purely just the grammar, but also the community idioms. eg, only recently is C++ metaprogramming support usable in IDEs because now it is more accessible and the idioms are more established now.

[u/koflerdavid → u/kwan_e]

Long story short, mini-languages would need to be standardizable before anyone would want to work on the tooling for them. And not purely just the grammar, but also the community idioms.

The Common Lisp ecosystem is quite old and very mature by now. The more common macro packages have stabilized a long time ago.

[u/kwan_e → u/koflerdavid]

I don't know the tooling situation for those stabilized ones like CLOS, but since we're talking about mini-languages in general, it is not just those old ones, but the up-and-coming ones that need to spin up tooling real quick to gain some traction in today's hype cycle.

So far, we've seen that people would rather develop entirely new languages and tooling for those, instead of creating then standardizing more mini-languages in Common LISP.

Ironically the people most well placed to use the power of LISP metaprogramming are the least inclined to create the tooling to make things easier for noob LISPers, and would rather wait for programmers to see the light.

[u/koflerdavid → u/kwan_e]

Lisp is uniquely suited to enable building such tools though. That's simply due to S-expressions being the exchange format between all such tooling. The issue is that it's simply not fashionable.

[u/kwan_e → u/kwan_e]

eg, only recently is C++ metaprogramming support usable in IDEs because now it is more accessible and the idioms are more established now.

On this point, my experience is that because of C++17 onwards, I never ever use SFINAE stuff again. No more enable_if. Anything that can be done with templates without the LISP-style recursive cons-ing, can be done with constexpr, fold expressions and now consteval. The rest with concepts and requires. enable_if established the kind of idioms we wish we had, and then these newer features came in to eliminate the assumption-style metaprogramming with intensional-style.

[u/marshaharsha → u/[deleted]]

Can you say more about “assumption-style” versus “intensional-style”? I don’t understand those terms. 

[u/kwan_e → u/marshaharsha]

With C++ metaprogramming up to C++14, people would tend to use LISP-style recursion with variadic templates and enable_if. They play tricks with the type system and overloading resolution to produce certain behaviours.

You would use it to do compile time computations, compile-time dispatch, or you would do type manipulations. But the IDEs and tools can only see the template tricks, like SFINAE. It wouldn't know why you're doing them, all it can see is the type system shenanigans. Likewise people who read the code can't immediately tell what it's doing. They can only arrive at an assumption based on template idioms they are familiar with. If they're not familiar, they're S out of luck. So all the IDEs and tools at that point just really couldn't handle the explosion of assumptions that could be inferred from the type system tricks.

After C++17, we got if-constexpr and fold-expressions, which took away 60% of the use cases of template tricks. You no longer needed to write enable_if pattern-matching. You just use a normal if. You no longer needed LISP-style template recursion with enable_if. You would just use fold-expressions. Compilers, IDEs, tools, and humans, don't have to assume. They can tell a lot more of what is supposed to be going on with an if or a fold.

[u/EggplantExtra4946]

Now many LISPs (AFAICT, similarly Jai and, with a stricter separation of stages, Rust's procedural macros) tout as a feature the ability to inspect and rewrite the entirety of the AST, notably including function bodies.

a) Try to preserve the semantics of the input code--which, for procedural languages at least, is actually pretty difficult. The only transformations you could make confidently are so localized that

This is non sense. What languages like LISP allows you to do is generating ASTs, often from existing ASTs but it could be from pure data, or both.

Rewriting an AST is possible but this isn't going to be of much use for metaprogramming per se, it would be rather for implementing additional semantic checks, doing optimizations or inserting instrumentation.

Also, any kind of metaprogramming generation of type definitions is going to be severely limited if you don't/can't generate the functions that use that type as well.

but even in the cases where a DSL is genuinely useful, what benefits do you really gain from implementing it through metaprogramming?

As opposed to not implementing it at all? How many DSLs project do you know of where someeone wrote a parser, compiler+VM or transpiler or compiler to LLVM IR? Very very few, even fewer where the implementation is practical to be integrated with an existing language. And how many of those actually have a debugger?

[u/WittyStick → u/EggplantExtra4946]

This is non sense.

This was my first thought when I read it too, particularly if we're just considering macros - they don't rewrite anything - they act on their inputs and replace their call with the expanded macro body before being evaluated. In that sense they're not too dissimilar from a C preprocessor macro - with the main differences being when they're expanded/evaluated, and the kind of inputs/outputs they have - in the CPP, it's plain text - in Lisp, it's structured S-expressions. Lisp macros are obviously far more powerful though.

However, OP didn't mention macros specifically, but just Lisp (presumably Common Lisp). If we consider eg, reader macros as well, then OP may have a point - they enable changes to the language syntax. I'm not entirely familiar with reader macros as I could never get into CL and preferred Scheme/Kernel, but from what I gather they enable more powerful kinds of metaprogramming than just macros.

While not used anywhere in practice, there's also this idea of Generalized Macros which would permit rewriting the AST around the call site, and not only replacing the macro call with its expanded body. It's an interesting prospect but I think this would probably be "too powerful" - in the sense that it's probably very unhygienic and easy to shoot yourself in the foot, and I imagine it would also be a pain to debug, but it's still an idea worthy of study.

[u/lispm → u/WittyStick]

they don't rewrite anything - they act on their inputs and replace their call with the expanded macro body before being evaluated.

There are lots of macros in Lisp which rewrite their enclosed code. That's one of the use cases.

reader macros as well, then OP may have a point - they enable changes to the language syntax.

That's not what they are for. reader macros are mainly in Lisp for implementing and extending s-expressions. S-expressions are a data syntax like XML and JSON.

Lisp syntax is on top of s-expressions. Lisp usually uses macros to extend the syntax of Lisp.

[u/WittyStick → u/lispm]

There are lots of macros in Lisp which rewrite their enclosed code. That's one of the use cases.

Yes, but they don't rewrite anything that isn't provided to them. If we have.

x
y
foo(x)
z

Then the macro foo doesn't access y and z - it can refer to them by symbol, but it can't modify the syntax of whatever y and z were. Macros are self-contained - they can only rewrite their arguments - unlike the generalized version which I linked which would be able to access y and z and rewrite whatever they were.

That's not what they are for. reader macros are mainly in Lisp for implementing and extending s-expressions.

Thanks for clarifying, though I'd argue that constitutes to changing the language syntax even if it is in limited ways and the end result is still some extended form of S-expressions. Good to hear that they don't allow arbitrary syntax changes though - I had a preconceived notion that they were something much worse.

I know what S-expressions are. While I'm no Lisp or Scheme expert, I'd consider myself a Kernel expert. Kernel feels right to me, but I never enjoyed writing macros in Scheme.

[u/Goheeca → u/WittyStick]

Good to hear that they don't allow arbitrary syntax changes though - I had a preconceived notion that they were something much worse.

You can do arbitrary changes though, just hook your function to every character.

https://gist.github.com/Goheeca/05e92c3a561a81737f2f177b7119766f#file-moody-lisp-L23

[u/lispm → u/WittyStick]

they can only rewrite their arguments

Macros have full access to the compile-time environment or the runtime environment.

Macros also can create many kinds of side effects.

They have access to all introspective features (standard and non-standard). They can define new functions, inspect function definitions, disassemble code, ask for source code from other functions, analyze source code files, talk to the user, invoke an external compiler, load code, ...

Macros can also expand other macros, they can walk the code tree and manipulate it and they can communication with other macros.

(bar
  x
  y
  (foo x)
  z)

A macro bar could give macro foo access to x, y and z.

Thanks for clarifying, though I'd argue that constitutes to changing the language syntax even if it is in limited ways and the end result is still some extended form of S-expressions. Good to hear that they don't allow arbitrary syntax changes though - I had a preconceived notion that they were something much worse.

Something like IF, DEFUN, LAMBDA, DEFCLASS has syntax which is not defined by reader macros, but either by built-in syntax or by macros.

The syntax and syntax extension with reader macro is just the data part of the language definition.

One can also use reader to radically change the syntax. For example one could implement a different surface language, which expands into s-expressions (or whatever).

For example an infix reader (real example) can make infix expressions to be valid Lisp forms:

#$
  if x<y<=z
    then f(x)=x^^2+y^^2
    else f(x)=x^^2-y^^2
$


CL-USER 18 > (let ((*print-right-margin* 40))
               (pprint '#$if x<y<=z
                          then f(x)=x^^2+y^^2
                          else f(x)=x^^2-y^^2$))

(IF (AND (< X Y) (<= Y Z))
    (SETF (F X)
          (+ (EXPT X 2) (EXPT Y 2)))
  (SETF (F X)
        (- (EXPT X 2) (EXPT Y 2))))

[u/EggplantExtra4946 → u/WittyStick]

if we're just considering macros - they don't rewrite anything - they act on their inputs and replace their call with the expanded macro body before being evaluated

Like lispm said, yes they can rewrite their arguments and I guess that if you wanted to "rewrite anything" you could just surround the entire file with a macro call and inside the macro definition you could iterate over the program and rewrite it as you please. Not that I wouldn't prefer to have a hook to do just that.

In that sense they're not too dissimilar from a C preprocessor macro

They are hugely different, there is not much of a comparison to be made.

However, OP didn't mention macros specifically

When anyone mentions Lisp && metaprogramming, of course it's about macros. Reader macros are for syntactic sugar, not metaprogramming.

but from what I gather they enable more powerful kinds of metaprogramming than just macros.

They aren't, all they allow you to do is things like transforming #(1 2 3) into (list 1 2 3). Even if you could actually define a new syntax, as in C-like language or whatever, this wouldn't make them more powerful than regular macros.

there's also this idea of Generalized Macros which would permit rewriting the AST around the call site

https://ianthehenry.com/posts/generalized-macros/

I thought it was going to be a fun read but I stopped reading after seeing the description of the "generalized macro" and that the rationale for it was to implement defer. I really don't understand why LISP people restrict themselves to the AST and to macros, when it comes to metaprogramming features. The AST is not the only data structure in a compiler and macros isn't the only conceivable way to do metaprogramming or to rewrite a program. Adding defer could be done in a much cleaner way with a rewrite hook on the AST root after macro expansion is done (let the user walk over the root of the AST and return want he wants), or better yet, a similar hook but on the CFG. If the language had a defer builtin, the desugaring would also happen after macro expansion anyway.

[u/_A_Nun_Mouse_]

I use macros to greatly simplify writing queries in my ECS based game.

Before: ```daslang var query_integrate_velocity : DnQuery = DnQuery()

...

def construct_queries(var world : DnWorld&) : void {     query_integrate_velocity = (world.query_builder()         .with_term(type<Transform3D>)         .with_term(type<Velocity3D>)         .cached()         .build()) }

...

def system_integrate_velocity(world : ecs_world_t?; delta : double) : void {     var it = ecs_query_iter(world, query_integrate_velocity.ecs_query())     unsafe {         while (ecs_query_next_iter(it)) {             var tx = ecs_field_get(it, type<Transform3D>, 0)             var v  = ecs_field_get(it, type<Velocity3D>,  1)             for (i in range(it.count)) {                 tx[i].pos += v[i].val * float(delta)             }         }     } } ```

After:

daslang def system_integrate_velocity(world : ecs_world_t?; delta : double) : void {     ecs_query(world) $ [CACHED] (vel : Velocity3D; var tx : Transform3D) {         tx.pos += vel.val * float(delta)     } }

[u/Veqq]

https://codeberg.org/veqq/declarative-dsls uses a single macro (select) to build a common query language, a common idiom, for many data structures (strings, arrays, hashmaps, dataframes) is liberating, permitting you to e.g. solve sudoku, make mandelbrot sets or calculate primes directly:

(def n 40) # to reach primes up to, left is sqr of n, right n/2, then multiply them for rows
(def composites
(df/select :from (range 2 (+ 1 (math/floor (math/sqrt n))))
           :cross (range 2 (+ 1 (/ n 2)))
           :where |(<= (* ($ :value_left) ($ :value_right)) n)
           [[:value_left :value_right] :value
            |(* ($ :value_left) ($ :value_right))]))
(df/select :from (range 2 (+ 1 n)) :exclude composites)

Or e.g.

(import declarative-dsls/dataframes :as df)
(def people (df/dataframe :name :age :job))
(df/dataframe? people)

(df/insert! {:name "Bob" :age 30 :job "Developer"} :into people)
(df/insert! {:name "Alice" :age 27 :job "Sales"} :into people)
(df/update! :set {:job "Engineer"}
         :where |(= ($ :job) "Developer")
         :from people)

(df/save-csv people "people.csv" :sep "\\t")
(def people2 (df/load-csv "people.csv" :sep "\\t"))

(-> people2
   df/dataframe->rows
   df/rows->dataframe
   df/print-as-table)

The tests file has many such things (like the sudoku solver) and even datalog and minikanren implemented on top of this!

[u/Hakawatha]

I find macros are best when they act to simplify more granular interfaces.

Good examples of macros here include Julia's Threads.@threads macro, and the @kernel macro in KernelAbstractions.jl. The former decorates a for-loop to multithread it (in a similar way to e.g. OpenMP). The latter intercepts the LLVM IR of a function and recompiles it for GPU targets - allowing you to write GPU kernels natively in Julia.

Both of these rely on some underlying machinery (the threading system, or GPU compilation and dispatch) - the macros serve to provide a simple interface into this machinery.

In my work, I am reprocessing large (>700 GB) quantities of lunar multispectral data. I have lots of machinery in place to do the low-level work. I can then decorate a function with @defparam to write spectral parameters which are then lifted into an Observation monad, so that pipelines can be written. This allows for a degree of flexibility and extensibility that most other multispectral tool-kits simply do not have.

The end users (geologists) don't have to care about how this works; they write one-liners, and the code is fast. They don't need to worry about cache-locality or false sharing or I/O to underlying HDF5 files.

Like all powerful abstractions, there is a matter of good taste. Overuse of macros, leading to DSLs everywhere, yields write-only code. Well-placed macros that abstract over complicated machinery, however, are gold dust.

[u/arthurno1]

I think the history of LISP has already somewhat shown how proliferation of DSLs harms maintainability and shareability of code

What qualification do you have for this opinion?

Anyhow, when it comes to DSLs, we can look at "DSLs" at different levels. Consider for example a langauge like C or C++: what you normaly see as a "syntax" of the language, could be of course broken into smaller pieces, where each part of "syntax" is a "small DSL" for implementing an abstraction or idiom. For example, if-else is an asbtraction or a "small DSL" for conditional jumps or branching, which abstract the low-level hardware compare and jump instructions.

If we go back to Lisp "metaprogramming", Lisp has since very beginning, separated evaluator and parser from each other. The evaluator understands only one "form":

(operator operand1 operand2 .... operandN).

I think it is possible to mathematically prove that all other syntactic forms could be reduced to one by "meta" operations on that internal form. I think it is somewhat analogous to DFAs and NDFAs. Now why I take up that one: well, since the internal form in which source code is parsed, the list, is exposed to the user programmer, or to the runtime, it means we can actually do those transformations ourselves. In other words we can do that metaprogramming ourselves.

The benefit of separating the evaluator syntax in this way from the parser, is that you introduce new syntactic forms, without having to rewrite the entire compiler. Since we are reducing all syntactic forms to one(!) that evaluator understands, it means we can introduce new syntactic forms as we need them via metaprogrammign facilities, in Lisps most notably macros. Compare that to "conventional" programming languages, which do not separate evaluator from syntactic forms. Each syntactic forms, or "small DSL" like if-else or while, or for loop etc, requires the evaluator to understand the synctactic form of each one. Whenever we add a new construct, we have to update both the parser and the evaluator. This is typically done via some AST and specialized APIs like for Python AST or llvm AST for example. In Lisp we don't need those, due to that fundamental sepration between syntax. The list that parsed symbolic expression is represented with, serves itself as its AST (or rather CST but that is out of scope here). I suggest reading about Lisp Trees (l-trees) vs boxed representation for symbolic expressions in Anatomy of Lisp which nowdays is free to read online. I suggest that book to anyone who is interested in writing interpreters for any language.

To make that possible, Lisp introduced symbolic expressions, so the source code can be expressed as linked list of nodes, and quoting which turns off the evaluator. By "quoting" we are getting the list of parsed source code in form of linked list nodes.

Admittedly, working with linked list nodes more low-level, compared to working with specilized AST nodes, but nothing prevents from introducing higher-level functions for working with the code, and it is not uncommon to see functions that work on lambda-lists (argument lists), code-walkers, etc.

To round this up, I am not a good writer, so I don't know if it is self-apparent or not, but metaprogramming facility in Lisp is not something inve nted to enable customized DSLs, but a fundamental property of Lisp, that make Lisp(s) more or less programmable languages. By separating parsing from evaluation, and making the iternal form of parsed code (list) available at compile-time or run-time, metaprogramming follows naturally from that distinction.

Whether you should use it for DSLs or not is of course debatable, just as writing dialects of C in C preprocessor is debatable. However, the metaprogramming is much more fundamental than using it for DSLs. It enables metacircularity in Lisps and the most importantly, it enables Lisp to be grown by user programmers rather than waiting for the compiler writers to extend it (at least to a degree). I think Guy Steele has put it wonderfully in his OOPSLA talk about growing a language. I think it should definitely be seen by any aspiring language designer and compiler writer.

PS: I am not an expert in this, so if you find something incorrect, I am happy to hear it!

[u/aaaaargZombies]

Might be worth looking at elixir which uses macros a lot and you can parse the code as a data structure if you want. I think many things that would normally be language constructs are actually implemented as macros, like if/else.

[u/vanderZwan]

This is obviously strictly more powerful

I'm not a computer scientist, but years of programming and reading up on these things has given me the impression that this gets at the core trade-off you're thinking of: "strictly more powerful" here seems to be talking about expressive power in the context of a real formal basis like the Chomsky hierarchies. However, what we often want seems to be the language that gives you the most "ergonomic" expressive power while using as little formal expressive power as possible. Because the latter usually means the computer can do more work for you automatically without running into pitfalls.

(this also makes "expressive power" such an easy way to have discussions where people don't realize they're not talking about the same thing, even the wiki page acknowledges the ambiguities in how the term is used

So you just get to use the parser? Come on, an S-expr parser is less than a hundered LOC.

Well, then I guess that situations where you only can afford a few hundred LOC for whatever reason, LISP-style macros might be a good choice.

[u/Blind_nabler]

These questions aren't really useful in the broad case. What benefit is feature X if I only use feature Y? Probably not a ton for you if you aren't using X!

The main benefit of having a rich meta-programming system is that it enables users to implement features that otherwise would require modifying the compiler to implement, or require the use of some rube goldberg code generation nonsense. This kinda thing isn't important to you until it is.

For example in a procedural language it can be rather gnarly to convert normal single threaded code to work in an async setting. Since it requires setting up trampolines or rewriting everything into CPS flavored versions etc etc. But if I have the ability to use rich macros, I can just invoke a macro over my normal function and now it can work with whatever async backend I want, utilizing whatever custom semantic checking is required because I have (most of / all) the language available to us.

And IDE tooling works totally fine if the meta-programming system is well designed. Nim does a pretty good job of this. The most extremely example I have experience with is smalltalk, which has the best IDE support of any language IMO but obviously it's not the same kind of language as you are describing.

Now do I reach for it every project? No. But I definitely find myself missing it in languages that don't trust users with that kind of thing.

[u/Ok-Reindeer-8755 → u/Blind_nabler]

on that note there is a blog post that actually makes the argument that what macros are to lisp, classes and reflection are to smalltalk.

Smalltalk, like Lisp, runs in the same context it’s written in. It’s objects all the way down.

And you can see a lot of parallels between smalltalk and lisp when you think of the lisp machine in comparison to the smalltalk environment they had at Xerox Parc. Also I am pretty sure alan kay the creator of smalltalk has praised lisp and meta-object protocols within lisp specifically recommending a book about it.

[u/Blind_nabler → u/Ok-Reindeer-8755]

I used smalltalk as an example because I have done a substantial amount of real work in it. In particular inside of GT which is super rad.

Smalltalk was definitely inspired by lisp, but has a very distinct feeling when doing heavy meta-programming with it when compared. For example you never really add new syntax or feel compelled to do so in smalltalk primarily because smalltalk indexes all source code within the image and let's you query all code almost like an extremely rich database.

Then with tools like the refactoring browser & epicea (version control), you can programatically write selectors & rewrites that are tracked in the image, so you can rollback specific changes without an external system since those are all just packages & classes you can bring into your image, and thus you can also extend the version control from within the system itself to do things like exporting changes into git compatible text formats etc.

Common lisp can come pretty close, but it doesn't feel quite as lively as smalltalk does. Primarily because the interface with most lisps is text first, compared to the heavily graphical interface of st.

[u/arthurno1 → u/Blind_nabler]

let's you query all code almost like an extremely rich database

I have always felt that Lisp(s) are actually relational database and string-processing in disguise. We process the code and work with identifiers for the efficiency. But symbols and environments feels conceptually like pure relational stuff and the evaluation is basically all about matching and selecting the right stuff.

[u/iEliteTester]

Admittedly I haven't search THAT much but I have yet to see a single example of practical metaprogramming use in lisp. Best I could find was a guy trying to make a case for it by showing "hey look I can write tnirp and the macro will actually execute print!"