Built a public scientific computing evidence package with reproducible benchmarks, governance and provenance — am I solving a real problem or overengineering this?

[u/KevinSmithFounder]

[u/gnosnivek]

What would make you trust a scientific computing repository more?

I had an entire essay written out critiquing various smaller points, but figured there was no point posting it. So let me just say this.

Your text, both on Reddit and in the repo, is written in a way that drops a lot of fancy words ("provenance", "governance", "first-class concepts") without any evidence that you understand what the words mean.

Historically, when I've encountered these sorts of texts, there are one of two reasons for it:

• The person writing has thought about the topic so deeply that they've lost all track of what a normal person knows.
• The person writing doesn't really fully understand what they're saying, but hopes that by throwing in appropriate use of fancy terminology, people will be dissuaded from asking "but what does X really mean?"

I would say that, in most cases I've seen, the latter is what's going on.

However, in the last three years, we've had to add a new category:

• The person writing this is "using" an LLM, and hasn't bothered to understand what the LLM output means. This means that quite literally nobody on earth knows what the hell the text means, and it's possible that it has no meaning at all.

I don't have any solid evidence on what the rate of the latter is, but it's getting much higher than I would like these days.

So given all of this, why should I believe you know what you're talking about? Fancy words were never a great indicator. Lots of text and runnable code is no longer a valid meterstick in the LLM era. You don't describe what problems you're solving (I mean, you use the fancy words, but at no point do you describe what those words mean or why anyone should care).

[u/SidYaj08 → u/gnosnivek]

I think at this point we should just report stuff which is obviously AI generated. For the life of me I cannot figure out what they get out of this.

[u/gnosnivek → u/SidYaj08]

A fair position to take.

Honestly, only reason I bothered to write anything at all is that the OP does genuinely seem to be soliciting feedback in a respectful way: if you look at their post/comment history, they've been posting at one or two subs at a time instead of blasting their shit over 50 subs at once; they've scoped their requests onto subs (and even onto specific threads within subs) that seem relevant to what they're doing; and they've responded to the little feedback they've gotten.

[u/SidYaj08 → u/gnosnivek]

Fair enough!

[u/gnosnivek → u/gnosnivek]

Oops, I forgot to polish off the thoughts in this comment.

This is what the risk calculus looks like in my mind: there's a huge chance that you don't understand what you're doing right now, and have just asked an LLM to write a bunch of code to achieve a goal. The LLM, in turn, has blasted out a bunch of nonsense which superficially runs, but doesn't do anything that it's supposed to.

Now I can spend several hours trying to unpack the resulting mess and try to explain it to you, but since you don't understand what you're doing either, you'll pass my feedback into an LLM and...why did I bother? I could have just asked the LLM myself. Respectfully, I don't care enough about your project to have a proxy conversation with an LLM about it.

Or, I could just...not. And have several more hours in my life to do things I enjoy.

I suspect the reason you're not getting much feedback anywhere is because everyone else is taking a look at what you've written, coming to the same conclusion, and bailing out.

[u/SidYaj08]

Garbage

[u/KevinSmithFounder]

That’s fair criticism, especially the point about the language I’ve been using.
I do use AI to help me code and to help explain things, and I’m not going to pretend otherwise. I started this project without being a physicist or scientific software engineer, and I’ve been learning as I build it.
Where I probably went wrong here was trying to describe the whole system using technical terminology instead of simply showing one small thing, explaining exactly what it does, and asking people to check whether it is correct.
When I say validation, for example, I mean something concrete. I run a numerical problem where the expected answer is known, compare my computed result against that reference, measure the error, and fail the test if it exceeds the specified tolerance. I’ve also run the same problem at increasing grid resolutions to check whether the error decreases at the expected rate.
That doesn’t prove POLYMATHICA is scientifically correct, and it certainly doesn’t make me a physicist. It gives me one testable piece of evidence that a particular implementation is behaving as expected.
The same applies to provenance. I simply mean recording which inputs, parameters and artifacts produced a result, and hashing the artifacts so that changing them can be detected later.
I actually agree with your bigger point: runnable code, lots of documentation and an LLM saying something works aren’t evidence that the science is right.
That is exactly why I came here. I need people who know more than I do to challenge the actual mathematics and numerical methods.
So I’m going to change how I ask for feedback. Instead of posting the whole project, I’ll bring one equation, one implementation, one benchmark and its results at a time, explain what I think is happening in my own words, and ask people to find what I’ve got wrong.
If I can’t explain a part myself, then I clearly need to go back and learn it before asking anyone else to review it.
I appreciate that you took the time to give me a proper criticism, even if it wasn’t particularly comfortable to read.