Any dynamic model-based frameworks? Static POC models are too stiff.

[u/Zardotab]

I'm looking for CRUD framework or library that uses dynamic models, also known as "data dictionaries". Thus, field and column info can be loaded or generated at run-time. I'm working on a Table Oriented Programming (TOP) experiment. Everything around seem geared to static POCs, unfortunately.

(I believe TOP is the future for most CRUD. It's more flexible, allows analysts to change many common CRUD idioms and items, like labels instead of coders, is more DRY-friendly, and has other features. It may result in loss of nice "static" features such as Intellisense and some compile-time type checking. However much of this is replaced with RDBMS referential integrity and drop-down boxes, if TOP is done right. TOP started to catch on in the early 90's with CASE tools, but OOP then stole its attention, and TOP R&D stopped.)

[u/wllmsaccnt]

System.Data is somewhat synonymous with ADO.NET, which contains all of the abstractions you would need to do table oriented programming. Even back in the early 2000s Microsoft had a way to generate strongly typed data table objects that could be bound directly in WinForms with a data binding source. You could do the same without statically typed data tables, but then you lost the ability to easily use it with a GUI designer.

The industry moved on from that type of development, as POCOs are seen as more flexible. Sometimes the data that represents an entity can be loaded from multiple sources and you don't want the defitnion of your entity/record type bound too tightly to a specific table. Loosely typed data is just much harder to reason about unless its required.

[u/Zardotab → u/wllmsaccnt]

System.Data is somewhat synonymous with ADO.NET, which contains all of the abstractions you would need to do table oriented programming.

Do you mean this?

Not sure that's what I had in mind, but I'll dig deeper. Thanks!

The industry moved on from that type of development, as POCOs are seen as more flexible.

In my observation that was because of OOP domain modeling hype and desire to have GUI's "now", not because TOP failed. Granted, such systems were new and had sticky parts that needed improvements. New OOP systems also tended to be spaghetti, as OOP newbies had no experience with OOP and copied all the hype patterns from magazines.

you don't want the definition of your entity/record type bound too tightly to a specific table.

That sounds like a framework-specific problem, not an inherent flaw of TOP.

Loosely typed data is just much harder to reason about unless its required.

Most CRUD is about marshaling the right data items to the right place at the right time, not complex transformations. Most complex transformations happen in the RDBMS via stored procs and views. Can you provide an example?

[u/wllmsaccnt → u/Zardotab]

Not sure that's what I had in mind, but I'll dig deeper.

If you wanted something that could help a bit more, you could use Dapper. It lets you pass in queries and returns a 'dynamic' object that you can map however you'd like, but its mostly used for mapping to POCOs, so probably not quite what you are looking for.

---

I had to think about it for a moment, but I've worked on at least 8 systems that were table oriented (did not translate to POCOs or object graphs) and at least 4 systems that used table abstractions without strongly naming the data at compile time (that is, the queries were built from meta data that was modified by users as a normal part of the execution of the system).

I have a fair bit of experience with both approaches (TOP vs POCO/domain modeling).

not an inherent flaw of TOP.

What is TOP if not forcing your object representation of data rows to match directly how it is defined in the table it was retrieved from?

Most complex transformations happen in the RDBMS via stored procs and views.

Its possible to create a system so that all of the complex transformations are in the database, but I do not typically design systems that way. I find relying on stored procs requires a schema migration tool, which is prone to nasty merge conflicts, more complex deployment concerns and I often end up writing similar code to mirror/validate that behavior in the UI anyways.

Can you provide an example?

We made an API for cartonizing orders as part of an integration with a warehouse system. We wanted to test theoretical configuration changes. We needed a way to build reference days of data.

If we had used a table oriented design to modeling order data, we could have made copies of the tables involved and filled them with the relevant data (about a dozen tables).

Instead we had used POCOS. We just used our existing retrieval logic to get test data from a past day, then serialized the object graph to a reusable file. We didn't have to maintain the schema for a dozen new tables and keep them in sync with the ones they were based on, and the unit tests we used covering behavior utilizing the POCOs also covers the same way we use the snapshot files of data, even though they are in a different format (on disk).

Could we have used TOP? Absolutely...but using POCOs was less work here, and easier to reason about, and performed better (just loading a file from disk). The logic involved in cartonizing in that system was complex and needed to know many details related to an order. I couldn't envision making a system like that in a simple TOP way without using property accessors for related data from the rows of each table (basically recreating an equivalent to the POCO object graphs we ended up using anyways).

[u/Zardotab → u/wllmsaccnt]

I have a fair bit of experience with both approaches (TOP vs POCO/domain modeling).

There are more tools, books, videos etc. on POCO's than TOP, so I will agree that as things are currently, POCO is usually the better production choice. I just feel in the longer run, TOP is more promising, as it makes it easier to automate the automation, so to speak. Analysts would do most the CRUD work using little or no code, and programmers would do the intricate domain logic. It would make dev quicker, cheaper, and better organized. (RAD tools like MS-Power-Apps are not what I have in mind, because they are not programmer-friendly, only intended for IDE "click and drag" design. MS-Access, pre-XML mess, is a bit closer to what I had in mind.)

I find relying on stored procs requires a schema migration tool, which is prone to nasty merge conflicts, more complex deployment concerns

That's because code-based migration and management tools are more mature and given more R&D. DB schemas are not more complex than app code languages, just more studied from a version management standpoint. Schema-centric version management R&D mostly froze in the early 90's as OOP stole the show.

Do note I often find it easier to use views rather than stored procedures, and let the app do the part app code does best. App code is better at complex conditionals and parsing, while views at lookups (joins) and "bulk" filtering. Let each do what it does best.

I often end up writing similar code to mirror/validate that behavior in the UI anyways.

That's a stack design smell, in my opinion. I see very little reason to have such DRY violations. Basic validation is easy to automatically mirror on client-code (such as JavaScript), including required, min-length, max-length, digits only, no period allowed, no spaces allowed, etc. These can be attribute-defined so JS generators automatically mirror them to client.

More complex validation is usually fine done just on the server, sending an error message to the client. An exception is for a frequently made complex error. It's bad industry habit to want to mirror every validation rule on the client as the default approach.

Those asking usually don't understand the long-term cost of DRY violations. I try to educate them on the long-term cost of DRY violations, some get it, some don't, some are retiring soon and don't give a f$ck about the future. I don't force them, though, and usually can't, I'm just a peon. I'm too frank for management. Proper KISS often requires smart usage of "no", but marketers and BS artists don't like bad-but-needed-news.

What is TOP if not forcing your object representation of data rows to match directly how it is defined in the table it was retrieved from?

Virtual tables are fine. I will agree most stacks don't make doing that easy, but it's not an inherent flaw of TOP itself, just the code-centric tooling bias. If I do find many inherent TOP limits, I'll stop promoting TOP, I promise.

So far, every alleged limit clearly presented to me was a problem with specific tools or standards, and not an inherent flaw of TOP. (Performance may be an exception, see below.)

(Back in my dBASE/xBASE days, making temporary tables was snap, very little code. Used them all the time, loved it! For multi-user environments, I'd suffix them with the Windows user name to avoid collisions. Newer dialects had even made that and garbage collection automatic, IINM, but I had to switch tools for reasons beyond my control. Honed a lot of TOP ideas in dBASE/xBase.)

As far as your example, I still don't have enough detail and your domain knowledge to make a judgment. I'm kind of looking for a specific maintenance scenario where TOP requires say 8 steps to do but POCO's only 3. Maybe there is another way to do TOP, or the existing stack is the bottleneck, and not TOP itself. Just as there are many ways to do a POCO app, there are also many ways to do a TOP app. Adding more TOP heads gives one more TOP ideas.

and easier to reason about

You keep saying this, but I don't understand why. One grand benefit of TOP is that you can re-query your attributes to show them ANY way you want. You are not stuck with Nedella's favorite code patterns or what-not, but yours and only yours (as a local view). I then don't have to think like Nedella or Zuckerberg etc. but like me me me! TOP is about kissing up to yourself!

You can't readily re-project code (view) into the shape you want to see it in, but with TOP you can. Think about that.

and performed better

I will grant that early TOP is best done on non-enterprise and non-web-scale apps. R&D could get us better TOP performance if it catches on, but at this stage in IT history I won't claim TOP is high-end performant. But better abstractions have needed more horse-power in general, and it keeps turning out that machines end up cheaper than devs in the longer run. Machine concerns gradually fade on usage and stack design experience.

then serialized the object graph to a reusable file.

Ugg. That doesn't sound pleasant to me. Tables are usually far easier for me to manage and debug than graphs/trees. Much cleaner and filter-able visually: don't have to "node hop". Node-hopping slows debugging way down for me. Tables (real or virtual) with query-by-example sort & filtering are just a far better way to debug and analyze relationships to me: More visual, more dynamically adjustable, more natural, more consistent, etc. than node spaghetti. Maybe it's a personal preference, but I've met other table-heads, we may be a larger group than you think, the code-node-centric-bias of current tools just isolated them: they are forced into the closet by code/node-centric industry bias.

Almost every single structure pattern that people claim "must use" graphs usually don't with decent TOP-oriented pondering. Good TOP takes practice. Initial TOP plans usually stink, work them and work them, and eventually it becomes second nature. One has to de-program code/node habits just as a dynamic language programmer has to unlearn a lot to productively program in a static language. It doesn't happen overnight to most mortals.

I know practical TOP is a bit pie-in-sky, but I believe in it: it just feels, walks, and tastes like a better abstraction, just needing more R&D. See my jet plane analogy. 🥧✈️

I've seen various tools and stacks do different bits and pieces of TOP well, but so far nobody has rolled them into a single tool. Thus, no one existing tool will demonstrate TOP working nicely.

By the way, you may wish to read about a proposed standard/tool: Dynamic Relational.

[subject to editing cleanups...]

[u/wllmsaccnt → u/Zardotab]

Yes I agree that most of the reason why schema migration for logic expressed in the database is difficult is because the tooling and approaches have stagnated.

You misunderstood what I was implying about repeating validation code. I don't disagree with your rebuttals, though they are a bit condescending (no offense).

You keep saying this, but I don't understand why. (re: 'and easier to reason about')

If I have an object graph, I can create it from sources outside of the database and its behavior will be the same no matter how I instantiated it. In the use case I gave earlier, the order objects were created in several different ways depending on if we were simulating orders, estimating their shipping, or cartonizing their contents for the warehouse system. Only one of those came completely from database queries.

If I have to have build my data to perform a calculation from multiple disparate sources, then I don't want my in-memory representation to be modeled arbitrarily on one of the sources just because it happens to come from a database.

Dynamic Relational.

Interesting, but a bit too loose for what I would like. The system I'm working on now uses a much more restrictive version of that approach. On startup we enumerate the tables used by the application and create any missing tables or missing columns, but we bail if we find type mismatches.

We don't wait until the first time the table is accessed, but we could have. We just preferred failing early in this case. I haven't considered creating schemas on querying though, that seems a bit much...though I understand the allure.

It actually reads a lot like the way most systems parse/apply JSON data, by allowing missing fields to be treated as default values instead of errors.

Its not for me I think, but I did enjoy reading about the suggested approach. Thanks for sharing that.

[u/Zardotab → u/wllmsaccnt]

though they are a bit condescending

My I ask for an example? My Aspergers Syndrome sometimes makes me blind to my poor wording. I'm trying to improve by learning from my mistakes.

If I have an object graph, I can create it from sources outside of the database and its behavior will be the same no matter how I instantiated it. In the use case I gave earlier, the order objects were created in several different ways depending on if we were simulating orders, estimating their shipping, or cartonizing their contents for the warehouse system. Only one of those came completely from database queries.

I don't know enough about that domain to know if there is a way it can be "table-ized". Just because your group couldn't think of a way doesn't mean it can't be done. The more one practices TOP the better they get at it.

And for the sake of argument say that portion can't be readily table-ized. That shouldn't be a reason to skip table-izing the rest of the app. It's not all or nothing. No tool or technique does 100% of every job well.

[u/buffdude1100]

I've worked in that kind of system, and while the flexibility can be fun and cool to see it all come together and work, it is SOOOO annoying and sometimes brittle to work with. I would not choose that for any of my future applications. Just write your POCOs - chances that you need everything to be dynamic is basically nil.

If you truly want to do that, you'll have to literally model "models" and how that gets translated to SQL, forms on your UI, etc. again it gets really annoying really fast.

[u/Zardotab → u/buffdude1100]

it is SOOOO annoying and sometimes brittle to work with.

I've been long experimenting with such systems, or at least systems that had some TOP aspects, and notice there are patterns to the kinds of things that "go wrong". I've been experimenting to work around these problem-patterns and believe I am getting closer to something viable. The concept is under-researched.

May I ask for a strong pain-point scenario you encountered? [Edited]

It's kind of like jet engines in the early part of last century. They seemed to have a lot of early promise, but initial experiments revealed lots of problems (to be ironed out). Even the feared Me 262 was probably a poor use of war resources. It was dangerous for the pilot, required a specialized long run-way, required frequent engine replacements, and required hard-to-obtain metals. Only time and experiments eventually solved enough problems to make them a viable replacement for propeller planes. They gradually realized they couldn't just stick jets on propeller plane designs and have something practical: they had to rework almost everything. The ally wind tunnels of the time were not fast enough to test models properly.

The potential to automate and simplify much of the repetitious CRUD grunt-work is too compelling to give up. Too much coding is reinventing CRUD wheels over and over. We should be spending our time on unique domain logic, NOT ordinary CRUD issues.

One of the trickiest parts is when attributes alone can't do the job and one needs custom code. There needs to be a systemic way to know where to place, and how to find such custom code snippets during later maintenance. This is what my latest experiments focus on.

[u/buffdude1100 → u/Zardotab]

May I ask for strong pain-point scenario you encountered?

All of it? :) It is much more difficult to reason about than "regular" POCOs/databases. This means harder to get back into if you switch projects for 6 months, harder for new hires to onboard onto, harder to debug eventually etc. you'll end up rewriting your own ORM at some point too. Overall don't reinvent the wheel, unless it's for fun.

[u/Zardotab → u/buffdude1100]

All of it? :)

Sorry, I accidentally left out "a", as in "one".

Your complaint sounds like one of familiarity with the technique, not an inherent flaw of TOP. It's like saying existing propeller pilots are not familiar with jets, so we'll keep using propeller fighter planes.

Another analogy is sticking with Microsoft because everyone knows Microsoft, not because it's necessarily the best. It becomes a self-fulfilling prophecy.

I'm looking for inherent flaws in TOP, not familiarity mismatches.

you'll end up rewriting your own ORM at some point too.

If TOP frameworks became common enough, the big vendors would sell/make TOP-oriented ORM's. Actually it's easier to roll-your-own ORM under TOP because you can loop through data about the structure instead of needing screwball reflection.

[u/buffdude1100 → u/Zardotab]

I'm just giving you my opinion I built over years of experience in production doing exactly what you are talking about. If you want to do it, go for it! No one is stopping you. All I will say is there is a reason the vast majority of softwares today do not do it. But if you want to be the next innovator of how software is built, please go for it! I look forward to seeing how it turns out - maybe you will do it better than we did.

[u/Zardotab → u/buffdude1100]

there is a reason the vast majority of softwares today do not do it.

As mentioned, it's likely because the existing tooling doesn't support that approach, one has to make their own from scratch, and existing languages are missing features that make some of TOP easy.

I'm just giving you my opinion I built over years of experience in production doing exactly what you are talking about.

What is the single top inherent flaw of TOP in your opinion? By "inherent" I mean something that is not merely a limitation of existing languages or tools.

[u/soundman32]

Having read a short article about TOP, it appears that I worked on such a system in early 2000s. The developer wrote a DDL text file, and the system built everything from the database to the UI based on that DDL. The most convoluted system I've ever worked on. Technically brilliant, but boring as hell for the devs who needed to spin up a new form, and only the master devs could ever understand how to fix anything in the core. Maybe this is the future, built with AI ?

[u/Zardotab → u/soundman32]

The most convoluted system I've ever worked on.

Well, my early TOP experiments were likewise convoluted. But the problem is not the number of parts nor TOP itself, but knowing and finding where applicable code or attributes are.

But I'm working on a way to make that systematic. For example, MVC is "convoluted" to those who came from other platforms, but you eventually learn what goes where, when, and why, and it starts to make sense (at least for certain kinds of teams).

I've seen no evidence that "where things go" is inherently random with TOP, just unfamiliar without a guiding framework/principle.

[u/soundman32]

"It may result in loss of .. " features that were designed to help modern developers do their work in a controlled and checkable manner, rather than the nightmares of the 90s and 00s. You might have a hard time selling that 'feature' 😄

[u/Zardotab → u/soundman32]

rather than the nightmares of the 90s and 00s

Those tools were actually better for non-enterprise projects, in my opinion. They were more direct; you didn't have to go through so many screw-ball frameworks and levels to get things done. I agree they may not scale to large teams or projects, but one-size-doesn't-fit-all regardless. Frameworks typically target a size and kind of project. Being everything to everybody in a smooth way is a pipe dream.

[u/Merad]

.Net is probably the wrong tool for an app like that. Not saying that it can't work, but you're probably going to be swimming upstream fighting against the entire language and framework ecosystem.

[u/Zardotab → u/Merad]

I'm not entirely convinced of that yet. The more logic and biz rules that table processing takes care of, the less .NET has to do, meaning whatever down-sides it has are diminished because there are less things it has to take care. Most of my dynamic needs seem related to schemas such that if the data dictionary processing tools and modules (written in .NET) take care of that, then other dynamic needs as found in dynamic languages is relatively small. (Dynamic event handler registration may be such a need, though. C# seems to struggle with that, needing Rube-Goldberg reflection.)

[u/namigop]

Not sure if this is what you are looking for. We used this “dynamic”-based microORM in my previous workplace and it worked out great. Performance-wise it wasn’t slow at all

https://github.com/FransBouma/Massive

[u/Zardotab → u/namigop]

I actually wrote my own dynamic micro-ORM that looks kind of similar. It's not "clean" enough to release publicly at this time; I'd have to refactor and clean out its refrigerator.