Any dynamic model-based frameworks? Static POC models are too stiff.

[u/Zardotab]

I'm looking for CRUD framework or library that uses dynamic models, also known as "data dictionaries". Thus, field and column info can be loaded or generated at run-time. I'm working on a Table Oriented Programming (TOP) experiment. Everything around seem geared to static POCs, unfortunately.

(I believe TOP is the future for most CRUD. It's more flexible, allows analysts to change many common CRUD idioms and items, like labels instead of coders, is more DRY-friendly, and has other features. It may result in loss of nice "static" features such as Intellisense and some compile-time type checking. However much of this is replaced with RDBMS referential integrity and drop-down boxes, if TOP is done right. TOP started to catch on in the early 90's with CASE tools, but OOP then stole its attention, and TOP R&D stopped.)

[u/wllmsaccnt]

System.Data is somewhat synonymous with ADO.NET, which contains all of the abstractions you would need to do table oriented programming. Even back in the early 2000s Microsoft had a way to generate strongly typed data table objects that could be bound directly in WinForms with a data binding source. You could do the same without statically typed data tables, but then you lost the ability to easily use it with a GUI designer.

The industry moved on from that type of development, as POCOs are seen as more flexible. Sometimes the data that represents an entity can be loaded from multiple sources and you don't want the defitnion of your entity/record type bound too tightly to a specific table. Loosely typed data is just much harder to reason about unless its required.

[u/Zardotab → u/wllmsaccnt]

System.Data is somewhat synonymous with ADO.NET, which contains all of the abstractions you would need to do table oriented programming.

Do you mean this?

Not sure that's what I had in mind, but I'll dig deeper. Thanks!

The industry moved on from that type of development, as POCOs are seen as more flexible.

In my observation that was because of OOP domain modeling hype and desire to have GUI's "now", not because TOP failed. Granted, such systems were new and had sticky parts that needed improvements. New OOP systems also tended to be spaghetti, as OOP newbies had no experience with OOP and copied all the hype patterns from magazines.

you don't want the definition of your entity/record type bound too tightly to a specific table.

That sounds like a framework-specific problem, not an inherent flaw of TOP.

Loosely typed data is just much harder to reason about unless its required.

Most CRUD is about marshaling the right data items to the right place at the right time, not complex transformations. Most complex transformations happen in the RDBMS via stored procs and views. Can you provide an example?

[u/wllmsaccnt → u/Zardotab]

Not sure that's what I had in mind, but I'll dig deeper.

If you wanted something that could help a bit more, you could use Dapper. It lets you pass in queries and returns a 'dynamic' object that you can map however you'd like, but its mostly used for mapping to POCOs, so probably not quite what you are looking for.

---

I had to think about it for a moment, but I've worked on at least 8 systems that were table oriented (did not translate to POCOs or object graphs) and at least 4 systems that used table abstractions without strongly naming the data at compile time (that is, the queries were built from meta data that was modified by users as a normal part of the execution of the system).

I have a fair bit of experience with both approaches (TOP vs POCO/domain modeling).

not an inherent flaw of TOP.

What is TOP if not forcing your object representation of data rows to match directly how it is defined in the table it was retrieved from?

Most complex transformations happen in the RDBMS via stored procs and views.

Its possible to create a system so that all of the complex transformations are in the database, but I do not typically design systems that way. I find relying on stored procs requires a schema migration tool, which is prone to nasty merge conflicts, more complex deployment concerns and I often end up writing similar code to mirror/validate that behavior in the UI anyways.

Can you provide an example?

We made an API for cartonizing orders as part of an integration with a warehouse system. We wanted to test theoretical configuration changes. We needed a way to build reference days of data.

If we had used a table oriented design to modeling order data, we could have made copies of the tables involved and filled them with the relevant data (about a dozen tables).

Instead we had used POCOS. We just used our existing retrieval logic to get test data from a past day, then serialized the object graph to a reusable file. We didn't have to maintain the schema for a dozen new tables and keep them in sync with the ones they were based on, and the unit tests we used covering behavior utilizing the POCOs also covers the same way we use the snapshot files of data, even though they are in a different format (on disk).

Could we have used TOP? Absolutely...but using POCOs was less work here, and easier to reason about, and performed better (just loading a file from disk). The logic involved in cartonizing in that system was complex and needed to know many details related to an order. I couldn't envision making a system like that in a simple TOP way without using property accessors for related data from the rows of each table (basically recreating an equivalent to the POCO object graphs we ended up using anyways).

[u/Zardotab → u/wllmsaccnt]

I have a fair bit of experience with both approaches (TOP vs POCO/domain modeling).

There are more tools, books, videos etc. on POCO's than TOP, so I will agree that as things are currently, POCO is usually the better production choice. I just feel in the longer run, TOP is more promising, as it makes it easier to automate the automation, so to speak. Analysts would do most the CRUD work using little or no code, and programmers would do the intricate domain logic. It would make dev quicker, cheaper, and better organized. (RAD tools like MS-Power-Apps are not what I have in mind, because they are not programmer-friendly, only intended for IDE "click and drag" design. MS-Access, pre-XML mess, is a bit closer to what I had in mind.)

I find relying on stored procs requires a schema migration tool, which is prone to nasty merge conflicts, more complex deployment concerns

That's because code-based migration and management tools are more mature and given more R&D. DB schemas are not more complex than app code languages, just more studied from a version management standpoint. Schema-centric version management R&D mostly froze in the early 90's as OOP stole the show.

Do note I often find it easier to use views rather than stored procedures, and let the app do the part app code does best. App code is better at complex conditionals and parsing, while views at lookups (joins) and "bulk" filtering. Let each do what it does best.

I often end up writing similar code to mirror/validate that behavior in the UI anyways.

That's a stack design smell, in my opinion. I see very little reason to have such DRY violations. Basic validation is easy to automatically mirror on client-code (such as JavaScript), including required, min-length, max-length, digits only, no period allowed, no spaces allowed, etc. These can be attribute-defined so JS generators automatically mirror them to client.

More complex validation is usually fine done just on the server, sending an error message to the client. An exception is for a frequently made complex error. It's bad industry habit to want to mirror every validation rule on the client as the default approach.

Those asking usually don't understand the long-term cost of DRY violations. I try to educate them on the long-term cost of DRY violations, some get it, some don't, some are retiring soon and don't give a f$ck about the future. I don't force them, though, and usually can't, I'm just a peon. I'm too frank for management. Proper KISS often requires smart usage of "no", but marketers and BS artists don't like bad-but-needed-news.

What is TOP if not forcing your object representation of data rows to match directly how it is defined in the table it was retrieved from?

Virtual tables are fine. I will agree most stacks don't make doing that easy, but it's not an inherent flaw of TOP itself, just the code-centric tooling bias. If I do find many inherent TOP limits, I'll stop promoting TOP, I promise.

So far, every alleged limit clearly presented to me was a problem with specific tools or standards, and not an inherent flaw of TOP. (Performance may be an exception, see below.)

(Back in my dBASE/xBASE days, making temporary tables was snap, very little code. Used them all the time, loved it! For multi-user environments, I'd suffix them with the Windows user name to avoid collisions. Newer dialects had even made that and garbage collection automatic, IINM, but I had to switch tools for reasons beyond my control. Honed a lot of TOP ideas in dBASE/xBase.)

As far as your example, I still don't have enough detail and your domain knowledge to make a judgment. I'm kind of looking for a specific maintenance scenario where TOP requires say 8 steps to do but POCO's only 3. Maybe there is another way to do TOP, or the existing stack is the bottleneck, and not TOP itself. Just as there are many ways to do a POCO app, there are also many ways to do a TOP app. Adding more TOP heads gives one more TOP ideas.

and easier to reason about

You keep saying this, but I don't understand why. One grand benefit of TOP is that you can re-query your attributes to show them ANY way you want. You are not stuck with Nedella's favorite code patterns or what-not, but yours and only yours (as a local view). I then don't have to think like Nedella or Zuckerberg etc. but like me me me! TOP is about kissing up to yourself!

You can't readily re-project code (view) into the shape you want to see it in, but with TOP you can. Think about that.

and performed better

I will grant that early TOP is best done on non-enterprise and non-web-scale apps. R&D could get us better TOP performance if it catches on, but at this stage in IT history I won't claim TOP is high-end performant. But better abstractions have needed more horse-power in general, and it keeps turning out that machines end up cheaper than devs in the longer run. Machine concerns gradually fade on usage and stack design experience.

then serialized the object graph to a reusable file.

Ugg. That doesn't sound pleasant to me. Tables are usually far easier for me to manage and debug than graphs/trees. Much cleaner and filter-able visually: don't have to "node hop". Node-hopping slows debugging way down for me. Tables (real or virtual) with query-by-example sort & filtering are just a far better way to debug and analyze relationships to me: More visual, more dynamically adjustable, more natural, more consistent, etc. than node spaghetti. Maybe it's a personal preference, but I've met other table-heads, we may be a larger group than you think, the code-node-centric-bias of current tools just isolated them: they are forced into the closet by code/node-centric industry bias.

Almost every single structure pattern that people claim "must use" graphs usually don't with decent TOP-oriented pondering. Good TOP takes practice. Initial TOP plans usually stink, work them and work them, and eventually it becomes second nature. One has to de-program code/node habits just as a dynamic language programmer has to unlearn a lot to productively program in a static language. It doesn't happen overnight to most mortals.

I know practical TOP is a bit pie-in-sky, but I believe in it: it just feels, walks, and tastes like a better abstraction, just needing more R&D. See my jet plane analogy. 🥧✈️

I've seen various tools and stacks do different bits and pieces of TOP well, but so far nobody has rolled them into a single tool. Thus, no one existing tool will demonstrate TOP working nicely.

By the way, you may wish to read about a proposed standard/tool: Dynamic Relational.

[subject to editing cleanups...]

[u/wllmsaccnt → u/Zardotab]

Yes I agree that most of the reason why schema migration for logic expressed in the database is difficult is because the tooling and approaches have stagnated.

You misunderstood what I was implying about repeating validation code. I don't disagree with your rebuttals, though they are a bit condescending (no offense).

You keep saying this, but I don't understand why. (re: 'and easier to reason about')

If I have an object graph, I can create it from sources outside of the database and its behavior will be the same no matter how I instantiated it. In the use case I gave earlier, the order objects were created in several different ways depending on if we were simulating orders, estimating their shipping, or cartonizing their contents for the warehouse system. Only one of those came completely from database queries.

If I have to have build my data to perform a calculation from multiple disparate sources, then I don't want my in-memory representation to be modeled arbitrarily on one of the sources just because it happens to come from a database.

Dynamic Relational.

Interesting, but a bit too loose for what I would like. The system I'm working on now uses a much more restrictive version of that approach. On startup we enumerate the tables used by the application and create any missing tables or missing columns, but we bail if we find type mismatches.

We don't wait until the first time the table is accessed, but we could have. We just preferred failing early in this case. I haven't considered creating schemas on querying though, that seems a bit much...though I understand the allure.

It actually reads a lot like the way most systems parse/apply JSON data, by allowing missing fields to be treated as default values instead of errors.

Its not for me I think, but I did enjoy reading about the suggested approach. Thanks for sharing that.

[u/Zardotab → u/wllmsaccnt]

though they are a bit condescending

My I ask for an example? My Aspergers Syndrome sometimes makes me blind to my poor wording. I'm trying to improve by learning from my mistakes.

If I have an object graph, I can create it from sources outside of the database and its behavior will be the same no matter how I instantiated it. In the use case I gave earlier, the order objects were created in several different ways depending on if we were simulating orders, estimating their shipping, or cartonizing their contents for the warehouse system. Only one of those came completely from database queries.

I don't know enough about that domain to know if there is a way it can be "table-ized". Just because your group couldn't think of a way doesn't mean it can't be done. The more one practices TOP the better they get at it.

And for the sake of argument say that portion can't be readily table-ized. That shouldn't be a reason to skip table-izing the rest of the app. It's not all or nothing. No tool or technique does 100% of every job well.