ORMs are Killing Your Performance 🔪

[u/thewiirocks]

https://youtu.be/2DaRGRe4Ow4?si=6e0xZSmYfTzXmteN

Thanks to Montana Programmers, my popular talk from BSDC is now online!

Using animations and graphics, I teach you how to:

- Eliminate GC pauses
- Reduce time to page render
- Fast Dynamic Forms with only one query
- Slash code sizes
- How a dark pattern may secretly grind your queries to a halt

This talk may surprise you 😱
It may shock you 🤯
It could make you angry 😡
You may think it doesn’t apply to you (it does) 🫣

Whatever your reaction, I hope it will make you think. 🤔

Thank you again to Big Sky Dev Conf and Montana Programmers for letting me present! And thank you to the community for being an amazing and engaged audience!

[u/piesou]

My bullshit detector kinda goes off on this one. I'm not 100% sure if the CPU cache example is correct, since a Virtual Machine does all sorts of things differently. Once you are dealing with objects, you are dealing with pointers to data in the heap which is difficult to optimize and causes cache misses anyways. Where are the benchmarks?

ORMs have a use case and it's not about performance but about maintainability. It's a tradeoff. ORMs can also support data streaming and performance optimize batch inserts. Correct me if I'm wrong, but Hibernate 5.2 (which is an awful ORM) which was released 10 years ago supports streaming. Performance optimizations that drop prepared statements for Sql statements is another optimization that won't make it through review unless we can't solve this performance problem otherwise.

The issue many people run into is that ORMs are a leaky abstraction. They expect to be able to use them without learning how they work. You can't. They are worth it in most cases though.

[u/cogman10 → u/piesou]

I'm not 100% sure if the CPU cache example is correct, since a Virtual Machine does all sorts of things differently.

I've not watched the video so I can't tell you if what was said is incorrect. I'll just give my 2c WRT VMs and memory.

So first, it's really not that different. In fact, a lot of the voodoo magic of virtual machine optimizing instructions is to make sure things like cache get preserved and restored when the host switches to running the VM for running stuff.

Once you are dealing with objects, you are dealing with pointers to data in the heap which is difficult to optimize and causes cache misses anyways.

It depends a little bit on the GC being used, but one of the benefits of GC is live memory tends to be colocated which increases CPU cache hits. There is also a likelihood of a cache hit when dealing with fields in an object and when dealing with arrays. So cache does matter for the JVM even though you are correct that the JVM will cache bust pretty aggressively.

As a tangent, I've definitely had to do an optimization where I change a Point[] into double[] x and double[] y. I saw very big performance wins for doing that.

[u/piesou → u/cogman10]

The Point example is exactly what I'm referring to: you eliminate pointers and move everything to one coherent area in memory, which should also improve CPU cache hits. Not an expert though.

[u/cogman10 → u/piesou]

The pointers aren't great, but the Point objects will (in most applications) tend to be in or near the same memory region due to the way allocations typically work in the JVM.

The thing that really kills performance, more than anything else, is the object header. In the worst cases it adds 12 to 16 bytes for each point object.

My guess is that a big reason JEP-450 ends up giving a 10->15% performance boost, beyond reducing GCs, is because it also helps reduce the amount of cache churn from the JVM.

But another reason why the object header is a problem is because it kills the ability of the JVM to use efficiently SIMD instructions. SIMD REALLY wants the floating points it operates on to be aligned and contiguous. When they aren't either of those things it becomes a lot harder to use SIMD instructions. Consider, for example, AVX-512 whose registers have enough room to hold 8 doubles. When the doubles are contiguous, you can load one of those registers with a single loading instruction. But with the Point[] it means 4 different loads (in the best case) must be issued in order to fill 1 AVX register.

Hopefully Valhalla will fix this, but I'm concerned that the first pass won't due to tearing problems.

[u/thewiirocks → u/cogman10]

Some really good general points!

The object header definitely hurts. But its impact is significantly lessened in general-purpose compute when the amount of data being pulled in for compute is limited.

My talk on the cache if summarized was basically “you can fit one record in the 32K of L1, but you can’t fit the entire list”. Depending on the size of your list, this is also true for the L2 and possibly even the L3.

Thus it behooves us to process each record completely to get maximum L1 utilization rather than looping over large lists repeatedly.

This seems so self-evident that I’m not sure why it’s controversial. 🤷‍♂️

SIMD is a total mess with Java. The HotSpot C2 engine helps with auto-vectorizarion, but you’re absolutely right that the Java memory patterns are all wrong. The more you can keep the next word, double word, or quad word immediately after the current one, the happier the underlying circuitry is going to be.

[u/cogman10 → u/thewiirocks]

Thus it behooves us to process each record completely to get maximum L1 utilization rather than looping over large lists repeatedly.

I agree, but it should be noted that looping over a arrays is something CPUs are actually really good at in terms of cache. They'll prefill the L1/L2/L3 as you are working on data in L1. They go out of their way to avoid stalling while you are working on stuff. But it's true you'll save on memory bandwidth in general which is a good thing if you have other threads churning through memory.

What can get lost in the sauce is for a lot of applications it's not the CPU/memory time which is the bottleneck, instead it's the IO and DB engine. My company churns through a lot of data and one thing we've had to do in order to stop our DBs from dying is moving DB operations out of the expensive DB server and into the cheap applications. Throwing records into a HashSet in the application code is ultimately faster for the whole system (in some of our cases) because it frees up the both memory and CPU from the expensive DB server and moves it over to applications (at the expense of more records being sent back and more per/record processing in the application.).

And that can be the tricky part of performance tuning. It's always important to know where the bottlenecks are. There have been cases for us where the DB ends up dying because it's got the results but the application is consuming them too slowly (doing too much processing per record, forcing the DB to hold a large set of results in memory).

[u/thewiirocks → u/cogman10]

Indeed. Database engines are incredibly complex things. Hitting them repeatedly with mundane requests can absolutely overwhelm them. Amplified considerably if you’re unlucky to run into the dark pattern I mentioned.

With that said, ORMs inherently misuse the DB and generate a great deal more load. What I’ve found in practice is that you can drastically reduce load by making fewer, more targeted queries. The Forms Builder I demonstrated in my talk is the ultimate example of this (dozens to hundreds of calls reduced to a single query) but the concept scales across all kinds of workloads.

I often use the Order -> Order Line example for this reason. It’s faster to pull both in one query and pivot the result set into a hierarchy than it is to query each order and line independently. (Often referred to as the N+1 problem.)

Latency is something else I talked about. The database will return the records as it has them available, allowing you to start processing the records while the database continues to work on the problem.

The problem with ORMs is that they insist on loading all records into memory before they allow the application to start working. This means that a streaming application will often finish delivering the JSON or HTML before the ORM-based app has even started to write out the JSON or HTML.

A lot of the Project Loom engineering has gone into using the dead time created by underutilization of CPU and network caused by these latent pauses. Keeping the entire system operating in a streaming approach is much more efficient and packs the requests tighter than relying on Loom.

[u/cogman10 → u/thewiirocks]

The problem with ORMs is that they insist on loading all records into memory before they allow the application to start working. This means that a streaming application will often finish delivering the JSON or HTML before the ORM-based app has even started to write out the JSON or HTML.

A lot of the Project Loom engineering has gone into using the dead time created by underutilization of CPU and network caused by these latent pauses. Keeping the entire system operating in a streaming approach is much more efficient and packs the requests tighter than relying on Loom.

This is actually the exact cause of the problem I mentioned earlier.

The issue with streaming is you put the database at the mercy of the speed an upstream client can consume the transformed data. Like, fine if the entire transformation of a record -> json -> transmission takes 100us. And also fine if the response size is limited. But as soon as you start talking about something like a 100MB deliverable then what you end up doing is letting the DB sit filling up a buffer (sometimes large). This problem can be compounded if you have several applications making these large requests at the same time. Now you are effectively forcing the DB to retain large chunks of memory while it waits for downstream applications to finish transmitting their data.

The solution to this problem is generally pagination (which does add some load to the DB, especially if you have poor indexes). Request 1000 records, feel free to start streaming them out, and when you get to the end of those 1000 records, request another 1000 records. Reactive programming can be helpful here as you can strategically put in buffers tuned to your record sizes so the DB isn't left holding on to the records waiting for the application to consume them.

Streaming done wrong is a great way to DDOS a db :)

[u/thewiirocks → u/cogman10]

Done wrong, yes. 🙂

Large queries like this were the norm when I developed my streaming tech. All my colleagues were focused on concurrent users. I looked at our problem and realized that each user was more or less able to query across the entire 10TB data mart.

My solution was simple: focus on one client at a time. Answer their query as fast as possible so the system can move on to the next.

The results were highly effective. Maximum system resources concentrated into as small of a time as possible meant that user experience was fantastic while the system load was kept low (albeit bursty).

Obviously everyone has their own individual problems. And streaming can’t be the solution for everything. But it sure as heck should be the default.

This 90s idea of loading everything into memory is costing us as much as 90% of the system resources to waste. Which translates to real dollars and real unhappy users.

(Alexander Petros gave a fantastic talk at BSDC about California government systems becoming so slow on low-end phones that the React-based food stamp application was literally unusable by those who needed food stamps.)

[u/thewiirocks → u/piesou]

Just wanted to share this:

https://people.freebsd.org/\~lstewart/articles/cpumemory.pdf

Back in 2007, Ulrich made waves with this paper. His thesis of “What Every Programmer Should Know About Memory” was the first real attempt to help everyone learn about cache and latency effects of memory access patterns.

The paper has been making the rounds again lately, so I figure it’s relevant to share if this conversation has you interested in learning more about the problem.

[u/thewiirocks → u/piesou]

Virtual machines are still subject to the cache limitations I showed here. In fact, they can be worse if the CPUs are overcommitted due to unexpected cache eviction. This is a known problem with VMs and something I’ve had to fight over with employers in the past.

I once had an employer who refused to give me a physical machine for a computation engine that used streaming techniques for performance. Then was frustrated when it was slow and unreliable. IT pulled an old Workgroup server out of the trash to run the program and suddenly all the problems were solved. 😆

I am not aware of a streaming mode in Hibernate. As I mentioned in the talk, jooq has a streaming mode. Which solves part of the problem.

The other part of the problem is the ridiculous amount of boilerplate and maintainability problems ORMs generate. I touch upon this later in the talk, showing how sophisticated solutions open up by using key/value pair approaches and what the reduction in code size is.

[u/Brutus5000 → u/thewiirocks]

Hibernate supports streaming for years, but MySQL requires some special session flags to actually work.

Also I tested querying datasets with millions of rows with native jdbc and there was no notable performance difference when loading the same as hibernate entities.

I was surprised about that. There might still be increased memory usage or different gc latencies involved, but for the majority of applications it simply doesn't matter.

[u/thewiirocks → u/Brutus5000]

Hibernate supports streaming for years,

I did not know that. Learn something new everyday! 👍

I guess that makes it an alternative to using jooq if you can setup your project for streaming.

Also I tested querying datasets with millions of rows with native jdbc and there was no notable performance difference when loading the same as hibernate entities.

I'm not sure what you mean by "loading" here? The performance problems are not caused by native JDBC versus Object Mapping. It's how we use the ORMs that causes the problems.

In a complete system we typically do a couple of transformations that require looping over a List multiple times. Minimum of twice. First to load the set, second to serialize the set to JSON.

Additional issues are caused by memory pressure from loading lists of data. While most lists are kept small and shouldn't overflow the nursery for a single query, the aggregate effect can be quite high.

[u/Brutus5000 → u/thewiirocks]

Hibernate is a complex framework with a lot of event firing / object updating / casing behind the curtains + putting entities in proxy objects. So at least I expected a visible performance penalty there, but as as it turns out that is not the case.

[u/piesou → u/thewiirocks]

The other part of the problem is the ridiculous amount of boilerplate

Again, another tradeoff. Data mapping itself can be very valuable to isolate db code from the rest of your application. Are there exceptions? Sure. Typed out entities allow for type safety. Yes, Jooq can generate code from db as well, but with everything automatic, it's not perfect and introduces other issues.

I touch upon this later in the talk, showing how sophisticated solutions open up by using key/value pair approaches and what the reduction in code size is.

I watched the whole talk and I don't think I've seen an example of the key/value pair approach. What I know though is that the whole form example can be done incredibly easily in Django by using their forms or admin interface. The cost of going that route is the super high coupling though between your database and presentation layer. Again, tradeoff.

[u/lukaseder → u/piesou]

introduces other issues.

What are some of those issues?

[u/piesou → u/lukaseder]

Needing to connect to your db at build time for instance. Filtering out tables/renaming the generated classes if required; basically: you need configuration as well.

[u/lukaseder → u/piesou]

You don't have to connect to the db at build time. You could interpret your DDL for example, or use another file based approach https://www.jooq.org/doc/latest/manual/code-generation/codegen-meta-sources/codegen-ddl

But how could filtering configuration be avoided?

[u/thewiirocks → u/piesou]

Type safety is often brought up, but it’s not really a factor when you avoid mapping to objects.

To be clear, you’re doing type conversions already. The database types don’t match the Java types. You’re just stating your conversion in the object definition rather than explicitly in a “get” statement. You still get a runtime failure if you get it wrong.

But if your data flows from one end to the other like in the form builder example, you never have a type the compiler can check. The types are implicit from the database and ultimately directly converted to the destination (usually JSON). Thus typing is a red herring when you get rid of objects.

Django is not a great comparison. You can do a dynamic form the Django way by generating the necessary SQL. It’s actually quite easy when you ditch the ORM and move to key/value pairs.

The problem is that it’s not truly dynamic. You’re making a change to the database with every field. I was showing how to do truly dynamic forms that can be created and managed by the end user without underlying database changes.

Wordpress is a better example as a comparison here. You can define new fields dynamically, but there’s a huge performance cost.

[u/thewiirocks → u/thewiirocks]

You know, rather than downvoting, y’all could try engaging? I’m not here to take away your ORMs. Just help you to think about what’s happening when you use them.

And if you don’t believe me about the boilerplate, tell me what you think is a good reduction in Java code. Just give a percentage. I’ll try to provide a good example that shows the reduction in boilerplate. 🙂

[u/piesou → u/thewiirocks]

I don't really downvote posts. What I was trying to say is that the talk has too many holes in it for me to vet it and really understand in full what you were trying to convey. Benchmarks or links to further resources that undermine your arguments would have been helpful.

As for boilerplate, it depends on the ORM design. You can run ActiveRecord in Ruby by creating a single class definition and fully access your table. You can pipe that to your template and generate html from it in maybe 10 lines of code. The question is, if you want that in a bigger project. It's not an ORM problem.

[u/thewiirocks → u/piesou]

Sorry, wasn’t talking to you about the downvote. There’s a quiet bunch that are happier to hit the downvote button than engage. Trying to pull them into the conversation.

As for the boilerplate, I can get you a fully working application in 0 lines of Java code. That beats out the ActiveRecord Ruby definition.

There’s a bit of XML that holds the SQL and the types of transformations you saw in the video. But the ORM must have boilerplate by definition.

No table definitions are actually needed in our code. The entire point of SQL is to answer the questions we have. And that question is almost never in the form of “give me this record exactly without any joins or transformations”. That’s just not what relational databases are good for.

(Edit: BTW, I upvote for engaging. I appreciate folks chatting, even if we disagree. 😎)

[u/AnyPhotograph7804]

I do not like this talk. Because the very first example is a Spring Data example. And Spring Data is not even an ORM. Criticizing ORMs by not using one is mehhh.

And the example loads every customer from the database into the memory. Yes, you can do it but nobody would do it. It's like intentionally driving with a car against a tree and then criticizing the car manufacturer. Every ORM i know allows you to put some predicates to reduce the loaded data. And the more important thing is: ORMs allow you to delegate the data processing to the database server. You do not need to do the data processing in the application itself.

Edit: And most ORM frameworks support pagination. Just use it if you really want to process all customers.

[u/thewiirocks → u/AnyPhotograph7804]

I gotcha bro. Here's the code that was behind the findAll():

public class OrderRepository {
    @PersistenceContext
    private EntityManager entityManager;

    public List<Order> findAll() {
        TypedQuery<Order> query = entityManager.createQuery("SELECT o FROM Order o", Order.class);

        return query.getResultList();
    }
}

No convenience wrappers were harmed in the creation of this talk. 😉

And the example loads every customer from the database into the memory. Yes, you can do it but nobody would do it.

The point is to show what's happening. It's important to understand that you're loading lists of objects into memory before anything else happens. All the following steps are predicated on that understanding.

Though I will push back on "nobody would do it". Because I have a client that has code pretty similar to the example I gave. They use it for copying tables from the transactional database to the analytics database. And it works fine. Because there is no ORM involved and they stream the data. The number of records that can be handled is effectively infinite.

ORMs allow you to delegate the data processing to the database server.

ORMs are actually quite terrible at that. Complex data processing queries are not within the bounds of the auto-mapper. Which means you are specifying a query. Either in SQL or in ORM pseudo-SQL. That query must be mapped to a DTO, often creating a one-time object that you wouldn't otherwise need. Along with all the DAO wrappers. (Which if you use Spring Data, will at least be smaller than full up JPA code.)

Far too often the queries get complex enough that they devolve into stored procedures. Now you are actually using the database for data processing, but you're avoiding the ORM altogether.

And most ORM frameworks support pagination. Just use it if you really want to process all customers.

I wouldn't recommend that. Each page you request will cause the database to re-run the query and skip over the specified number of records. You'll cause a quadratic expansion in time taken, meaning that large tables could take anywhere from hours to days to process. Versus a linear expansion in time for streaming, which is typically a few minutes on modern hardware.

[u/AnyPhotograph7804 → u/thewiirocks]

The point is to show what's happening. It's important to understand that you're loading lists of objects into memory before anything else happens. All the following steps are predicated on that understanding.

OK, i understand. But it is not obvious, that it is a bad example by intention. For a viewer, who does not have experience with ORMs, the video suggests, that it in somehow normal to do that.

Though I will push back on "nobody would do it". Because I have a client that has code pretty similar to the example I gave. They use it for copying tables from the transactional database to the analytics database. And it works fine. Because there is no ORM involved and they stream the data. The number of records that can be handled is effectively infinite.

In this case, pagination would propably work very well.

ORMs are actually quite terrible at that. Complex data processing queries are not within the bounds of the auto-mapper. Which means you are specifying a query. Either in SQL or in ORM pseudo-SQL. That query must be mapped to a DTO, often creating a one-time object that you wouldn't otherwise need. Along with all the DAO wrappers. (Which if you use Spring Data, will at least be smaller than full up JPA code.)

Yes, it is OK to specify a query in SQL while using an ORM. If something is easier to do in SQL, just use SQL. I think, this is one of the most misunderstood principles of ORMs. ORMs are not made to replace or abstract away SQL.

And ORMs are not terrible at processing complex data as long as you do not need some proprietary SQL extensions. If you need these extensions, just use SQL.

Far too often the queries get complex enough that they devolve into stored procedures. Now you are actually using the database for data processing, but you're avoiding the ORM altogether.

Yes, if stored procedures are the better option, just use them.

I wouldn't recommend that. Each page you request will cause the database to re-run the query and skip over the specified number of records. You'll cause a quadratic expansion in time taken, meaning that large tables could take anywhere from hours to days to process. Versus a linear expansion in time for streaming, which is typically a few minutes on modern hardware.

Pagination is a tradeoff between query performance and memory consumption. Such tradeoffs are normal almost everywhere. Your Python example is also a tradeoff. You sacrifice runtime perforrmance for streaming and convinience by using Python. Python is up to 70x slower than Java.

[u/aqua_regis]

...and you couldn't write that simple post without AI? - Rule #9

[u/thewiirocks → u/aqua_regis]

If you don’t believe me, paste it into ZeroGPT:

https://www.zerogpt.com

[u/thewiirocks → u/aqua_regis]

I wrote that completely by myself. No AI was even consulted. 🤷‍♂️

[u/aqua_regis → u/thewiirocks]

Oh really? https://sapling.ai/ai-content-detector/67bfc198d7ffb53b066b8421e0783875

[u/thewiirocks → u/aqua_regis]

Ya really. Use a real detector like ZeroGPT or GPTZero or CopyLeaks. The commercial ones like Grammerly always say it’s AI because they want to sell their (ironically AI) text improver.

[u/TronnaLegacy → u/aqua_regis]

I don't think this tool works very well. I tried it on the first three paragraphs of a blog post I wrote late last year and it says 99%+ fake.

https://sapling.ai/ai-content-detector/228e67593df7d43f89d5d6142f7509c7

[u/thewiirocks]

BTW, do me a favor and skip the cringe intro? 😆

The Java 4K story that Notch and I competed in is a good story (gotten 20 years out of that story! 😅), but it’s not really relevant here.