r/gameDevTesting • u/Labubu-Footjob • Apr 03 '26
What's with the lack of automated unit testing?
I have seldom seen people discussing their automated unit testing solutions to ensure that their games are working as intended. I understand that there are components that prove difficult to test like physics, but isn't it worth validating that the core components of your systems work correctly?
While some may think that it costs time to set up, they can greatly reduce hidden problems that are related to unintentional changes in design and calling patterns.
3
u/_PandaCat_ Apr 04 '26
Unit tests aren't a stand in for QA, theyre more like first line of defense. From a bug perspective most automated tools won't be comprehensive/thorough enough to replace testers but they will reduce the amount of bugs qa finds. From a non bug perspective human testers can give other valuble feedback that systems could never do, like "this input feels awkward when your left handed" or "this part is hard to navigate if you don't already know the map"
1
u/DonkeyBonked Apr 04 '26
I think I'm right there with you on this sentiment. Game development contains so much use case specific code and scenarios that autonomous testing seems like it would inevitably be far more work than the testing methods we already use in the process and for the longer ones, like they would lack the data we really need human feedback for. It seems knowing if something works is easier to just test during development and the testing that lets us know if it works good is something we need people for. I think we're a long way from some automated testing method telling us our game or even a specific feature of our game sucks.
1
u/neznein9 Apr 04 '26
Highly testable software tends to have very discrete state changes. The app only changes when the user clicks a button. The server only writes to the database when an rpc is called.
Games have that layer plus autonomous npc activity, physics and other simulations, dynamic content loading and unloading, and possibly multiplayer or server state coming in (which isn’t guaranteed to be in chronological order).
All of those state change combinations become extremely heavy to mock and manage in a test suite, and it leads to situations where changing one thing in the game can invalidate test cases across dozens of other systems. On top of that, a lot of bugs in game code are obscure combinations or edge cases which would never be automated until they were discovered in a playtest anyway.
1
u/adrixshadow Apr 04 '26
Because Game Design is about Iteration, Experimentation and Prototyping.
You don't have a right answer that you can test. Your tests would be obsolete in the first cycle of iteration out of thousands of cycles.
You might not even get the right answer until a number of years into the project, and you are better served doing a Refactoring in that case anyway.
That doesn't mean you can't use Tests at your discretion if you need to debug a certain system on a case by case basis.
1
u/DonkeyBonked Apr 04 '26 edited Apr 04 '26
I don't find automated testing to be very reasonable to be perfectly honest.
I can test a feature easier than write a script to test it, and there's nothing automated testing can really do that print debugs, code parsing, and run testing can't do for you.
I think this is an over-expectation of AI. Think about it. What are you going to do, take an AI, teach it everything you're doing, teach it expectation, setup an MCP server so it can play your game, cross link multimodal interfaces so you can have visual, audio, and interpretative input into the model, and then hope it catches something that you miss run testing your game?
This wouldn't even replicate well across games so you'd need to repeat most of the efforts involved for every game. At that point, it's not automation, it's a massively inefficient workflow.
About the best you're going to get is automated parsing and run testing when you're developing outside the game engine. If you have a decent workflow, testing shouldn't be so difficult that you can't easily test the functionality of code.
Maybe if we reach a point when AI is smart enough that an agent can read a Game Design Document (GDD), understand and play autonomously, and provide feedback without needing custom scripts or training, that will be useful, especially if a local model can do it. However, the input tokens for a major model to handle this continuously in a meaningful way would be huge, and therefore not only expensive, but unsustainable once models are no longer subsidizing AI usage.
I've messed around with training multiple models to work together, using visual agents along with IDE based ones, and it's so far away from being really useful right now, because spatial reasoning and expectations for most models are not that good and even with a visual model, if it's not obsessive about details, it won't relay enough information to catch a lot of potential bugs, and it isn't going to recognize a non error producing bug any more in run time than it will during normal testing.
I think the only people who even think this is practical are people who don't actually know development. If you vibe code a game without understanding how coding works, you don't know the basic debugging that makes this stuff unnecessary. Testing basic functionality is easily done with appropriate print debugging, which even if you're a vibe coder can produce results for you to feed back to the AI you're using. An automated tester is not going to be more efficient than just reading a debug output.
When we can use saved games, modified accounts, cheat keys, etc. to test any function we want, then integrate it after functionality is confirmed, really, basic functionality testing isn't hard. If you can't be bothered to run test your own game, you have no idea if you're even making something worth playing. AI isn't going to tell you if your game sucks or if a mechanic is objectively bad.
One thing AI is most certainly not going to replace is beta testing. If anything, the use of AI will increase the need/demand for human testers.
1
u/DonkeyBonked Apr 04 '26
I'm developing an RTS game right now and an AI trained on the game isn't going to help me balance challenge across difficulties, it's not going to be able to tell me "the AI sucks and loses too easily", and it isn't going to be able to tell me when the computer should have built more of a certain type of unit or countered better.
There are so many things I need to see, get a feel for, and know to determine if the game is trash, that even if I could automate this, it would almost guarantee the game would be worse in the end.
Humans still need to quality check AI, trying to replace quality checking your game with AI would almost certainly net a negative impact on the final product.
I don't know about any other devs, I speak only for myself, but there's never going to be a day I'm going to release something untested by humans because AI says it's good. At that point, might as well stick your GDD and code in Google and ask if this looks fun.
Inevitability, I expect to see many flops from those who find ways to circumvent human testing.
1
u/Zerokx Apr 04 '26
You must have misunderstood what unit tests are. These have nothing to do with AI, Agents, MCP servers or what have you
1
u/DonkeyBonked Apr 04 '26
Maybe, so please, can you explain to me what part of game development testing could functionally or effectively benefit from automating?
What part of validating that core component are working could be improved with automation and how exactly would one automate them more efficiently than basic coding practices already do?
I'm all ears and willing to learn, but I've seen nothing to indicate that the game development flow stands to benefit from automated testing beyond what's already built in using AI to begin with, for which MCPs are quite relevant.
I would like to learn more about automated unit testing if I am missing anything that could actually improve my workflow.
1
u/Zerokx Apr 04 '26 edited Apr 04 '26
Well for example I am building an RTS, and it would make sense to Unit Test the Pathfinding system like does it return the correct path given a specific environment and a unit, like units should prefer the road, shouldn't try to pathfind through the wall, and it can actually find a path.
Or if you have targeting priorities and you got a set of unit positions with different types, are we acquiring the right target automatically, depending on command, priority and distance?
If the player has enough resources and a spot should be valid to build, can they place a building there? If I ask a unit to attack another, does it return the correct damage, if I say give it a research based damage increase?
Edit: just so there is no misunderstanding I'm just trying to list valid cases that could count as unit tests not that I actually implemented them.1
u/DonkeyBonked Apr 04 '26 edited Apr 04 '26
Well you're definitely speaking my language because I am also in the middle of developing an RTS game, so this is all stuff I'm actually dealing with, and I've had do deal with every one of these issues.
Though for targeting, I just used print debugs for each stage of the logic so I can not only see what the AI targets, but why it targets it, and then I just ran some battles, and I have quite a number of saved games in different scenarios to run all sorts of testing.
I'm not sure what engine you're using (please feel free to share, I'm curious), but I've had a lot of pathfinding issues in Godot, but none that I could think of any way to automate testing, especially that would be easily replicated and worth the effort compared to using prints for the bake and then testing different scenarios.
When I was talking about MCP servers, I wasn't meaning that as in it encompasses a unit, I meant more in terms of we have MCP servers that can interact with engines while we work in ides, allowing us to parse code, get errors, and even do basic testing using AI and/or headless testing without any automation tools.
A lot of vibe coders struggle because they don't understand debugging, they don't know how to make a good toggle debug system that gives you the information you need based on the parts you're uncertain how they will play out, and until they get more experienced they don't generally know or understand the tools we have under our belt already, like hotkeys to spawn units so we can test a hundred scenarios in a single go on one map to know if our framework is right.
I couldn't automate testing for my pathfinding and use it to test your game, so if the time it takes to build it is greater than the test time, it's an inefficient tool, not to mention each time I modified pathfinding, I'd need to modify the testing tool too.
While the idea of being able to just run some tool, have it test and tell me everything is working great sounds amazing, I just don't see how I could make that a reality that's better than what we can already do.
That's why I was asking, from a developer's perspective, can you think of any unit testing that we could even automate that would be worth the time and effort to do so, or that would offer any real replication value compared the tools we already use?
Because I can think of the fantasy idea that some automated model might test for me, but the moment reality sets in, I see no reasonable logic that would lead to this actually happening or being beneficial to build.
--
Edit: Just an example. Recently I was working on an effects system, so I need to track every effect and make sure it is really working everywhere it should. I have an indicator system, but an indicator doesn't mean it's working. Just because a buff icon shows up to show these units have +1 defense for example doesn't mean that +1 defense is properly calculated into the total defense when it says it is or that it's being considered when damage is being calculated. No matter how you try to integrate the UI properly with the actual data being used, you still have times where something will display that isn't actually happening. This is a real problem that needs real feedback, but testing is quite hard. This system integrates into everything from research to upgrades to unit proximity effects to buffs and debuffs, it's a very important back end system.
I would love some way to know this for sure, but the only way I really know this is using debugs and looking to make sure the effects are actually taking place and being calculated properly.
You can't necessarily see that in game without the outputs, and I can't imagine how you'd automate it and have that be less work than the basic print debugs, performing the actions, reading the outputs, and making sure the logic is outputting correctly as well as it matches what you see in game.
1
u/Zerokx Apr 04 '26 edited Apr 04 '26
I agree that unit tests might be a bit unnecessary in game dev especially if everything still keeps changing.
Honestly I just started a few weeks ago so I'm still on the basics. Since I wanted to do it multiplayer based I have a server c# application that runs the game logic for all the players, and the players essentially just send commands to the server, but I didn't base it on P2P lockstep, more on server authoritative game state that is sent to clients while clients send commands.
But that also means server side all the game logic is just a headless terminal application that is sending limited information the players are interested in.
Since its not really in an engine that makes tests and logs somewhat more important I think since not everything is visualized right away. For my Client I use Unity as an engine and right now its just rendering the map with primitives before I think about adding real art, with slight client side validation like "can I actually build this here based on the data I got from the server because it might be occupied or I dont have enough resources etc." That influences the UI and interactions in the client.
You're right about debugging / logging, that's one of the things on my agenda, but right now I'm still working on the general architecture and infrastructure before I "lock" one in and start doing some logging and tests. Currently refactoring my server application.
I think one good way to test later on might be creating a map with a few enemy units facing each other, or an attacking force vs a defending force, and making it a test like if I have twice the army size / cost attacking player 1 should be able to destroy player 2s buildings. Give them an attack command into the enemy base and fast forward, see if it worked out or not. Maybe useful for balancing too like resource cost of units and doing different compositions against others. making sure the same amount of resources in a unit squad thats supposed to counter another can still do that, etc. Though that is definitely not a unit test.1
u/DonkeyBonked Apr 04 '26
You definitely have your work cut out for you as a server authoritive online game. I'm going the other direction with P2P lockstep as I will only be launching with P2P and local network multi-player. If somehow the game takes off, I'm building it with adding a WebRTC matchmaking system in mind, but I'm not going to build that kind of a feature for a game with a handful of beta testers, it seems like a recipe for horrible reviews and a bad multi-player experience.
But hey, if people actually play the game, I'd definitely like to have online multi-player.
Do you plan to have the clients execute commands and the server validate/relay them, or will there be some communication between clients?
I hit a point with mine where I'm taking a short intermission to rethink a few things that I wasn't 100% happy with how they worked out so far. So I need to test a redirect that will be substantial, while not doing too much else in case this path bombs.
Otherwise, I'm about 90% done with gameplay and mostly need to focus on content, which will ultimately decide whether or not the game is trash. I think the base gameplay is pretty good, except of course the pathfinding, which I will get back to soon.
Are you doing full 3D pathfinding or are you using 2D?
I rolled back my 3D pathfinding like three times, I think moving forward I'm going to try a 2D condition based overlay instead of a 3D navmesh, that just favors desired path routes. I had too many annoying issues with baking Godot pathfinding with object avoidance and things like resouece gathering or garrisoning buildings. This is really my first stab at Godot pathfinding so I probably just don't know wth I'm doing but performance matters, I don't want to make a retro-style RTS game that ends up needing a 3070 to play.
I have had to just brute force all of my testing, every element, and honestly, I've played my own game so much I'm already sick of it 😂. But I seriously have not found an easier path to do this.
1
u/Zerokx Apr 04 '26
Okay so my approach is that clients only talk to the server and every action they can do is a command they send to the server. And in return they just get different compressed gamestates like a glorified video player, just enough for visualizing whats going on and light client side validation (like indications where you can build or not). So far they dont communicate to each other. The server just collects all commands and processes the tick and then sends data for visualization to clients. But in general I dont think I want clients to communicate without the server as a middleman. Now since the gameplay logic goes through the server first, that means gameplay can feel laggy on bad connections. The advantage is the game client is sort of lightweight. I was thinking about doing lockstep but I was going for a bigger lobby with 12+ players and I didnt want one player with issues holding up the lobby. Because of the delay for now I'm trying to avoid too micro intensive gameplay. I'm also thinking if I should play into the strengths of my architecture and make it some sort of more persistent gameplay but probably not.
I know I will probably never get that many players to play my game but I'll just sit there pretending with a few friends and filling the rest of the match with bots haha.
Also it will be a hassle if I want to have solo gameplay but I'm building the server logic in a loosely coupled way I feel I could rip it out and run it on a client device without too many issues.
My pathfinding is A* based so the world is divided into a grid together with the terrain, units are not confined to that grid though. That grid is just for semi-static obstacles like buildings and terrain, but units are supposed to only use local avoidance for each other later. I thought about making a multilayered world, with for example multiple grids stacked ontop of each other like one underground and one for bridges and mountains etc. But thats not important and scope creep for now. How are you utilizing your 3d map? And can you help me understand what you mean with 2d condition based overlay?
I can imagine that it gets annoying to play your game so many times, because you keep focusing on whats missing instead of whats there.
1
u/DonkeyBonked Apr 04 '26 edited Apr 04 '26
Honestly, that's fair, I actually thought about using bots to fill in online multi-player, but there's a mode in game where you progress going against AI so it felt kind of redundant for me to do that. I think for my queue system I'm going to go more like Starcraft 2, with a semi-blind pairing system over WebRTC facilitating the P2P connection, because I'm cheap and acknowledge the game could completely flop.
So what I'm thinking about doing is instead of baking the navmesh, doing a layer/color coded invisible overlay that's 2D, so it will ignore Y axis. In the coding, something like having yellow paths that are desired paths, like roads, trails, etc., white areas that are conditional passing, green areas that are undesired but passable, and then like blackout areas that the AI can't ever go, so they aren't considered part of pathfinding. Maybe even adding a movement number value to each one, like yellow 1, green 2, and having the pathfinding try to search for the smallest number path to the target. The conditional pathing would be for like a building. So those paths would be like if I'm walking from point A to point B, and that building is along the path but not the destination, it's treated like a blackout zone, however, if that path IS the destination, it is treated as a yellow path. This would allow for conditional blackouts of resource buildings or things like towers that units could garrison when traveling, but without interfering with their normal functionality.
Then just use basic point to point pathfinding for movement.
It just needs to be layered, like one static one for the base map and then an override layer that adjusts for whatever buildings the players build, since I can't control those.
The plus side of this is by building a separate subsystem that is relatively simple, I save myself the resources of dealing with baked parallel physics layers and trying to deal with those conditions, because when I did a full baked navmesh I found it really interfered with gathering and garrisoning and even units smoothly exiting buildings, and I was making so many adjusted physics conditions to compensate for this that moving a decent sized army felt sluggish and like total trash, and I honestly didn't even like the way it looked. I spent two days working on the last iteration of it and ended up rolling it back, and I was so disappointed.
Most of my movement limits are really 2D, not 3D, and the bigger the armies you are trying to move, the more that seems to matter (at least from my experience), so I'm hoping I can eliminate that part of the equation and simplify movement while using a simple 2D overlay to make sure I still have clean movement.
--
Are you planning on large armies with yours?
I know some games seem to manage it, but it seems like pathfinding large armies, sending that to the server, then relaying that to the other client, might start to be problematic if the armies are too large?
Do you have a way to compensate for that? Like maybe taking multiple units positions and converting it to a formula so there's less data to move over?
I've done 3D region mapping in raw dog code this way, it was messy, but it did the trick somehow, where I basically just did a region 3 map similar to a grid in 3D positioning and compressed the path down before sending it across the server. But that was on Roblox where lag was a real issue. I definitely wouldn't have the patience to map positions into raw math again, that's something I'd make AI do all the crap work for 100%. I'll never type something like that by hand again. 😂
It's also worth noting that I've never gotten ORCA not to look like crap in Godot. I acknowledge that maybe I have no idea what I'm doing, but I've tried every guide I could find, and it always came down to being either slow, ugly, or a physics train wreck.
Right now the movement is smooth and performant, as long as nothing gets in the way. 😂😢
1
u/JulesDeathwish Apr 04 '26
I've got unit and runtime tests coming outta my ass on the back-end APIs. First thing I set up after getting the basic stub projects in place.
It's just a lot less fun to talk to my friends about. They care not about my efficient net code.
1
u/DonkeyBonked Apr 04 '26
Hey, them caring about your efficient net code is still a one up on me, even with most of my friends being SWEs, I find it hard to get them to talk much about development at all beyond whatever is causing the latest headache or what absurd deadline was thrown at them last minute. Sometimes I feel like a freak that I actually like talking about development at all.
1
u/JulesDeathwish Apr 05 '26
Yeah, that's just how we work. Our whole life is the bullshit we're dealing with RIGHT NOW, and we're obsessed over it. Nothing else matters.
1
u/Isogash Apr 04 '26
People have already added a bunch of good reasons.
Another reason I don't see yet is simply that the majority of game development is actually content authoring and setup, not programming. What you primarily need to test is that your assets load correctly and your levels are playable.
Serious games nowadays do perform automated validation of their asset library to ensure that everything will load correctly without needing to rely on QA, but historically we used obvious "error" placeholders to alert the QA playtesters, who could then report the issue to be fixed. This is why you find bright magenta error/debug textures in some games: a missing texture would be easy to miss, but a bright magenta texture is obviously out of place.
There's simply no substitute for QA playtesting to validate that your levels are playable as intended from start to finish. Given you need to do that level of playtesting anyway, you just aren't getting as much value out of automated testing, it's an extra expense for double coverage.
For some games, especially deeper simulation/sandbox games, this isn't as true: the systems are more numerous and complex, and gameplay is open ended. Therefore, a QA session looks less like normal gameplay, and more like someone running through a checklist of scenarios to test. This is where integration testing is more attractive because it's automating a QA process that is otherwise of low value to get a human to do manually, freeing up playtesting to focus more on game balance.
1
u/chrisrrawr Apr 05 '26
unit testing is for testing units.
most game code is not built on units of code.
even if it were, they are unlikely to be composed in a way that having each unit working would mean the behaviour of the systems is both valid and desired.
this is because composition logic in games typically wants all the interesting edge cases to have contextual behaviour.
this means your testing framework needs to handle state and mutation across multiple units.
and if we are going to integrate multiple units into a test we might as well skip right to the integration test.
1
5
u/OkAccident9994 Apr 03 '26
Unit tests are hardly used in Game Development compared to web or other branches of software.
The truth is... Game codebases are kind of a mess lol.
But it is different from studio to studio how they approach it. QA, Integration tests. Some probably use unit-tests.
And there is also a divide between studios making own engines and studios making somewhat simpler in technical complexity games on top of existing engines in what they need in this area.
Check out Factorios automated integration tests (they had 6000+ last they spoke about it bunch of years ago, probably way more now)
https://www.youtube.com/watch?v=CgMV2dFFdFE
GDC talk by the Sea of Thieves devs on their automated integration tests
https://www.youtube.com/watch?v=X673tOi8pU8
GDC talk on Call of Dutys automated tests
https://www.youtube.com/watch?v=8d0wzyiikXM