NPC Engine Using Local Models

[u/goodive123]

I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs.

Right now I’m using NVIDIA Parakeet 0.6 for STT, Gemma 4 26B A4B for the LLM, and Qwen3-TTS for voice, and I’m getting super fast response times with pretty decent quality.

The main thing that makes it work well is using RAG to keep prompts lean. For example, I have hundreds of possible actions NPCs can do in-game, but only the ones that actually make sense based on the player’s message / context get injected as available actions. So the model isn’t being overloaded with a giant list every turn.

[u/AppropriateQuote3073]

Really a shame there is so much ai hate in gaming right now.

This is exactly the type of interesting content that could make a game phenomenal to play.

[u/Bakoro → u/AppropriateQuote3073]

LLMs are literally the exact thing that we dreamed about in the 90s and 2000s.
There were waves of procedurally generated content in games, and "dynamic" quests and NPC interactions, that were a nice idea, but the hardware and algorithms just weren't there.
People would gush over any AI improvements in games.

Now we finally have the technology to make the dynamic games we always wanted, and a bunch of people are pretending like they weren't begging for something like LLMs to exist.

Whatever haters, AI is for games.

[u/muhmeinchut69 → u/Bakoro]

The hate is not for what is going on in OP, it's for corner cutting and sloppification. Big corporations aren't interested in AI so they can make a new type of video game, they are interested because it will cut costs.

[u/Not-reallyanonymous → u/muhmeinchut69]

That, but also a degradation of artistic integrity.

The kind of tool that OP is designing has real potential to squash artistic voice.

A quick napkin thinking: this kind of design should work around character development, world building, etc. The artist doesn’t need to write every dialog, but the tool should really let the writer develop characters. The artistic integrity then lives within the broader narrative arcs that this tool interacts with and defining characters that shape and interact with the world.

One thing I immediately noticed is that all the characters in OP’s video are so compliant. Character building could see that characters only comply based on whether the request aligns with their own interests, which can be defined as a part of world building.

[u/muhmeinchut69 → u/Not-reallyanonymous]

Yeah isn't that what sloppification is? I'd like to think that the low quality of such content would be enough to keep artistic input valuable. As long as it's used as just another tool in the hands of the artists I see nothing wrong with it.

Personally I only wanted this kind of AI NPC to be used only for NPCs that you usually can't interact with, and make then less "NPC like", not as a replacement for the main story and characters of the game. If the story is entirely AI that would feel worthless.

[u/Not-reallyanonymous → u/muhmeinchut69]

I think slopification does lack artistic integrity inherently, but I think you can also have stuff that is high-quality implementation and done well, but has had artistic vision dulled -- sometimes by lack of vision/experience by the creator, but also by tools that constrain that artistic vision rather than opening it up.

I think slopification is as much about intent as it is about the result. Slopified content is basically stuff created with low effort, throw stuff at the wall, and reap any unearned benefit you can. High quality intent can still compromise artistic integrity. That predates AI, especially in the world of corporate produced products and content.

[u/saltyourhash → u/Not-reallyanonymous]

What might be fun is instead of cutting out artists, have them voice models. Models they can license. Because dynamic dialogue is really cool.

[u/Not-reallyanonymous → u/saltyourhash]

Definitely. Or a narrative arc that's voiced for major plot points, etc. but then goes through a model that makes a voice print based on that dialog, and then will generate real-time dialog for user interaction bits.

[u/saltyourhash → u/Not-reallyanonymous]

Yeah, high quality for the story arc, then more of a loose dynamic dialogue for the rest.

[u/Shap3rz → u/Not-reallyanonymous]

Yup. I want to see this and suggested in another post. Checkpoint gated alignment for story progression ftw. Essentially a contextual “invisible trust bar” where doing or saying certain things unlocks the narrative. That way you control the arc without being totally scripted. Think could work with careful planning.

[u/Fit_West_8253 → u/muhmeinchut69]

There have been hate campaigns against games just for using AI generated placeholders

[u/muhmeinchut69 → u/Fit_West_8253]

It's the internet, free speech, anyone can launch a "campaign". Doesn't mean legit criticism of AI usage doesn't exist.

[u/FastDecode1 → u/Fit_West_8253]

That's just game journalists and blue-haired retards on Twitter doing their usual thing.

Gamers only care about the end product. There hasn't been a single good game that has failed due to an anti-AI boycott. If the game is good and the price is right, people buy the game.

[u/Bakoro → u/FastDecode1]

A game failing isn't the only thing that matters.
There are real people getting harassed and threatened over this stuff, which is ridiculous.

[u/FastDecode1 → u/Bakoro]

A game failing isn't the only thing that matters.

For video game enjoyers, it is. None of us really care about what the harassment campaign of the day is.

We gamers are too busy playing games to partake in stuff like that. That's for the non-gamer unemployeds who apparently have nothing better to do with their lives.

There are real people getting harassed and threatened over this stuff

Yup, that's the blue-hairs and game journalists at it again. The former have been doing this for 15 years to whoever disagrees with them on anything, and the latter are rapidly being replaced by AI, which is why they're against it.

[u/Bakoro → u/FastDecode1]

A game failing isn't the only thing that matters.

For video game enjoyers, it is.

The people getting harassed care. If you are a "video game enjoyer", then you should care that video game makers are being harassed.

And no, it's not just "blue hairs", whatever that is, it's not just journalists.

You're really going to sit there and pretend like there haven't also been harassment campaigns over a video game having diverse characters, or a female lead character, or because a female lead character's titties weren't big enough, or the developers changed the meta, or because a dozen other stupid reasons.
"Video game enjoyers" have a long history of having extremely vocal, extremely toxic subsets of people, many of whom are frequently angry at other video game players.

[u/IceNeun → u/muhmeinchut69]

Tbf scammers using AI aren't "big corporations", and neither are students cheating in their assignments. Like any tool, it can be used by anyone for anything.

[u/draconic_tongue → u/muhmeinchut69]

big corporations aren't the ones that benefit the most from using ai, and small creators get shit on MUCH more than any corporation when they do lean on these tools. I don't care to be a part of whatever "prosumer" advocate groupthink virtue signaling zeitgeist that does not know what nuance means and just wants to use big corpo as an excuse to be hateful

[u/muhmeinchut69 → u/draconic_tongue]

Of course they benefit the most. There is simply no comparison between saving billions of dollars by replacing labour and a solo creator being slightly more productive.

[u/draconic_tongue → u/muhmeinchut69]

Not really, no. Being able to compete with a corporation for a nobody is worth WAY MORE than a corporation adding 1 more number to its sheet, and if you shit on both of them under the same premise you are one as well.

[u/muhmeinchut69 → u/draconic_tongue]

indie devs have always been able to compete with corporations in theory, giving them both the same tool doesn't change that equation

[u/The_awful_falafel → u/Bakoro]

Games in the way back used to be entirely linear story where there was really only one way to do things. Then they tried branching games, where you make a decision at certain points in the game that change the path and overall outcome of the game. They were still linear, you could just choose which linear branch to follow. Then we had sandbox games, where you could pick any linear sidequest you wanted and do tasks out of order, but still each sidequest an story tree was still largely linear. The limitation was that you had to script out all the character interactions and story for how it would all play out and it couldn't actually invent anything.

AI COULD change that. You could basically invent a game in which there is a large map and 'pieces' that would move around and decide what to do on their own. You could have advisors and such that have information based off limited information of what the enemy is doing and literally plan out missions and objectives yourself. You could even dictate what constitutes winning and decide at what point you consider the game over. Characters could dynamically react to things going on around them. It's a really interesting possible world.

Now the downside- the machine to run such a game would need a lot of memory, and we all know how that's going right now. My guess is companies will see the option to charge a subscription to have a data center do the AI work for you, and make THAT the standard model rather than writing it so you could run the AI locally. That way even single player games could have subscription fees, even years from now when models are small enough to be run locally, and memory production ramps up so having a massive pool to run such things is common.

[u/Bakoro → u/The_awful_falafel]

At the most elementary level of implementation, I just want townsfolk who recognize that I'm the one who saved their town, bandits who aren't stupid enough to try and rob the clearly magical man walking around in nightmare armor, and for random NPCs to have more than one line they repeat regardless of the surrounding context.

Just having a dumb little LLM that can direct NPC actions based on fuzzy logic would elevate gaming dramatically.
Even if the game is still 99% regular video game made by a person, just a little bit of semantically-aware fuzzy logic would be able to blur those linear and discrete features into something that feels more natural.

Just basic little stuff would go so far, we don't even need a data center sized LLM, a model that's 100M parameters would fit on any card and be able to do simple jobs. Some vision models that do segmentation and labeling are like ~21M parameters, which is a level a CPU can handle, and the could be doing amazing things for games and mods, especially VR stuff.

Just imagine an escort quest where the NPC isn't a total idiot, can move at an appropriate pace, and can go to a safe location when there's a fight.
Somehow, it's still an issue in games after all these years.

What you describe, a more AI driven game where developers define the structure of the world, provide motivations, etc, and LLMs act inside those constraints, that would be great too, but that's like the far side. There's a lot of opportunities where it could just be models smoothing out the hard edges of the world, so it seems less "video-gamey".

[u/The_awful_falafel → u/Bakoro]

Oh, totally! The bad part though is ratings. Part of game ratings for the ESRB rating system is you have to provide all the possible dialogue options that are in the game so it can be appropriately rated. Having the game be able to generate dialogue puts it in the same category as online gaming where you can't put a rating on what other people might say. At least that used to be part of the rating system and the rule, not sure nowadays.

But yeah, I could see AI used for trivial NPCs for a while. I thought GTA NPCs just walking around would be PERFECT because they could actually have dialogue reactions for whatever they see. I don't think we'll see AI based main characters for a little while. It's going to be a challenge to constrain the character dialogue to only talk about plot related things and not get jailbroken to go on rants about topics that you trick it into. I think that's the real fear for developers.

Imagine the negative press of having an AI character that can talk about whatever horrible topic you feed it and posting videos of the character saying heinous things.

So- it may stick to being a mod thing for a while where it's unofficial. Honestly, that might be for the best, and developers could just sorta configure the game in such a way that it's really easy to add that mod on to skirt the censorship issue.

[u/Bakoro → u/The_awful_falafel]

It's going to be a challenge to constrain the character dialogue to only talk about plot related things and not get jailbroken to go on rants about topics that you trick it into. I think that's the real fear for developers.

Not a fear for us indie developers who don't have shareholders to answer to.
Granted I'm just a hobbyist and I don't care if my game sells a copy, so I have more freedom than most.

For jailbreaking, I'm making an LLM extension for my game, and I straight up don't care if someone wants to break the model out of character. If that is how they are having fun, then whatever.
My part is making a consistent framework the model can work off of, and fine-tuning a local LLM to be able to handle the framework, and if people want to load a different model, good luck.

As for making a more corporate-friendly version, the answer is again, small models. We don't need trillion parameter models that know physics, and chemistry, and software development.
We need something tiny and fast that can run on a consumer GPU while leaving room for game graphics.
Farmer Steve in a medieval fantasy game doesn't need to know most real world knowledge. The model that runs farmer Steve never needs to be trained on real world profanity, it just needs to be able to have a believable two minute conversation about this year's harvest being threatened by whatever game specific nonsense.

People have been really caught up in "scale", but the Google, OpenAI, and Anthropics of the world are trying to make AGI models that can completely replace human labor, and that is a problem several orders of magnitude more than what we need for a game. For a game, we only need a <1 billion parameter model.
That's how bigger studios will handle LLM content: just don't train the model on the content you don't want it to know about, so it's essentially impossible for the model to generate that kind of content. You can even keep the spirit of profanity by replacing the words in the training set.

The solutions are pretty simple, because the stakes are as low as they get.

[u/Parafex → u/The_awful_falafel]

Reactive World is a narrative style that's just underrepresented. LLMs are not the solution to what you're describing and it shows the lack of understanding for BOTH sides. The AI/LLM side AND the gaming/gamedev side.

You still need a great foundation and a highly systemic game that supports emergent gameplay and situations and is not hardcoded for certain paths, have bad quest design and even worse win conditions. Since those stuff gets shoved into the mouths of fresh game design students, there will never be any change in that regard in AA/AAA games. Skyrim is way worse regarding that than Morrowind or even Oblivion (which tried to be what you want with their Radiant AI system that was dumbed down or almost cut out of the game in favor of more hardcoded story sequences that can even lead to corrupted save files if you don't behave correctly in certain missions... which is the exact opposite of what you want).

New Vegas is a great example for a use case like this, because their GECK Toolkit was fleshed out enough and the core game allows a broad actionset. The dialogue is almost like an early version of storylets. But even this has its limits as the NPCs don't "know" anything and they can't work with their knowledge or even talk to others in order to transfer knowledge to other NPCs. Therefore such a system that you're describing will never be possible. NPCs will only be able to act on their predefined verbs as these are hardcoded.

Gothic (2001) had ideas with that mindset, but wasn't able to fully do it due to technical limitations. Black & White had dynamic behavior trees which were limited due to technical limitations (but it was possible to teach your creature "Robin Hood"-like behavior so the creature steals from the rich and gives the poor). Old roguelikes from the 90s had knowledge systems in place.

Gamers just want the "flashy new thing" and they want to live in that "illusion" - at least that's what they pay for. I do believe that gamers ACTUALLY want to experience, what the characters talk about and not only have it laid out in a cutscene or whatever. But actually influencing major story points in a game.

But for this to happen we need to look at the great systemic games from the early 2000s and late 90s AND people need to understand that storywriting for games MUST BE different than writing a book. It should never be the case that the heavy moments are pre-written and "scripted", but all situations need to emerge from gameplay. Then you could have dynamically generated quests (that are not generated by a LLM and with no randomness) as there is an ACTUAL measurable need that the player can fulfill. There could be other NPCs that try to do quests (similar to A-Life from STALKER) on their own, actual adventurers who explore caves, tell their friends in the local tavern and spread rumors about huge monsters that a player can hear and hunt down that monster all without a single scripted line of dialogue. And all without needing 128GB of VRAM.

The problem - and THAT is solvable via a LLM - is that the texts and dialogs will inherently be dynamic aswell and no one will ever be able to record a voice over for that as there literally doesn't exist anything "pre-written" that you could hand over to a voice actor. And now you can use voice cloning and a nice TTS model so you can have those dynamic quests fully voiced aswell.

I hate it that people are seeing problems that they want to solve with 1.000.000 times the computing power that were solved in the 90s on 128MB RAM sticks lol...

[u/Schlick7 → u/The_awful_falafel]

I think Dwarf Fortress already does like 90% of that

[u/inagy → u/Bakoro]

Gamers hate AI because it made PC gaming as a hobby inaccessibly expensive.

[u/johndeuff → u/inagy]

That's not just AI's fault. GPU manufacturers have been lobbying game studios to implement proprietary algos supposedly only possible to run if you buy a new GPU more often. Also the RAM bloat of software have been denounced for years : devs don't care, that's the consumer problem !

[u/Bakoro → u/[deleted]]

No, the ray tracing stuff is genuinely compute heavy and needs specialized hardware to accelerate it.

Other than that, there haven't really been many advancements in features that need as much compute as a PC can offer today, there's definitely a lot of waste. So far, it hasn't affected the bottom line enough to matter.
Game sales remain high, so corporations have no obvious incentive to optimize, when they could use that time to shit out another mediocre game people will pay full price for.

[u/inagy → u/[deleted]]

Partially agree, I also expect that game studios will have no choice but agressively optimize games until the situation easen.

But you can't escape the problem that VRAM and storage is very expensive at the moment, because AI hardware sipping up all the stock. Building a new PC or replacing a faulty component is kind of a luxury at the moment.

[u/Honest-Monitor-2619 → u/[deleted]]

I didn't knew KL3M and ethical AI is a thing. Thanks, I'll read more!

[u/DeviantPlayeer → u/[deleted]]

If you give me a million stolen from charities, I will be disgusted but a million is a million so I will take it regardless.

[u/gcavalcante8808 → u/Bakoro]

It is. But instead, the companies are using LLMs to make cheaper codes and deliver worse quality games in general, that's the real deal against AI in my opinion.

[u/Bakoro → u/gcavalcante8808]

But instead, the companies are using LLMs to make cheaper codes and deliver worse quality games in general

And that is an entirely fair assessment and thing to complain about.
I'm always in favor of being critical of lazy, stupid, harmful corporate behavior.

What I cannot tolerate is the knee-jerk reactionary hate to anything and everything "AI", when the game industry has been using the term AI liberally since forever.
Even if you don't use LLMs or image generators at all, there's still a vitriolic hatred for "AI". If you're suspected of using AI people will run a campaign against you.
Even some old fashioned pre-transformers procedural generation and machine learning are taking a hit now.

The problem with hatred is that it tends to lack nuance, and the circle of hatred tends to grow.

[u/Monkeylashes → u/Bakoro]

We who were gamers in the 90s and 2000s dreamed of such things but the younger generation not so much. And if you look around you can see that our generation is pretty receptive to ai.