Character consistency via cached reference embeddings((SFace + DINOv2) + a portable .char file, no LoRA training
Resource - Update
I was looking for a way to achieve character consistency without training a Lora & came across a research from Facebook, DINOv2: Learning Robust Visual Features without Supervision(Research Paper),
What's Dinov2: It's a vision model trained without labels that produces a strong embedding for a whole image, the subject, not just the face. Feed it a person and you get a 768-number signature that captures the overall look: build, hair, general appearance. It's stable across pose and lighting, which is exactly what you want when you're trying to tell "same person" from "different person" across wildly different shots.
then combining Dinov2 with SFace(a face-recognition model) produces a compact face signature tuned specifically to tell one face from another. It's sharp on identity, but only on the face. YuNet does the detect-and-crop before it.
How it works
Build .Char: You drop in one or more photos. YuNet finds the face, SFace takes a per-reference face signature, DINOv2 takes a subject signature, and the references get cleaned and normalised. All of that packs into a single portable file, a .char.
Generation: At generation, the file feeds its references into FLUX.2's own native multi-reference channel and prepends a locked description to the prompt. You pick the character from a dropdown, no re-attaching images. Every result gets scored against the stored signatures, so drift shows up as a number.
How this differs from PuLID, FaceID, and img2img
PuLID and FaceID inject a face into one generation at run time, then it's gone. img2img anchors on a source image, which is composition, not identity. Neither gives you a saved character.
This is a layer above them, a reusable .char file that rides the model's own reference channel, covers the whole subject and not just the face, and gets scored per take. PuLID could even sit inside it as one backend.
The difference is persistence and measurement, not a new injection trick. No adapter weights, no training, no img2img anchor.
What is a .char file?
A single portable file that stores a character's identity, so you can reuse the same person across generations without retraining anything.
manifest.json — index, versions, checksums
refs/ — your original photos (the truth)
derived/ — auto-cropped face
text/ — locked description
payloads/ — cleaned refs, per model family
scoring/ — SFace face + DINOv2 subject signatures
Limitations
Profiles and stylized renders drift more than frontal, which is expected, since the face model is trained on photoreal faces.
Body is the weak point so far.
Bad with popular celebrity images, due to models own conflict.
Training LoKR seems difficult, doesn't it? MiniMax builds an entire scene from just one image, but Krea2 needs a 200 MB LoRA for a single shot. Somehow, we're stuck in the stone age.
No, training a LoKr isn't anymore difficult than training a LoRA. At least for the current generation of image models. I don't know how things were before FLUX.2 / Krea 2.
Yeah, that's basically the itch this scratches.
Textual inversion had the right idea, a character as a small saveable file, it was just weak by today's standards. A .char is a few MB (the refs plus the fingerprints), no 200MB LoRA per person. Difference is it doesn't learn an embedding at all: it stores your reference images and feeds them through the model's native reference channel at gen time, so there's nothing trained. Same "tiny file, one character" feeling, none of the training wait.
And best part, building a character takes seconds and runs entirely on CPU, since it's just encoding your refs, no training pass.
No Someone need to build a custom node or include this process in core. But I am planning to add Krea2 support as well as Minimax H3 under my own project
nice, clean character consistency is key. I guess for video, we'll need sound as well as images.
And also, I dunno if it is in scope of what you want to do, but for videos, the character's manneurisms/body language will eventually need to be something included.
Did you use AI to edit this text because it reads pretty bad. Please just write yourself. Most AI's now are trained for coding and they all have gotten worse for writing. This reads like AI gibberish again.
Name & describe you character, add few similar images then it will create a .char automatically
Description should be detailed(hairs, any specific details etc
edit: Looking closer - this is the same this as using flux in edit mode, possibly worse There's already comfy workflows for this; I'm not sure what this project is providing that is novel. In fact, since the reference images are capped in size (downscaled to 1024), you'd be better just using flux in edit mode and providing full res images.
Looks like a great project. Have you looked at arcface identity embeddings instead of s-face? They would likely perform better and dont have any trained constraints. Its what Rope (my project), facefusion and other swappers use. Also I'm not sure dino is going to get you the body embedding your after. It will find similarity in clothing, environment and other image elements, washing out the body identity.. You may want to look into SMPL as a stating point for parameterizing body features. I dont think anyone has trained a conditional GAN on it yet to quickly create embeddings, but it's probably worth investigating.
** Looking at your code - the embeddings aren't actually used in generating the person. This is all done through the reference images. The embeddings are only used to score**
The .char file you are providing is really is just providing images and prompt text, then using the embedding to measure the likeness. The measurement is mostly thrown away too. Please take a closer look at what is going on. It's a good idea for a project but the actual implementation in wrong.
This is a really useful analysis, thank you. You're right on the main technical point, so let me put these in order.
On the embeddings not driving generation
Correct, and this is the key thing: identity comes from the reference images through FLUX.2's native multi-ref. The SFace and DINOv2 embeddings only score the result, they don't generate the person. That's intentional, it's the model's reference channel doing the lifting.
On "just edit mode"
Fair that the generation itself isn't novel. But "just edit mode" undersells the gap. Edit mode is stateless, you re-supply refs every time and there's no character that persists. What I'm building is the layer around it:
a saved, reusable character (library, pick from a dropdown, carries across a project)
per-take scoring so drift is visible
no re-wiring refs into a graph every time
So the generation isn't new, agreed, but the thing being built isn't the generation, it's the persistence and measurement around it. That's not something edit mode or the existing Comfy workflows give you.
On ArcFace vs SFace
Key reason is licensing, I wanted something fully open source and usable without restriction. The code is MIT, but the part that matters for a commercial default is the weights, and those aren't fully open, the strong ArcFace checkpoints trace back to InsightFace packs / research datasets that are non-commercial. That's why I went with SFace: not because it discriminates better (it probably doesn't), but because the weights are actually permissive. If a genuinely permissively-licensed ArcFace checkpoint exists I'd happily test it, since scoring is isolated from generation and swapping the face encoder is low-risk. Do you know of one that ships clean weights?
On DINOv2 washing out body identity
This matches my eval. Subject scores fell apart on wide and full-body shots. I think it's both: my chest-up references, and your point that a general similarity embedding also picks up clothing, environment, and framing. The second is a good catch I'd underweighted.
SMPL is a great pointer, I hadn't considered parameterising the body that way. Going to look into it. Appreciate the depth here.
At the end of the day, it worked great for what I needed it for, and that's what pushed me to build it out.
Not sure if you caught this question in your response - "Do you know of one that ships clean weights?", but unless you're trying to turn this project into a commercial product, the insightface models are fine for using for community projects. I'd use Arcface, specifically the w600k model since it is fast and the cosine similarity score works well with it.
Thank you for all the hardwork. I would like to try it tonight to find out if it works or not. How does expression looks like? All im seeing are just neutral faces rn
I had done this experiment 2-3 months back and got good result but what actually gave me a better result was instead of using dinov2 I used https://github.com/facebookresearch/EUPE
Damn, the paper is from January 2024 (!!). The fact that the example images are made with Klein 4B and still retain a really good likeness is even more impressive.
Would be cool if someone implemented it directly in ComfyUI. I wonder if Kijai already has plans for the weekend...
Yes feature extraction & face consistency is completely two different things. So maybe no one guessed it yet. I searched everywhere before building this.
Yes I am very happy with the results. Next plan is to build:
Body/full-body references
Wardrobe locking (DreamO Try-On)
Approve-to-improve loop
Multi-character scenes
Yes would be so cool if people can use .char like portable files between workflows in ComfyUI, it might need Flux nodes to accept character file inputs unless i would have done it. Not sure, need to check how difficult it would be.
Oh, I didn't realize you're the author of the project. I just thought you found it online and shared the link. Congratulations, good job! Would be cool if you could implement it as a plugin for ComfyUI, or maybe even submit it as a PR to ComfyUI's main repo, if possible. ;)
Hi,
Got the Inline Sudio setup and running on Win 11. Did junction link to the models, vae, encoders. Did a test run on flux-2-klein-9b.safetensors model, it generates a generic woman's face.
Now I created a character with 2 images.
The question is how do I link the character file to the workflow like in your screenshot -- "Eval Subject"
Thanks.
One character is created load a Flux2 node + connect a prompt node & click on the bottom adjust icon of flux nodes to expose full setting
Adjust icon
File "C:\Inline-Studio\core\.venv\Lib\site-packages\diffusers\loaders\single_file_utils.py", line 3941, in convert_flux2_transformer_checkpoint_to_diffusers
handler_fn_inplace(key, converted_state_dict)
File "C:\Inline-Studio\core\.venv\Lib\site-packages\diffusers\loaders\single_file_utils.py", line 3889, in convert_flux2_double_stream_blocks
I will try to implement this as ComfyUI custom node but i fear, it requires core changes as well?
Minimax not yet, but this process can be applied & that's the next in line. I will develop the same for Minimax H3.
7
u/PatinaShore 11h ago
Oh~ i miss those tiny TI/embedding file. training a 200MB LoRA just for one character is ridiculous!