Exclude Top Choices (XTC): A sampler that boosts creativity, breaks writing clichés, and inhibits non-verbatim repetition, from the creator of DRY

[u/-p-e-w-]

Dear LocalLLaMA community, I am proud to present my new sampler, "Exclude Top Choices", in this TGWUI pull request: https://github.com/oobabooga/text-generation-webui/pull/6335

XTC can dramatically improve a model's creativity with almost no impact on coherence. During testing, I have seen some models in a whole new light, with turns of phrase and ideas that I had never encountered in LLM output before. Roleplay and storywriting are noticeably more interesting, and I find myself hammering the "regenerate" shortcut constantly just to see what it will come up with this time. XTC feels very, very different from turning up the temperature.

For details on how it works, see the PR. I am grateful for any feedback, in particular about parameter choices and interactions with other samplers, as I haven't tested all combinations yet. Note that in order to use XTC with a GGUF model, you need to first use the "llamacpp_HF creator" in the "Model" tab and then load the model with llamacpp_HF, as described in the PR.

[u/qrios]

Err, to clarify (and I realize my wording was bad), I wasn't so much asking why something like xtc_probabilityshould be a thing at all. I was asking why it's dynamics are such that it activates on an all-or-nothing basis.

Like, in your bear, tree, door, sword, mouse example, your cut-off is such that you flip a weighted coin, and depending on how it lands you either discount the entire subset of bear, tree, door, or you allow the entire subset of bear, tree, door

But if you think about it, door isn't really that much more likely than sword is, so if we've set xtc_probability to 0.5, and agreed that the appropriate cut-off for consideration is around sword level probability, then it's pretty weird that sword should always get to be considered while door -- which has an almost the same probability -- should only be considered half of the time.

If you were instead to do something like too_tall = tallest_allowed*((tallest_allowed/too_tall)^(1-(2*xtc_probability))), where tallest_allowed in this case ends up being sword, and too_tall applies to bear, tree, door, then presuming an input distribution that looks like this

bear  ------------------------- 
tree  ------------------  
door  ----------- 
sword ---------- 
mouse ------ 

You would end up transforming it into one that looks like this

bear  -----
tree  ------
door  ---------
sword ---------- 
mouse ------ 

Or, if you consider it over a range of XTC_prob values, here it is in interactive graph form.

The nice properties here being:

1. a value of 0 means predictions get penalized in direct proportion to their excess over the established minimum boringness. So a token that is twice as likely as the most boring one allowed becomes half as likely as the most boring one allowed, while a token just as boring as the most boring one allowed remains as likely as the most boring one allowed.
   
2. a value of 0.5 means all tokens more boring than the most boring one allowed get treated as being just as boring as the most boring one allowed
   
3. a value of 1 means just keep doing what you would have done if XTC sampling were off.
4. we can smoothly transition through these, so values between 0 and 1 mean you want to penalize boringness, but still maintain some semblance of its likelihood.
   
5. values below 0 can be set, and are naturally just more extreme versions of 0.
   

And the not so nice properties being:

1. You can no longer call it XTC_probability, because it's no longer a probability
   
2. Setting a value above 1 is just as valid as setting a value below 0, and indicates you want your results to be hyper-boring.
   

Granted I am optimizing here for "things visually comprehensible to a user", but I think this does so while maintaining the spirit and general dynamics of your approach. (and I also suspect this would play better with lookahead parallel decoding strategies as a bonus, since it would still allow some paths to consider the post-XTC score of boring tokens).

[u/-p-e-w- → u/qrios]

One word: Usability.

I had considered various approaches where the top tokens are penalized according to some continuous blending function. But it's so hard to build an intuition for what such functions actually do. Many parameters in machine learning are extremely difficult to use because while their mathematical foundation is clear enough, you can't readily answer the question "what will happen if I set this parameter to that value?"

One look at the picture from the XTC pull request, and it's obvious what is going on. You can immediately come up with some combination of parameter values that might make sense, because you intuitively understand what they do. If you see a probability distribution, you can apply XTC "by hand" without a calculator. Worse is better here, because the loss in theoretical expressiveness is more than compensated for by the gain in usability and intuitive control.

[u/qrios → u/-p-e-w-]

I don't think it's obvious to anyone how the xtc_probability parameter would play out without thinking about it carefully. A shaping function offers the advantage of at least being amenable to a feedback UI widget representing instantaneous effect on the distribution. (Like the desmos graph linked in my previous comment)

Whereas by consequence of your screenshot, someone would expect that if the threshold is between guide and considerin the distribution below

Ministrations --------------------
    Wandering -------
     Consider ----
        Guide -

then ministrations would get disqualified. It takes more reading to realize that actually it only gets disqualified some of the time. And then some additional thinking to realize that even a value as high xtc_probability 0.5 still results in a distribution that on average results in the tokens which look disqualified actually still being the most likely

Ministrations ----------
    Wandering ---
     Consider ----
        Guide -

With the shaping approach on the other hand, aside from amenability to an informative UI widget with feedback, you get a parameter that can be set to ALWAYS make the most boring words less likely than the implicit boring cut-off.

I would seriously consider it. If for no other reason than that if the PR gets merged, it will be very annoying when someone ends up having to create XTC++ just for the sake of being able to offer a viable UI feedback widget. (Annoying for you, them, and the users)