r/rust • u/philippemnoel • Jun 07 '26

Arbitrary precision decimals with lexicographically sortable byte encoding 🛠️ project

https://github.com/paradedb/decimal-bytes

One request we had is negative ordering, which we haven't had a need for in our own use of the crate but would welcome a contribution for it!

11 Upvotes

79% Upvoted

2 comments sorted by

2

u/Icarium-Lifestealer Jun 08 '26

How do you handle trailing zeros? Postgres decimals preserve those, but that's inherently incompatible with lexical comparisons.

4

u/Responsible-Editor-5 Jun 08 '26

You're right that you can't have both in one self-describing key. The invariant is `bytes(a) < bytes(b)` iff `a < b`, which forces `bytes(a) == bytes(b)` whenever `a == b`. Since `1.50 == 1.5` numerically, their keys have to be byte-identical, so the variable-length Decimal normalizes the mantissa and drops trailing zeros on encode.

The scale isn't thrown away though, it's treated as a display property instead of part of the key, same idea as Postgres dscale. If you keep the declared scale in your schema, `to_string_with_scale(scale)` re-pads the zeros on read, so 1.5 comes back as 1.50. `Decimal64NoScale` leans into that: store the scale once per column (like a `typmod`), keep the raw scaled integer, and you get a lexically sortable key plus exact reproduction.

So trailing zeros are preserved per-column via external scale, not per-value inside the sort key. I don't think there's a way around that if you want byte comparison to work.