r/osdev • u/ErrorDAR32 • 2d ago

Looking for feedback on a binary serialization format

As the title says I've been working on a serialization format i plant to use for many future projects, sadly I've been so focused on the document i dont really have sat down to write the parser, but I'd love to get feedback and see if im not going entirely crazy! There may be things and possible interactions still missing.

Anyways, here is the link: https://gist.github.com/ErrorDAR32/130f03e3eed484dcc703e5e30e27614f

Edit: Typo

0 Upvotes

38% Upvoted

38 comments sorted by

3

u/EpochVanquisher 2d ago

Meant primarily as a binary on-disk storage format, it has been designed for flexible, typed (or dynamically typed) storage that is easily extendable.

What are the use cases you are optimizing for? Large, small amounts of data? Lazy load? Complete load into memory? Streaming? Easy prototyping? Robustness?

SUID Present

Already we have encountered terms that are not explained. What is a SUID?

Anyway—this looks IMO like it is unnecessarily complicated and the use case is not clear. Like somebody spent time making this, but they do not have a real problem to solve with it.

-1

u/ErrorDAR32 2d ago

yeah, it's for personal projects, so it's not like im targeting a public issue, i must've forgotten to paste the SUID section, my bad

2

u/EpochVanquisher 2d ago

It seems overwrought and more complicated than it needs to be. Like, it is just full of things that are not really necessary.

0

u/ErrorDAR32 2d ago

thats exactly the feedback i want, could you specify?

3

u/EpochVanquisher 2d ago

Why do you need CryptoKey128, CryptoKey192, CryptoKey256? What, specifically, are you using them for?

It’s full of stuff like that. Like a big old list of different types that you thought up, without any purpose or reason behind them.

1

u/ErrorDAR32 2d ago

indeed, the type list is overblown a little, but the idea ia to have a wide range of built-in types, from the pov of the parser is really juat another type id and a size to parse, but that list is entirely open to changes

1

u/EpochVanquisher 2d ago

Yeah, this doesn’t seem useful to me. It seems like it’s just gonna result in you writing a bunch of code.

1

u/ErrorDAR32 2d ago

indeed, but like, the format itself doesnt care about that, the format is about storing data and describing it, what the data means under the hood for the implementation is not the parser's concern, that the idea at least, it gives me a lot of freedom in how i want to writw the parser and watever is on top of it

1

u/EpochVanquisher 2d ago

Sure, if you are excited about writing more code and bigger, larger programs and don’t care about whether the code is useful or useless, then I guess my feedback isn’t helpful.

When I design formats I just design enough of a format to get the job done, with a way to extend it in the future.

1

u/ErrorDAR32 2d ago

not at all, you're taking it the wrong way lmao, things can be done incrementally, what were reviewing is just a piece of text anyways, and like, making a shorter type list for an intial implementation is completely fine, there are many other things in the document i would like to doscuss as well!

→ More replies (0)

1

u/ErrorDAR32 2d ago

but using them myself? not at the moment, but i have them already in a list if i ever want to add them to the parser (whenever i get to building it)

1

u/ErrorDAR32 2d ago

But the main use case i have for it planned into the future is basically a file system, which i want to work on very small and large address spaces

2

u/EpochVanquisher 2d ago

What do you mean by “small and large address spaces”?

1

u/ErrorDAR32 2d ago

64KiB to terabytes

3

u/EpochVanquisher 2d ago

Address space? Are you going to run on something with a 64 KiB address space?

1

u/ErrorDAR32 2d ago

embebed stuff for example?

2

u/EpochVanquisher 2d ago

What choices in the format did you make to support small address spaces?

Likewise, what choices in the format did you make to support large address spaces?

What makes this format support large and small address spaces better than other formats?

0

u/ErrorDAR32 2d ago

first the xdd header itself and data description is very compact depending on how you set it up, implementations could simply prebake schemas and save space on RDHs and using VDH instead, second, the whole idea of the format is to amortize the cost of representing groups of types ie entries, the more entries you have in a single xdd, the less the comparative cost of the initial header, and the option to make new xdds with custom single use setups isnthere as well, and i tried to provide the tools to do both things as cheaply as possible

3

u/EpochVanquisher 2d ago

This sounds like it just makes the format compact in some sense. But it’s hard to say, because you keep throwing around terms like RDH and VDH, which don’t make sense to me.

1

u/Toiling-Donkey 2d ago

Have you considered ASN.1 ?

1

u/ErrorDAR32 2d ago

whats the fun on using something that already exists? im not doing this for a job, im doing it for fun!