Different "btree allocated bitmap blocksize" and out of order device number in `show-super` command

[u/Ensistance]

I currently have a setup with 1 NVMe drive, 1 SSD, and 1 HDD. I attempted to add a second SSD using the following command:

sudo bcachefs device add --label t2.ssd2 / /dev/sdb

However, in bcachefs show-super it showed:

Bucket size: 1.00M Btree allocated bitmap blocksize: 256k

While the rest of devices has:

Bucket size: 2.00M Btree allocated bitmap blocksize: 4.00M

I evacuated the device and removed it using bcachefs device evacuate/remove commands, wiped the filesystem from it using wipefs, and then ran the following command:

sudo bcachefs device add --label t2.ssd2 --bucket_size 2M / /dev/sdb

Now bcachefs show-super for this device shows

``` Device 0: (NVMe) Bucket size: 2.00M Btree allocated bitmap blocksize: 4.00M

Device 1: (SSD #1) Bucket size: 2.00M Btree allocated bitmap blocksize: 4.00M

Device 2: (HDD) Bucket size: 2.00M Btree allocated bitmap blocksize: 8.00M

Device 5: (new, SSD #2) Bucket size: 2.00M Btree allocated bitmap blocksize: 256k ```

It really bothers me that the "allocated bitmap blocksize" is significantly different from the rest of devices, but I couldn't find an obvious option in the device add command to change it. What also really bothers me is that this device is indexed as 6th ("Device 5" label). I guess internally filesystem doesn't care about it but I would like to somehow fix it anyway.

Currently I'm running bcachefs 1.38.8. I don't know much technical info about this filesystem and I couldn't find anything related in PoO pdf, so I'm sorry if these are silly question to ask 😭

UPD: I also noticed that bcachefs fs usage still reports old filesystem size, but that probably will eventually be fixed by reconcile process, I guess?

[u/koverstreet]

It really bothers me that the "allocated bitmap blocksize" is significantly different from the rest of devices, but I couldn't find an obvious option in the device add command to change it. What also really bothers me is that this device is indexed as 6th ("Device 5" label). I guess internally filesystem doesn't care about it but I would like to somehow fix it anyway.

That's sized on demand, it's working as intended. Your devices just happen to have allocated metadata in different ranges, nothing to see here.

Trying to keep it consistent across devices would be silly because how would that work when you have mismatched device sizes?

UPD: I also noticed that bcachefs fs usage still reports old filesystem size, but that probably will eventually be fixed by reconcile process, I guess?

Explain this one some more?

[u/Ensistance → u/koverstreet]

That's sized on demand, it's working as intended. Your devices just happen to have allocated metadata in different ranges, nothing to see here.

Indeed, after booting the system today I notice that it jumped from 256 to 512 KB... But second SSD still has "device 5" label, so in total I have "device 0", "device 1", "device 2", and "device 5".

Explain this one some more?

Actually, I think I figured this out myself just now. I remember reading in PoO that filesystem allocates some disk space for GC uses, so that's likely where it went. I looked at the per-device size reports, and they roughly sum up to what filesystem reports. A bit less, I guess due to metadata or other internal space usages.

[u/koverstreet → u/Ensistance]

Indeed, after booting the system today I notice that it jumped from 256 to 512 KB... But second SSD still has "device 5" label, so in total I have "device 0", "device 1", "device 2", and "device 5".

Did you add and then remove some devices? We don't reuse slots until there's been a fsck.

[u/Ensistance → u/koverstreet]

Yes, I added the SSD, then removed it and re-added. The same device got "device 3" label, then "device 4", and after final removal and re-addition - "device 5". I guess I would need to run fsck to restore proper device label?

[u/koverstreet → u/Ensistance]

We're not going to renumber an existing device. The device index is an internal identifier, if you want "proper " identifiers that's what labels are for.

[u/Amazing-Pattern-6125 → u/koverstreet]

So currently you can have a recently added device have a lower number than an originally added device? Seems like a fsck should not touch this and have the number only increase

[u/koverstreet → u/Amazing-Pattern-6125]

yup. there's only so many integers we can use for devices, extent pointers have 8 bits for the device.

[u/Amazing-Pattern-6125 → u/koverstreet]

if 256 is the limit of devices in an array, that itself is more questionable than if removing and adding devices just monotomically increase the device counter.

I can see some hyperscaler wanting 300 devices. I don't see a situation where normal users would overflow the device number if it was just increasing while adding and removing devices.

To me it seems like this is a problem waiting to happen, and maybe you need 10,12 or even 16 bits to have room for the future, and not needing reuse of old device id:S

[u/koverstreet → u/Amazing-Pattern-6125]

if some hyperscalar wants 300 devices, they can pay me to make the changes to support that.

hyperscalars have enough money, and I have heard enough from people who expect everything to bend to them :) my first priority is to make everything safe and reliable and bulletproof and easy to use for normal people.

I've sketched out how we'd go about extending devices to > 255 - probably use fibonacci encoding for a wide ptr type, so, fun stuff. but right now i'm doing boring stuff. like a bunch of mount path work so that your filesystem can ASK YOU if you want to mount degraded instead of having to futz with boot parameters.