excited to announce that Wizard Zines is now shipping Canadian orders from Canada! now shipping is cheaper & there's no risk of being hit with duties
get your zines at https://wizardzines.com
Timeline
Post
Remote status
Context
6
@b0rk awesome news. were you able to find a local printer?
@vga256 no, we just shipped a batch of zines to Montreal from our US printer, right now the volume is too low to justify a local print run
@mwl i imagine it’s easier for 28 page zines
@b0rk true, true. 1000 200-page books basically takes over a room.
Whelp. You've convinced me I'm in the wrong business, congrats! 
Replies
10
@djfiander @mwl @b0rk if you actually do a zine about ZFS please make sure to mention that it doesn't have a real fsck, and yes, this can and *has* caused data loss in production.
to me. personally.
not that i'm bitter.
not that this wouldn't be an unforgivable act of hubris by the zfs developers even if it had never caused anyone any data loss even in testing, either.
@zwol @djfiander @mwl @b0rk ZFS has scrub and zdb, two tools that are far more powerful than any fsck that has ever existed
@feld zfs scrub does not validate or correct the on-disk data structure, it only checks the block checksums. That's what I mean by "doesn't have a *real* fsck."
The data loss I experienced was due to a buggy migration of the on-disk data structure. All the block checksums validated, but files became inaccessible. Scrub did nothing. A real fsck might have dumped a bunch of stuff into lost+found, but that would've been enough to avoid having to restore from backup.
@zwol
> The data loss I experienced was due to a buggy migration of the on-disk data structure.
You're referring to the MOS, right? This is nearly impossible with ECC RAM IIRC
> Scrub did nothing.
idk how you could even scrub if what you describe actually happened. The pool should have been in an unusable state if the MOS was corrupted.
But then again, that's what zdb is for. You could have walked back / rewind transactions with zdb. There's a million things you can do with zdb and most of them are undocumented so you have to read the source or talk to a ZFS developer, but I really doubt your data was completely "inaccessible"
I had a completely broken RAIDZ due to a maintenance mistake and I was able to recover most of the data with a custom kernel patch that allowed the pool to be mounted readonly and any files with missing blocks (allocated on the missing disk) were represented as 0-bytes. A lot of this work has been integrated into modern ZFS so you can mount broken pools and get data off them now.
I really doubt you were as screwed as you thought.
> The data loss I experienced was due to a buggy migration of the on-disk data structure.
You're referring to the MOS, right? This is nearly impossible with ECC RAM IIRC
> Scrub did nothing.
idk how you could even scrub if what you describe actually happened. The pool should have been in an unusable state if the MOS was corrupted.
But then again, that's what zdb is for. You could have walked back / rewind transactions with zdb. There's a million things you can do with zdb and most of them are undocumented so you have to read the source or talk to a ZFS developer, but I really doubt your data was completely "inaccessible"
I had a completely broken RAIDZ due to a maintenance mistake and I was able to recover most of the data with a custom kernel patch that allowed the pool to be mounted readonly and any files with missing blocks (allocated on the missing disk) were represented as 0-bytes. A lot of this work has been integrated into modern ZFS so you can mount broken pools and get data off them now.
I really doubt you were as screwed as you thought.
@feld I don't know what a MOS is, but the root cause of my problem was not hardware-level bit rot; it was a logic bug, that caused the data structure on disk to be semantically incorrect, *despite* all the checksums being valid, in a way that rendered a whole bunch of files inaccessible.
I don't *care* whether a manual recovery would have been possible somehow. If it had been *any other FS*, fsck -p would have fixed it. zfs has no true equivalent of that command. That's all that matters to me.
@zwol where's the bug report for this logic bug? You can't have been the only person on the planet to hit it if it was a legit logic bug in ZFS
@feld Indeed, I wasn't the only person affected. I don't have a link to hand, but it's in FreeBSD's bug tracker somewhere. This was roughly four years ago, if that helps.
@zwol alright I'll try to find it, thx
@feld (For maximum clarity, by "inaccessible" I mean the files showed up in a directory listing but open() would return ENOENT.)
@feld (Yes, I *am* going to repeat that entire message verbatim every time someone tries to tell me zfs doesn't need fsck because it has scrub.)