Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restore and Check. #191

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Restore and Check. #191

wants to merge 9 commits into from

Conversation

aversecat
Copy link
Contributor

@aversecat aversecat commented Oct 16, 2024 •
edited
Loading

This branch against main contains both the parallel restore library, tests, as well as the check code.

The check code has been reduced to our validation tool to identify inconsistencies in an umounted scoutfs meta device, only. All error injection and repair code has been omitted. The check code is used to validate the output of the parallel restore test cases.

  • - tests need to pass, but check fails. There either is a block reading problem in scoutfs check or there is an error inserting during restore.
  • - quota print outputs incorrect quota rules during test

New issues from @chaowang-versity:

  • - parallel_restore binary fails with more than 10 threads
  • - hardlinks are incorrect: After parallel_restore, hardlinked items have a hardlink count of 1 where that number should be larger than 1 for obvious reasons.
  • - util.h should be exported, because format.h uses DIV_ROUND_UP which is defined in it.

@aversecat aversecat added the enhancement New feature or request label Oct 16, 2024
@aversecat aversecat force-pushed the auke/restore_and_check branch from 67bbd3c to ae4b55a Compare October 17, 2024 20:37
@aversecat aversecat force-pushed the auke/restore_and_check branch from ae4b55a to a2364f2 Compare December 17, 2024 18:09
@aversecat aversecat changed the title Restore and Check. **WIP** Restore and Check. Dec 17, 2024
@aversecat aversecat force-pushed the auke/restore_and_check branch from a2364f2 to fac2fec Compare December 19, 2024 23:47
@aversecat
Copy link
Contributor Author

pushed el7 fixes into this.

@chaowang-versity chaowang-versity self-requested a review January 15, 2025 17:39
@zabbo
Copy link
Collaborator

zabbo commented Jan 21, 2025 •
edited
Loading

Let's pull out the supporting commits that are fine on their own and don't really have much to do with parallel restore. We can land them in their own PR and then focus on fixing up the parallel restore commits. I think the following should do it:
bb2003c Fix printing alloc list block extents
0d156e0 Import a few more functions to our list.h
3f3c2d8 Add userspace NSEC_PER_SEC
9cbe041 Add bloom filter index calc for userspace utils
fcdefb7 Add srch_encode_entry() for userspace utils
b3a8380 Add put_unaligned_leXX() for userspace
ad8a0f7 Add fls64() alias for userspace flsll()
9524397 Promote userspace btree block initialization
cd43c39 Add userspace version of our mode to type
29ca2ad Add userspace version of our dirent name hash
0020eac Add lk rbtree wrapper
1f7f40c Add test_bit to utils bitmap

@aversecat aversecat mentioned this pull request Jan 22, 2025
Merged
@aversecat
Copy link
Contributor Author

aversecat commented Jan 22, 2025 via email

zabbo and others added 9 commits February 10, 2025 12:56
@zabbo @aversecat
Add check command
3ae5502 
Signed-off-by: Zach Brown <[email protected]>
Signed-off-by: Auke Kok <[email protected]>
@zabbo @aversecat
Fix partial rename to check_meta_alloc
df9d585 
As I was committing the initial check command I had only partially
completed a rename of the function that checks the metadata allocators.

Signed-off-by: Zach Brown <[email protected]>
@zabbo @aversecat
Add {read,write}-metadata-image scoutfs commands
3dad5e6 
Signed-off-by: Zach Brown <[email protected]>
@aversecat
Generic block header checks: crc, magic.
7ecda34 
Generally as we call block_get() we should validate that if the block
has a hdr, at a minimum the crc is correct and the magic value is
the expected value passed, and the fsid matches the superblock. This
function implements just that. Returns -EINVAL, up to the caller to
report a problem() and handle the outcome. For now the code just hard
fails, which incedentally makes it fail the clobber-repair.sh tests
I wrote.

Signed-off-by: Auke Kok <[email protected]>
@aversecat
Add man page content for check.
9c180eb 
Adds basic man page content for the `check` subcommand.

Signed-off-by: Auke Kok <[email protected]>
@aversecat
Superblock checks for meta and data dev.
ee30367 
We check superblock magic, crc, flags. data device superblock is
checked but a little less thorough.  We check whether the device is
still mounted, since that would make checking invalid to begin with.
Quorum blocks are validated to have sane contents.

We add a global problem counter so we can trivially measure and
report whether any problem was found at all, instead of iterating
over all the problems and checking each individual count.

We pick the standard exit code values from `fsck` and mirror their
intentional behavior. This results in `fsck.scoutfs` can now be
trivially created by making it a wrapper around `scoutfs check`.

Signed-off-by: Auke Kok <[email protected]>
Signed-off-by: Hunter Shaffer <[email protected]>
@zabbo @aversecat
Add parallel restore
2f4c512 
Signed-off-by: Zach Brown <[email protected]>
Signed-off-by: Hunter Shaffer <[email protected]>
Signed-off-by: Auke Kok <[email protected]>
@zabbo @aversecat
Add test for parallel restore
05c83ee 
This is the benchmark binary that bulk creates filesystem items, xattrs
and is heavily threaded to scope the performance of the library. The
test script invokes it to validate some basic constraints.

Signed-off-by: Zach Brown <[email protected]>
Signed-off-by: Hunter Shaffer <[email protected]>
Signed-off-by: Auke Kok <[email protected]>
@aversecat
Copy a tree using parallel restore library.
edacd17 
This tool compies a source tree (whether it's scoutfs or not)
into an offline scoutfs meta device. It has only those 2 parameters
and does a single-process walk of the tree to restore all items
while preservice as much of the metadata as possible.

Signed-off-by: Hunter Shaffer <[email protected]>
Signed-off-by: Auke Kok <[email protected]>
@aversecat aversecat force-pushed the auke/restore_and_check branch from 92c433c to edacd17 Compare February 10, 2025 17:56
@aversecat
Copy link
Contributor Author

rebased onto main

@aversecat
Copy link
Contributor Author

This is failing tests on scoutfs df output. We'll need to scrub it or always force meta/data sizing, otherwise it'll always diff.

@zabbo
Copy link
Collaborator

zabbo commented Feb 18, 2025

This is failing tests on scoutfs df output. We'll need to scrub it or always force meta/data sizing, otherwise it'll always diff.

Yeah, the test shouldn't be letting the raw df output through to be compared.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chaowang-versity chaowang-versity Awaiting requested review from chaowang-versity

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@aversecat @zabbo