FileIndex + New File Structure + File Prefetch + Parallel Write #8
Conversation
… number of entries in directory as if we've written files at level0
marino39
changed the title
| }) | ||
| func (r *reader[T]) prefetchNextFile(ctx context.Context) { | ||
| if r.currFileIndex+1 < len(r.fileIndex.Files()) { | ||
| go r.prefetchFile(ctx, r.fileIndex.At(r.currFileIndex+1)) |
There was a problem hiding this comment.
Are there limits for the number of concurrent prefetchFile tasks spawned simultaneously?
There was a problem hiding this comment.
Yes, it's always the next file. So it's kind of limited by that. So no more than 1 prefetch can be spawned.
| return r.readFile(index) | ||
| } | ||
| } | ||
| _ = file.Prefetch(pCtx, r.fs) |
There was a problem hiding this comment.
The error here is neither logged nor handled.
There was a problem hiding this comment.
This is fire and forget background task. If it fails the file will be opened as usual instead of using prefetched buffer. No need for logging or handling.
Please have a look at File.Open:
Lines 167 to 176 in 57ad4b3
|
Looking great overall, I added a few comments |
Thanks, I will address tomorrow. |
| // -- ethwal | ||
| // |-- 000563 | ||
| // | |-- 000256 | ||
| // | | |-- 000124 | ||
| // | | | |-- 28f55a4df523b3ef28f55a4df523b3ef28f55a4df523b3ef28f55a4df523b3ef <- ethwal file | ||
| // | | |-- 000278 | ||
| // | | | |-- 28f55a4df523b3ef28f55a4df523b3ef28f55a4df523b3ef28f55a4df523b3dd <- ethwal file | ||
| // | |-- 000025 | ||
| // | | |-- 000967 | ||
| // | | | |-- 28f55a4df523b3ef28f55a4df523b3ef28f55a4df523b3dd28f55a4df523b3ef <- ethwal file | ||
| // |-- .fileIndex | ||
| // | ||
| // The data structure ensures that there is no more than 1000 directories per level. The filename is a sha-256 hash of | ||
| // the first and last block numbers. The hash is used to distribute files evenly across directories. |
There was a problem hiding this comment.
what is the main reason for creating this 3-level folder structure?
"no more than 1000 directories per level" -- is this some limitation of GCP buckets? or is it so we don't need to list dirs with paging? anything else?
There was a problem hiding this comment.
The local filesystem start to have a issues with listing files ~20k files. It takes a while. The problem becomes even more visible when we are trying to list Google Storage bucket contents. It took a couple of minutes with aprox. 40k files.
In additon to that MacOS have been hanging when I have been working in proximity of ethwal directory.
Peter suggested to put files in directories with each of them having limited number of files that can be worked with easily.
There was a problem hiding this comment.
cool, thanks for the insights :)
FileIndex + New File Structure + File Prefetch + Parallel Write
No description provided.