Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk APIs on chunk management #530

Open
1 task done
Mili-ssm opened this issue Feb 2, 2025 · 0 comments
Open
1 task done

Bulk APIs on chunk management #530

Mili-ssm opened this issue Feb 2, 2025 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@Mili-ssm
Copy link
Contributor

Mili-ssm commented Feb 2, 2025

I've searched existing issues and couldn't find a duplicate.

  • I confirm this is not a duplicate.

Is your feature request related to a problem? Please describe.

With a bunch of stuff this feature will make the Chunk Linear format (PR #494) "usable".
pd: Bulk function/API is that use an slice/Vec as argument instead of an only one element.

The general point is that:

  • Bulk function can reach to better performance improvements or make some abstraction worth.
    (nice for more maintainable and flexible code that other wise could be not acceptable)

  • Bulk functions usually get advantage (without special code) of things like cache locality, nice for loops and CPU intensive code.
    (making better use of RAM and cache memory bandwidth)

  • Bulk function can make better async/sync code management as getting 1 lock for a entire slice of entitys/ops instead 1 lock for EVERY entity/op.
    (ie. 1 file lock to write many chunks instead of a 1 file lock for every chunk you want to write)

Describe the solution you'd like

I would like to use this issue (or other if you want) as guidelines for how to approach this thing, discuses about it, and follow some issues and PRs working for or depending on this.

Ez Guidelines

  • Be careful with Rayon. Is usually better to call the slice version than the one elemnt version with par_iter loop.

  • When you make 2 functions (one elment and slice version), the main function is the slice version. (even the only one)

//wrapper for ez of use
fn process_item(item: Item){
    process_items(&[item])
}

//main one
fn process_items(items: &[Item]){
 ...
}

Future Ideas

  • Bulk writes/reads api for chunks data from files.
  • Bulk writes/read api for entities.

Final

Please share your own ideas and opinions about this and where else it can be helpful.

@Mili-ssm Mili-ssm added the enhancement New feature or request label Feb 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants