Add opt-in materialized weight loading override#1993
Open
k1x0r wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opt-in
OSAURUS_MATERIALIZE_WEIGHTSenvironment variable that forces safetensor weights to be materialized instead of memory-mapped.This is useful on systems where memory-mapped model weights can be evicted or cause unstable performance under memory pressure. Materializing the weights keeps them resident in allocated memory and provides a workaround without changing the default loading behavior for other users.
When
OSAURUS_MATERIALIZE_WEIGHTS=1is set, Osaurus uses:When the variable is not set, the existing memory-mapped loading behavior remains unchanged.
Test Plan
Model used:
minimax-m2.7-jangtqBuild and launch Osaurus without the environment variable.
Load the model and verify that the existing memory-mapped safetensor path is used.
Quit Osaurus.
Launch it with the environment variable enabled:
Alternatively, add
OSAURUS_MATERIALIZE_WEIGHTS=1to the environment variables in the Xcode scheme.Load the same model.
Verify that
useMmapSafetensorsis set tofalseand the safetensor weights are materialized.Run an inference request and confirm that model loading and generation complete successfully.
Remove the environment variable and confirm that the default memory-mapped behavior is restored.
Screenshots
Not applicable. No UI changes.