Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multipart upload #1

Open
wants to merge 35 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
2426e1e
chore: apply prettier formatting
whoabuddy Jul 30, 2024
9d1554f
fix: use const for content type, update repo name
whoabuddy Jul 30, 2024
7d21914
fix: add multipart functions courtesy of claude
whoabuddy Jul 30, 2024
e566884
chore: use unused const
whoabuddy Jul 30, 2024
44f0c6c
fix: use expires_in instead of EXPIRY const
whoabuddy Jul 30, 2024
85e1bff
fix: add try/catch with specific errors
whoabuddy Jul 30, 2024
10cf4fb
chore: remove unused code
whoabuddy Jul 30, 2024
b0f33a4
fix: move object processing outside of response
whoabuddy Jul 30, 2024
643116e
fix: add more robust error messages
whoabuddy Jul 30, 2024
6fa2134
fix: rework multipart upload logic
whoabuddy Jul 30, 2024
f5029c6
fix: add more logging to mutlipart upload
whoabuddy Jul 30, 2024
8237cfd
fix: add more specific errors for multipart
whoabuddy Jul 30, 2024
a442f84
fix: update url format
whoabuddy Jul 30, 2024
2bcfbd7
fix: encode query params with url
whoabuddy Jul 30, 2024
9bc2520
fix: use oid to create key
whoabuddy Jul 30, 2024
a66b8b5
fix: update one last oid/key ref
whoabuddy Jul 30, 2024
4e2c117
fix: add logging to test url format
whoabuddy Jul 30, 2024
e0a7745
chore: add more logging
whoabuddy Aug 26, 2024
4d883c5
fix: add moar logging
whoabuddy Aug 26, 2024
2cc6086
fix: log from single call with multiline
whoabuddy Aug 26, 2024
1598029
fix: more logging and testing
whoabuddy Aug 26, 2024
a048538
fix: narrowing down the issue
whoabuddy Aug 26, 2024
ee3fda0
docs: add mermaid diagram of worker ops
whoabuddy Aug 26, 2024
85b0d40
chore: cleanup naming, use R2 over S3
whoabuddy Aug 26, 2024
cbfda28
fix: adjust some URL formation
whoabuddy Aug 26, 2024
faca6a8
fix: encode URL before fetch
whoabuddy Aug 26, 2024
9c79d9f
fix: dont encode url twice
whoabuddy Aug 26, 2024
13a2d56
fix: chasing down something small
whoabuddy Aug 26, 2024
68aacb6
fix: was still double encoding
whoabuddy Aug 26, 2024
4ef61e1
fix: something with the url
whoabuddy Aug 26, 2024
ca97518
chore: cleanup typos
whoabuddy Aug 26, 2024
1ee39fc
fix: expand bucket options
whoabuddy Aug 26, 2024
76fc7f1
fix: just a bit more logging
whoabuddy Aug 26, 2024
17b3da0
fix: small tweaks
whoabuddy Aug 26, 2024
89bfea4
docs: add old worker for side-by-side review
whoabuddy Aug 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,3 +167,33 @@ You're now ready to [start using Git LFS](https://github.com/git-lfs/git-lfs#exa
git fetch --all
git lfs migrate import --everything --above=25MiB
git push --all --force-with-lease

## Operation Diagram

```mermaidjs
sequenceDiagram
participant Client as Git LFS Client
participant Fetch as fetch function
participant Parse as parseAuthorization
participant Handle as handleMultipartUpload
participant Initiate as initiateMultipartUpload
participant Sign as sign function
participant S3 as S3/R2 API

Client->>Fetch: POST /objects/batch
Fetch->>Parse: Parse Authorization Header
Parse-->>Fetch: Return credentials
Fetch->>Handle: For each large object
Handle->>Initiate: Start multipart upload
Initiate->>Sign: Generate signed URL
Sign-->>Initiate: Return signed URL
Initiate->>S3: POST initiate multipart upload
S3-->>Initiate: Return UploadId
Initiate-->>Handle: Return UploadId
Handle->>Sign: Generate part URLs
Sign-->>Handle: Return part URLs
Handle->>Sign: Generate complete URL
Sign-->>Handle: Return complete URL
Handle-->>Fetch: Return upload details
Fetch-->>Client: Return batch response
```
Loading