Skip to content

Commit b8eaf6a

Browse files
vdyevdye
authored
Merge pull request #44 from github/vdye/technical-diagrams
Add technical documentation
2 parents f35cf24 + d98e9c9 commit b8eaf6a

File tree

3 files changed

+362
-1
lines changed

3 files changed

+362
-1
lines changed

‎.vscode/settings.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
// File formatting
33
"[markdown]": {
44
"editor.detectIndentation": false,
5-
"editor.insertSpaces": false,
5+
"editor.insertSpaces": true,
66
"editor.tabSize": 4,
77
"editor.wordWrap": "off",
88
"files.trimTrailingWhitespace": true,

‎docs/technical/architecture.md

Lines changed: 249 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,249 @@
1+
# Bundle Server Architecture
2+
3+
This document contains information about the architecture of the bundle server
4+
and how it is used with Git's [bundle-uri feature][bundle-uris].
5+
6+
[bundle-uris]: https://git-scm.com/docs/bundle-uri
7+
8+
## High-level design
9+
10+
The following diagram shows the relationship between the Git remote, Git client,
11+
and the bundle server with respect to typical end-user usage.
12+
13+
```mermaid
14+
graph TB
15+
remotes[("Remote host(s)\n(GitHub, Bitbucket, GitLab, Codeberg, etc.)")]
16+
subgraph server["Bundle Server"]
17+
direction LR
18+
19+
repos[[Repository storage]]
20+
bundles[[Bundle storage]]
21+
routes[Route list]
22+
web([git-bundle-web-server])
23+
24+
repos -. "git-bundle-server update" .-> bundles
25+
routes --> web
26+
bundles --> web
27+
end
28+
git(["git (clone|fetch)"])
29+
30+
style server fill:#4682b477
31+
32+
remotes --> repos
33+
web --> git
34+
remotes --> git
35+
```
36+
37+
### Components
38+
39+
#### Remote host(s)
40+
41+
The Git hosts corresponding to the repositories served by the bundle server. The
42+
bundle server can contain repositories from different remotes (e.g. one from
43+
GitHub, another from GitLab), but each repository will have only one upstream
44+
remote.
45+
46+
#### Repository storage
47+
48+
A collection of Git bare repositories cloned from the corresponding remote(s),
49+
each representing a configured route on the bundle server. Repositories are
50+
cloned into local storage at the path `~/git-bundle-server/git/<route>` (e.g.
51+
`~/git-bundle/server/git/torvalds/linux` for the route `torvalds/linux`).
52+
53+
These repositories are kept up-to-date with their corresponding remote using
54+
`git-bundle-server update`, either run manually or via the system scheduler
55+
automatically started with `git-bundle-server (init|start)`. The repos are the
56+
source of the bundles generated for the "Bundle storage" of each route.
57+
58+
#### Bundle storage
59+
60+
The base and incremental bundles for each active repository on the bundle
61+
server. Bundles are created from the bundle server's cloned bare repositories
62+
(see "Repository storage") and are stored on disk at the path
63+
`~/git-bundle-server/www/<route>`, alongside a "bundle list" listing each bundle
64+
and associated metadata. These files are served to the user via the
65+
`git-bundle-web-server` API.
66+
67+
#### Route list
68+
69+
The list of _active_ routes in the bundle server (i.e., those for which bundles
70+
are being generated and can be served via the web server).
71+
72+
#### `git-bundle-web-server`
73+
74+
The `git-bundle-web-server` executable built from this repository. It can be run
75+
in the foreground directly, or started in the background with `git-bundle-server
76+
web-server start`.
77+
78+
#### `git (clone|fetch)`
79+
80+
The Git client invoked by users, CI, IDEs, etc. Only the `clone` and `fetch`
81+
commands use a bundle URI.
82+
83+
To bootstrap a repository from a given bundle URI, clone with `git clone
84+
--bundle-uri=<uri>`. This will download all bundles from the bundle server
85+
before fetching the remaining reachable objects from the origin remote.
86+
87+
When using this bundle server, `git clone --bundle-uri` will set the
88+
`fetch.bundleURI` configuration key in the repository. Using this configuration,
89+
future `git fetch` calls will also [check the bundle server for new
90+
bundles][bundle-uri-fetch] according to the `creationToken` heuristic before
91+
fetching from the origin remote.
92+
93+
[bundle-uri-fetch]: https://git-scm.com/docs/bundle-uri#_fetching_with_bundle_uris
94+
95+
## Use with `git`
96+
97+
Although the contents of the bundle server can be downloaded manually, the
98+
intended use case of the bundle server is to supplement clones & fetches in Git.
99+
100+
In the following diagrams, we will be assuming use of characteristics matching
101+
_this_ bundle server implementation, namely the `creationToken` heuristic.
102+
Behavior in Git may differ if using a different server implementation.
103+
104+
### Downloading and unpacking a bundle list
105+
106+
The recommended use of this bundle server is as a source for a "bundle list": an
107+
ordered list of base and incremental bundles that, in order, can be downloaded
108+
and unbundled to populate the requested commits in a fetch or clone. At the core
109+
of the bundle URI code in both `git clone` and `git fetch` is a common process
110+
for downloading and unpacking the contents of a bundle list. The process is as
111+
follows:
112+
113+
```mermaid
114+
%%{ init: { 'flowchart': { 'curve': 'monotoneX' } } }%%
115+
flowchart TB;
116+
start{"Start"}
117+
subgraph downloadAndUnbundle["Download and unbundle from list"]
118+
direction TB
119+
120+
parse["Parse bundle list"]
121+
sort["Sort bundles by creationToken,\nhigh to low, select bundle with\nhighest creationToken"]
122+
creationToken{{"Current creationToken >= <code>minCreationToken</code>?"}}
123+
reqBundle["Request bundle from server"]
124+
downloadSuccess{{"Download successful?"}}
125+
markUnbundled["Mark unbundled"]
126+
markUnbundledSkip["Mark unbundled to\navoid retrying later"]
127+
deeperExists{{"Are there more not-yet-unbundled bundles\nwith creationToken <i>less than</i> current?"}}
128+
moveDeeper["Select next bundle with\ncreationToken less than current"]
129+
unbundleReq["Unbundle downloaded bundle"]
130+
shallowerExists{{"Are there more not-yet-unbundled bundles\nwith creationToken <i>greater than</i> current?"}}
131+
moveShallower["Select next bundle with\ncreationToken greater than current"]
132+
unbundleSuccess{{"Successfully unbundled? (not\nmissing any required commits)"}}
133+
end
134+
bundleServer[(Bundle Server)]
135+
done{"Done"}
136+
137+
style downloadAndUnbundle fill:#28865477
138+
139+
start --> parse --> sort --> creationToken
140+
creationToken ----------> |No| done
141+
creationToken --> |Yes| reqBundle
142+
reqBundle --> downloadSuccess
143+
downloadSuccess --> |No| markUnbundledSkip
144+
markUnbundledSkip --> deeperExists
145+
deeperExists --> |No| done
146+
deeperExists --> |Yes| moveDeeper --> creationToken
147+
reqBundle <--> bundleServer
148+
149+
downloadSuccess --> |Yes| unbundleReq
150+
unbundleReq --> unbundleSuccess
151+
unbundleSuccess ----> |No| deeperExists
152+
shallowerExists --> |No| done
153+
unbundleSuccess --> |Yes| markUnbundled --> shallowerExists
154+
shallowerExists --> |Yes| moveShallower --> unbundleReq
155+
156+
```
157+
158+
Note that this flow requires a `minCreationToken`: a creationToken value used to
159+
avoid redundant downloads of old bundles. This value depends on whether the
160+
algorithm is called from `git clone` or `git fetch`. Details on how this value
161+
is determined can be found in later sections.
162+
163+
### `git clone`
164+
165+
When performing an initial clone from a remote repository, the `--bundle-uri`
166+
option can point to a bundle list (recommended with this server) or to a single
167+
base bundle. In the case of a bundle list, the bundle URI will be stored along
168+
with a `minCreationToken` value in the repository config for subsequent fetches.
169+
170+
```mermaid
171+
%%{ init: { 'flowchart': { 'curve': 'monotoneX' } } }%%
172+
flowchart TB;
173+
user((User))
174+
subgraph git
175+
direction TB
176+
177+
setBundleUri["Set <code>bundleUri</code> to the value of --bundle-uri"]
178+
downloadUri["Download from <code>bundleUri</code>"]
179+
downloadType{{"What is downloaded?"}}
180+
unbundle["Unbundle response"]
181+
setCreationToken["Set <code>minCreationToken</code> to 0"]
182+
downloadAndUnbundle(["Download and unbundle from list"])
183+
bundleSuccess{{"Bundles downloaded and unpacked successfully?"}}
184+
saveUri["Set fetch.bundleUri to <code>bundleUri</code>"]
185+
saveCreationToken["Set fetch.bundleCreationToken to highest\nunbundled creationToken"]
186+
incrementalFetch["Incremental fetch from origin"]
187+
188+
style downloadAndUnbundle fill:#288654,color:#000000
189+
end
190+
bundleServer[(Bundle Server)]
191+
origin[(Remote host)]
192+
193+
user --> |"git clone --bundle-uri URI"| setBundleUri
194+
downloadUri <--> bundleServer
195+
setBundleUri --> downloadUri --> downloadType
196+
downloadType --> |Single bundle| unbundle
197+
unbundle --> incrementalFetch
198+
downloadType --> |Other| incrementalFetch
199+
downloadType --> |Bundle list| setCreationToken
200+
setCreationToken --> downloadAndUnbundle --> bundleSuccess
201+
bundleSuccess --> |Yes| saveUri
202+
downloadAndUnbundle <---> bundleServer
203+
bundleSuccess --> |No| incrementalFetch
204+
saveUri --> saveCreationToken --> incrementalFetch
205+
incrementalFetch <--> origin
206+
```
207+
208+
### `git fetch`
209+
210+
After successfully cloning with a bundle list URI (recommended) or manually
211+
setting `fetch.bundleUri`, `git fetch` will try to download and unpack recent
212+
bundles containing new commits.
213+
214+
```mermaid
215+
%%{ init: { 'flowchart': { 'curve': 'monotoneX' } } }%%
216+
flowchart TB;
217+
user((User))
218+
subgraph git
219+
direction TB
220+
221+
bundleUriExists{{"fetch.bundleUri config is set?"}}
222+
setBundleUri["Set <code>bundleUri</code> to the value of fetch.bundleUri"]
223+
creationTokenExists{{"fetch.bundleCreationToken config is set?"}}
224+
setCreationToken["Set <code>minCreationToken</code> to the value\nof fetch.bundleCreationToken"]
225+
setCreationTokenZero["Set <code>creationToken</code> to 0"]
226+
downloadAndUnbundle(["Download and unbundle from list"])
227+
bundleSuccess{{"Bundles downloaded and unpacked successfully?"}}
228+
saveCreationToken["Set fetch.bundleCreationToken to highest\nunbundled creationToken"]
229+
incrementalFetch["Incremental fetch from origin"]
230+
231+
style downloadAndUnbundle fill:#288654,color:#000000
232+
end
233+
bundleServer[(Bundle Server)]
234+
origin[(Remote host)]
235+
236+
user --> |"git fetch"| bundleUriExists
237+
bundleUriExists --> |Yes| setBundleUri
238+
bundleUriExists --> |No| incrementalFetch
239+
setBundleUri --> creationTokenExists
240+
creationTokenExists --> |Yes| setCreationToken
241+
creationTokenExists --> |No| setCreationTokenZero
242+
setCreationToken & setCreationTokenZero --> downloadAndUnbundle
243+
downloadAndUnbundle <--> bundleServer
244+
downloadAndUnbundle --> bundleSuccess
245+
bundleSuccess --> |Yes| saveCreationToken
246+
bundleSuccess --> |No| incrementalFetch
247+
saveCreationToken --> incrementalFetch
248+
incrementalFetch <--> origin
249+
```

‎docs/technical/web-server.md

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Web Server API Reference
2+
3+
This document contains an API specification for the web server created with
4+
`git-bundle-web-server`. It is primarily meant to be used by Git via the
5+
[bundle-uri feature][bundle-uris].
6+
7+
> **Warning**
8+
>
9+
> First and foremost, the goal of this API is compatibility with Git's bundle
10+
> URI feature. We will attempt to keep it up-to-date with the latest version of
11+
> Git but, due to both the newness of the feature and experimental state of the
12+
> server, we cannot make guarantees of backward compatibility.
13+
14+
[bundle-uris]: https://git-scm.com/docs/bundle-uri
15+
16+
## Get a repository's bundle list
17+
18+
Get the list of bundles configured for a given bundle server route.
19+
20+
<table>
21+
<tbody>
22+
<tr>
23+
<th>Method</th>
24+
<td><code>GET</code></td>
25+
</tr>
26+
<tr>
27+
<th>Route</th>
28+
<td><code>/{route}</code></td>
29+
</tr>
30+
<tr>
31+
<th>Example Request</th>
32+
<td><code>curl http://localhost:8080/OWNER/REPO</code></td>
33+
</tr>
34+
<tr>
35+
<th>Example Response</th>
36+
<td>
37+
38+
```
39+
[bundle]
40+
version = 1
41+
mode = all
42+
heuristic = creationToken
43+
44+
[bundle "1678494078"]
45+
uri = REPO/base-1678494078.bundle
46+
creationToken = 1678494078
47+
48+
[bundle "1679527263"]
49+
uri = REPO/bundle-1679527263.bundle
50+
creationToken = 1679527263
51+
52+
[bundle "1680561322"]
53+
uri = REPO/bundle-1680561322.bundle
54+
creationToken = 1680561322
55+
```
56+
57+
</td>
58+
</tr>
59+
</tbody>
60+
</table>
61+
62+
### Path parameters
63+
64+
| Name | Type | Required | Description |
65+
| ------- | ------ | --------- | ----------- |
66+
| `route` | string | Yes | The route of a repository created with `git-bundle-server init` for which the list of active bundles is requested. Route should be in `OWNER/REPO` format. |
67+
68+
### HTTP response status codes
69+
70+
| Code | Description |
71+
| ----- | ----------- |
72+
| `200` | OK |
73+
| `404` | Specified route does not exist or has no bundles configured |
74+
75+
## Download a bundle
76+
77+
Download an individual bundle.
78+
79+
<table>
80+
<tbody>
81+
<tr>
82+
<th>Method</th>
83+
<td><code>GET</code></td>
84+
</tr>
85+
<tr>
86+
<th>Route</th>
87+
<td><code>/{route}</code></td>
88+
</tr>
89+
<tr>
90+
<th>Example Request</th>
91+
<td><code>curl http://localhost:8080/OWNER/REPO/bundle-1679527263.bundle</code></td>
92+
</tr>
93+
<tr>
94+
<th>Example Response</th>
95+
<td><i>Binary </i><a href="https://git-scm.com/docs/git-bundle"><code>git bundle</code></a><i> bundle content.</i></td>
96+
</tr>
97+
</tbody>
98+
</table>
99+
100+
### Path parameters
101+
102+
| Name | Type | Required | Description |
103+
| -------- | ------ | --------- | ----------- |
104+
| `route` | string | Yes | The route of a repository containing the desired bundle. Route should be in `OWNER/REPO` format. |
105+
| `bundle` | string | Yes | The filename of the desired bundle as identified by the `route`'s bundle list. |
106+
107+
### HTTP response status codes
108+
109+
| Code | Description |
110+
| ----- | ----------- |
111+
| `200` | OK |
112+
| `404` | The specified bundle does not exist |

0 commit comments

Comments
 (0)