Skip to content

Commit 260ca77

Browse files
ianmcookianmcook
authored
[http] Add range request examples (#43)
* Add range request examples * Use '.part' file extension * Improve JS server readme * Improve JS server * Improve examples * Improve server example
1 parent 4884e4b commit 260ca77

File tree

7 files changed

+163
-1
lines changed

7 files changed

+163
-1
lines changed

‎http/get_range/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,4 @@
1919

2020
# HTTP GET Arrow Data: Range Request Examples
2121

22-
This directory contains examples of HTTP servers/clients that send/receive data of known size (`Content-Length`) in the Arrow IPC streaming format and support range requests (`Accept-Range: bytes`).
22+
This directory contains examples of HTTP servers/clients that send/receive data of known size (`Content-Length`) in the Arrow IPC streaming format and support range requests (`Accept-Ranges: bytes`).

‎http/get_range/curl/.gitignore

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
*.arrows
19+
*.arrows.part*

‎http/get_range/curl/client/README.md

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# HTTP GET Arrow Data: Range Request curl Client Example
21+
22+
This directory contains examples of `curl` commands that send HTTP GET requests with the `Range` request header.
23+
24+
To run this example, first start one of the range request server examples in the parent directory, then run the shell commands in `client.sh`.

‎http/get_range/curl/client/client.sh

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#!/bin/sh
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
21+
### Use range requests to download an Arrow IPC stream file in two parts
22+
23+
# Get the length of the file `random.arrows` in bytes
24+
curl -I localhost:8008/random.arrows
25+
# Content-Length: 13550776
26+
27+
# Download the first half of the file to `random.arrows.part1`
28+
curl -r 0-6775388 localhost:8008/random.arrows -o random.arrows.part1
29+
30+
# Download the second half of the file to `random.arrows.part2`
31+
curl -r 6775389-13550776 localhost:8008/random.arrows -o random.arrows.part2
32+
33+
# Combine the two separate files into one file `random.arrows` then delete them
34+
cat random.arrows.part1 random.arrows.part2 > random.arrows
35+
rm random.arrows.part1 random.arrows.part2
36+
37+
# Clean up
38+
rm random.arrows
39+
40+
41+
### Simulate an interrupted download over a slow connection
42+
43+
# Begin downloading the file at 1M/s but interrupt after five seconds
44+
timeout 5s curl --limit-rate 1M localhost:8008/random.arrows -o random.arrows
45+
46+
# Resume the download at 1M/s
47+
curl -C - --limit-rate 1M localhost:8008/random.arrows -o random.arrows
48+
49+
# Clean up
50+
rm random.arrows

‎http/get_range/js/.gitignore

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
/**/node_modules
19+
package-lock.json
20+
package.json
21+
*.arrows

‎http/get_range/js/server/README.md

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# HTTP GET Arrow Data: Range Request Node.js Server Example
21+
22+
The example in this directory shows how to use the Node.js package [`serve`](https://www.npmjs.com/package/serve) (which supports range requests) to serve a static Arrow IPC stream file over HTTP.
23+
24+
To run this example, copy the file `random.arrows` from the directory `data/rand-many-types/` into the current directory:
25+
26+
```sh
27+
cp ../../../../data/rand-many-types/random.arrows .
28+
```
29+
30+
Then start the HTTP server to serve this file:
31+
32+
```sh
33+
npx serve -l 8008
34+
```
35+
36+
> [!NOTE]
37+
> The npm package `serve` _should_ automatically set the `Content-Type` header to `application/vnd.apache.arrow.stream` when serving a file with extension `.arrows`, because [the Arrow IPC stream format is officially registered with IANA](https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream) and most web servers including `serve` use registration data from IANA to determine the media type of a file based on its file extension and set the `Content-Type` header to that media type when serving a file with that extension. However, this is not working with `.arrows` files in the `serve` package, seemingly because of a problem with the npm package [`mimedb`](https://github.com/jshttp/mime-db) which `serve` depends on. So the file `serve.json` is used to set the `Content-Type` header correctly when serving `.arrows` files.

‎http/get_range/js/server/serve.json

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"headers": [
3+
{
4+
"source" : "**/*.arrows",
5+
"headers" : [{
6+
"key" : "Content-Type",
7+
"value" : "application/vnd.apache.arrow.stream"
8+
}]
9+
}
10+
]
11+
}

0 commit comments

Comments
 (0)