Skip to content
This repository was archived by the owner on Dec 10, 2019. It is now read-only.

Commit f3f7688

Browse files
jisantucChris Brown
jisantuc
authored and
Chris Brown
committed
Include Uploads example notebook
1 parent 0d5a24d commit f3f7688

File tree

4 files changed

+271
-0
lines changed

4 files changed

+271
-0
lines changed

‎examples/Uploads.ipynb

+159
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Uploading imagery"
8+
]
9+
},
10+
{
11+
"cell_type": "code",
12+
"execution_count": null,
13+
"metadata": {},
14+
"outputs": [],
15+
"source": [
16+
"from rasterfoundry.api import API\n",
17+
"from rasterfoundry.models.upload import Upload\n",
18+
"refresh_token = '<your refresh token>'\n",
19+
"api = API(refresh_token=refresh_token)"
20+
]
21+
},
22+
{
23+
"cell_type": "markdown",
24+
"metadata": {},
25+
"source": [
26+
"The python client makes it easy to upload imagery to Raster Foundry\n",
27+
"from a variety of sources. There are three ways Raster Foundry can\n",
28+
"import data -- from local files, from S3, or from the Planet Labs API.\n",
29+
"\n",
30+
"This notebook will walk through creating uploads from local files.\n",
31+
"\n",
32+
"You'll need to make sure that Raster Foundry has permission to read from\n",
33+
"the bucket. This is easy using the `aws` module:"
34+
]
35+
},
36+
{
37+
"cell_type": "code",
38+
"execution_count": null,
39+
"metadata": {},
40+
"outputs": [],
41+
"source": [
42+
"from rasterfoundry.aws import s3\n",
43+
"bucket_name = '<your bucket name here>'\n",
44+
"s3.authorize_bucket(bucket_name)"
45+
]
46+
},
47+
{
48+
"cell_type": "markdown",
49+
"metadata": {},
50+
"source": [
51+
"You'll also need a little bit more information about the upload you're planning:\n",
52+
"\n",
53+
"- the path to your data (as a glob string or as a list)\n",
54+
"- the id of the datasource to link the upload to\n",
55+
"- your organization id\n",
56+
"- optionally a project id\n",
57+
"\n",
58+
"For this example, I'll use the UAV 3 Band datasource. You can browse more\n",
59+
"datasources in the Raster Foundry web application under `Imports`."
60+
]
61+
},
62+
{
63+
"cell_type": "code",
64+
"execution_count": null,
65+
"metadata": {
66+
"collapsed": true
67+
},
68+
"outputs": [],
69+
"source": [
70+
"# id for UAV 3 Band datasource\n",
71+
"datasource_id = 'c14c8e97-ba85-4677-ac9c-069cfef1f0b1'\n",
72+
"source_files = 'path/to/data/*.tif'\n",
73+
"# or:\n",
74+
"# source_files = ['path1', 'path2', 'path3', ...]"
75+
]
76+
},
77+
{
78+
"cell_type": "markdown",
79+
"metadata": {},
80+
"source": [
81+
"Fill in your organization id below and desired S3 bucket and prefix below (you can just reuse `bucket_name` if you used it above):"
82+
]
83+
},
84+
{
85+
"cell_type": "code",
86+
"execution_count": null,
87+
"metadata": {
88+
"collapsed": true
89+
},
90+
"outputs": [],
91+
"source": [
92+
"organization_id = ''\n",
93+
"s3_bucket = '' or bucket_name\n",
94+
"s3_prefix = 'raster-foundry-imagery'"
95+
]
96+
},
97+
{
98+
"cell_type": "markdown",
99+
"metadata": {},
100+
"source": [
101+
"If you leave `dry_run=True`, this won't actually upload the files to S3. This is useful if they're already there, or if you're just experimenting. If you'd like to upload the files to S3, either remove `dry_run` or set it to `False`."
102+
]
103+
},
104+
{
105+
"cell_type": "code",
106+
"execution_count": null,
107+
"metadata": {},
108+
"outputs": [],
109+
"source": [
110+
"uc = Upload.upload_create_from_files(\n",
111+
" datasource=datasource_id,\n",
112+
" organization=organization_id,\n",
113+
" paths_to_tifs=source_files,\n",
114+
" dest_bucket=s3_bucket,\n",
115+
" dest_prefix=s3_prefix,\n",
116+
" dry_run=True\n",
117+
")\n",
118+
"uc"
119+
]
120+
},
121+
{
122+
"cell_type": "markdown",
123+
"metadata": {},
124+
"source": [
125+
"You can then `POST` your upload to kick off processing:"
126+
]
127+
},
128+
{
129+
"cell_type": "code",
130+
"execution_count": null,
131+
"metadata": {},
132+
"outputs": [],
133+
"source": [
134+
"Upload.create(api, uc)"
135+
]
136+
}
137+
],
138+
"metadata": {
139+
"kernelspec": {
140+
"display_name": "Python 2",
141+
"language": "python",
142+
"name": "python2"
143+
},
144+
"language_info": {
145+
"codemirror_mode": {
146+
"name": "ipython",
147+
"version": 2
148+
},
149+
"file_extension": ".py",
150+
"mimetype": "text/x-python",
151+
"name": "python",
152+
"nbconvert_exporter": "python",
153+
"pygments_lexer": "ipython2",
154+
"version": "2.7.12"
155+
}
156+
},
157+
"nbformat": 4,
158+
"nbformat_minor": 2
159+
}

‎rasterfoundry/aws/__init__.py

Whitespace-only changes.

‎rasterfoundry/aws/s3.py

+98
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
import json
2+
3+
import boto3
4+
from botocore.exceptions import ClientError
5+
6+
7+
s3 = boto3.client('s3')
8+
9+
10+
RF_ACCESS_POLICY = {
11+
'Sid': 'RasterFoundryReadWriteAccess',
12+
'Effect': 'Allow',
13+
'Principal': {
14+
'AWS': 'arn:aws:iam::615874746523:root'
15+
},
16+
'Action': [
17+
's3:GetObject',
18+
's3:ListBucket',
19+
's3:PutObject'
20+
],
21+
'Resource': [
22+
'arn:aws:s3:::{}',
23+
'arn:aws:s3:::{}/*'
24+
]
25+
}
26+
27+
28+
def authorize_bucket(bucket_name):
29+
"""Authorize Raster Foundry to read and write from an S3 bucket
30+
31+
Args:
32+
bucket_name (str): the name of the bucket to authorize
33+
34+
Returns:
35+
int: the status code from the attempted policy change
36+
"""
37+
38+
rf_access_policy = RF_ACCESS_POLICY.copy()
39+
rf_access_policy['Resource'] = [
40+
x.format(bucket_name) for x in rf_access_policy['Resource']
41+
]
42+
43+
try:
44+
resp = s3.get_bucket_policy(Bucket=bucket_name)
45+
existing_policy = json.loads(resp['Policy'])
46+
except ClientError:
47+
existing_policy = {
48+
'Version': '2012-10-17',
49+
'Statement': []
50+
}
51+
52+
existing_policy['Statement'].append(rf_access_policy)
53+
new_policy_str = json.dumps(existing_policy)
54+
return s3.put_bucket_policy(
55+
Bucket=bucket_name, Policy=new_policy_str
56+
)['ResponseMetadata']['HTTPStatusCode']
57+
58+
59+
def unauthorize_bucket(bucket_name):
60+
"""Remove Raster Foundry authorization from a bucket
61+
62+
Args:
63+
bucket_name (str): the name of the bucket to unauthorize
64+
65+
Returns:
66+
int: the status code from the attempted policy change
67+
"""
68+
rf_access_policy = RF_ACCESS_POLICY.copy()
69+
rf_access_policy['Resource'] = [
70+
x.format(bucket_name) for x in rf_access_policy['Resource']
71+
]
72+
73+
try:
74+
resp = s3.get_bucket_policy(Bucket=bucket_name)
75+
existing_policy = json.loads(resp['Policy'])
76+
except ClientError:
77+
existing_policy = {
78+
'Version': '2012-10-17',
79+
'Statement': []
80+
}
81+
82+
if rf_access_policy in existing_policy['Statement']:
83+
new_statement = [
84+
x for x in existing_policy['Statement'] if x != rf_access_policy
85+
]
86+
existing_policy['Statement'] = new_statement
87+
if new_statement:
88+
new_policy_str = json.dumps(existing_policy)
89+
resp = s3.put_bucket_policy(
90+
Bucket=bucket_name, Policy=new_policy_str
91+
)['ResponseMetadata']['HTTPStatusCode']
92+
else:
93+
resp = s3.delete_bucket_policy(Bucket=bucket_name)
94+
else:
95+
# No work to do, so just create a mock response
96+
resp = {'ResponseMetadata': {'HTTPStatusCode': 204}}
97+
98+
return resp['ResponseMetadata']['HTTPStatusCode']

‎rasterfoundry/models/project.py

+14
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,20 @@ def __init__(self, project, api):
3636
self.name = project.name
3737
self.id = project.id
3838

39+
@classmethod
40+
def create(cls, api, project_create):
41+
"""Post a project to Raster Foundry
42+
43+
Args:
44+
api (API): API to use for requests
45+
project_create (dict): post parameters for /projects. See
46+
project_create
47+
48+
Returns:
49+
Project: created object in Raster Foundry
50+
"""
51+
return api.client.Imagery.post_projects(project_create)
52+
3953
def get_center(self):
4054
"""Get the center of this project's extent"""
4155
coords = self._project.extent.get('coordinates')

0 commit comments

Comments
 (0)