diff --git a/_posts/2021-05-28-crowdsourcing-safer-bicyle-lane-design.md b/_posts/2021-05-28-crowdsourcing-safer-bicyle-lane-design.md deleted file mode 100644 index 663c533..0000000 --- a/_posts/2021-05-28-crowdsourcing-safer-bicyle-lane-design.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -date: 2021-05-28 -title: "Crowdsourcing safer bicycle lane design (an eight month update)" -description: "Time to dust off the helmet cameras and take part in some cycling capture challenges." -categories: treks -tags: [bike, data bike, DMAMPO, Minnesota, Sustrans, Bristol, Des Moines] -author_staff_member: dgreenwood -image: /assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-meta.jpg -featured_image: /assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-sm.jpg -layout: post -published: true -redirect_from: - - /blog/2021/crowdsourcing-safer-bicyle-lane-design ---- - -**Time to dust off the helmet cameras and take part in some cycling capture challenges.** - -If you've been following our blog over the years you will have seen a few post with a focus on capturing 360 tours of bicycle lanes, using cameras and a variety of other sensors; - -* [My DIY Street View Bike (2019)](/blog/diy-street-view-bike-tours) -* [DIY Street View Bike v2 (2020)](/blog/diy-street-view-bike-v2) -* [Measuring the Condition of Cycle Paths With Only Your Phone](/blog/measuring-condition-cycle-paths-phone) -* [Trek View Trail Quality Index (2021)](/blog/trek-view-ride-quality-index) -* [Trek View Trail Quality Index: An Alternative Method Using Computer Vision (2021)](/blog/trek-view-ride-quality-index-computer-vision-part-1) - -We've now seen a ton imagery captured on bikes... - -## From Amsterdam (NL): - - - -_Credit: David Greenwood_ - -## To Bristol (UK): - - - -_Credit: Jim Gayes for [Sustrans](https://www.sustrans.org.uk/)._ - -## To Des Moines (USA) - - - -_Credit: DMAMPO_ - -## To You? - -We have been very inspired by how the [Slow Ways](https://beta.slowways.org/) team has managed to mobilise an entire country to map walking trails in the UK. - -In a similar vein, our aim is to get the millions of cyclists around the world capturing their local cycling infrastructure to help drive improvements. - -The beauty of it being, you don't need too much kit to get started, nor do you need to be particularly technical. - -1. Grab a 360 camera -2. Capture your imagery -3. Upload your videos / photos - -[We've created some guides for those just starting out](/trek-pack/). - -Using what we learn analysing the images from locations around the world, our aim is to provide local governments with the data to make the best investment decisions for new bicycle infrastructure to eliminate all avoidable cyclist deaths. - -We can provide the software, but we need groups who can help us to promote this initiative amongst their local cycling communities. If you manage or are involved in managing cycling infrastructure, [please do get in contact](/contact). \ No newline at end of file diff --git a/_posts/2022-03-25-creating-360-street-level-maps-europe-ski-resorts-part-1.md b/_posts/2022-03-25-creating-360-street-level-maps-europe-ski-resorts-part-1.md deleted file mode 100644 index 7d5cef0..0000000 --- a/_posts/2022-03-25-creating-360-street-level-maps-europe-ski-resorts-part-1.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -date: 2022-03-25 -title: "Creating 360 Tours of the Best Ski Resorts in France (Part 1)" -description: "I look back at my imagery I've captured this season and show how you can do the same." -categories: treks -tags: [GoPro, MAX, battery, ski, winter, Alps, mountain] -author_staff_member: dgreenwood -image: /assets/images/blog/2022-03-25/tignes-street-level-view.png -featured_image: /assets/images/blog/2022-03-25/tignes-street-level-view.png -layout: post -published: true -redirect_from: - - /blog/2022/creating-360-street-level-maps-europe-ski-resorts-part-1 ---- - -**I look back at my imagery I've captured this season and show how you can do the same.** - -[A few weeks ago I talked about the problems cold temperatures have on LiIon batteries, and how I was preparing my kit for cold weather](/blog/keeping-gopro-max-warm-extend-battery-life). - -As those that know me might have guessed, this was in preperation for going skiing! - -Here is some of the piste-level imagery I captured in the French Alps this year... - -## Val Thorens, Meribel, and Courchevel - -Val Thorens, Meribel, and Courchevel map - -[Explore it all here](https://www.mapillary.com/app/?lat=45.295186218182&lng=6.5789666618182&z=10&username%5B%5D=trekviewhq). - -## La Plagne and Les Arcs - -La Plagne and Les Arcs map - -[Explore it all here](https://www.mapillary.com/app/?lat=45.510255285185&lng=6.70060485&z=10&username%5B%5D=trekviewhq). - -## Chamonix Valley - -Chamonix Valley map - -[Explore it all here](https://www.mapillary.com/app/?lat=45.96727777239363&lng=6.874316773334613&z=10&username%5B%5D=trekviewhq). - -## Tignes - -Tignes and Val d'Isere map - -[Explore it all here](https://www.mapillary.com/app/?lat=45.452570735577&lng=6.8979346884615&z=10&username%5B%5D=trekviewhq). - -## How can I do this? - -Hopefully this has inspired many of you to try to record your own footage this season and place it on the map. The good news; it's not as difficult as you might think. - -Here is everything you need to know; - -1. [Buy a GoPro MAX and helmet mount](/blog/announcing-trek-pack-v2) -2. [Set it up so that it performs well in cold weather](/blog/keeping-gopro-max-warm-extend-battery-life) -3. Record footage in video mode (5.2k @ 24 FPS) -4. [Break the footage up into frames (optional, not required if only for upload to Street View / Mapillary)](/blog/turn-360-video-into-timelapse-images-part-1) \ No newline at end of file diff --git a/_posts/2022-08-12-open-letter-mapillary-update-api-docs.md b/_posts/2022-08-12-open-letter-mapillary-update-api-docs.md deleted file mode 100644 index 3dc56f5..0000000 --- a/_posts/2022-08-12-open-letter-mapillary-update-api-docs.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -date: 2022-08-12 -title: "An Open Letter to Mapillary: Update Your API Docs, Please" -description: "In this post I will go through some of the challenges (mainly though lack of documentation) with the v4 Mapillary API." -categories: developers -tags: [Mapillary] -author_staff_member: dgreenwood -image: /assets/images/blog/2022-08-12/mapillary-web-api-requests-meta.jpg -featured_image: /assets/images/blog/2022-08-12/mapillary-web-api-requests-sm.jpg -layout: post -published: true -redirect_from: - - /blog/2022/open-letter-mapillary-update-api-docs ---- - -**In this post I will go through some of the challenges (mainly though lack of documentation) with the v4 Mapillary API.** - -[Last year I was excited to hear about the new Mapillary API (v4), and the improvements and the future updates it would bring](/blog/migrating-from-mapillary-api-v3-to-v4). - -Alas, over a year later, [the API Documentation hasn't changed at all](https://www.mapillary.com/developer/api-documentation). - -It is a real shame, because it still lacks many of the key feature of v3 ([docs archived by the Wayback Machine](https://web.archive.org/web/20201106093440/https://www.mapillary.com/developer/api-documentation/)), and is missing endpoints that do exist. - -In this post I documented some of our approaches to cover shortcomings in the Mapillary API and the documentation. - -## How I use Mapillary - -Firstly I upload street-level images and videos. - -To do this I upload a zipped file of sequence images to the Mapillary API. - -Trek View Mapillary Developer App - -To initiate the upload, firstly, [I authenticate to the user to Mapillary](https://www.mapillary.com/developer/api-documentation/#authentication) using our OAuth app. - -Once the user grants my app access to their account (and Mapillary provides an OAuth token) we can start the upload flow. - -I then check to ensure each users token validity before upload and refreshes the token if needed, described in the Mapillary API documentation under refresh flow. - -Now I have a valid token, the upload of images can begin. Note, during uploads Mapillary does not have a concept of Sequences. You simply upload images (in a `.zip` file). Mapillary then processes the uploaded images into one or more Sequence server side. - -I am not 100% clear on how Sequences are created from uploaded images on the Mapillary server (e.g. how far images can be apart in a sequence, etc.), except for the fact that a Sequence can only hold 500 images. Thus, if 600 images, this will result in at least two Mapillary sequences. - -Whilst not documented [the actual upload flow can be deduced from mapillary_tools](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/upload_api_v4.py), which performs an upload of images like so; - -1. [Images in the upload are placed into a .zip file](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/commands/zip.py) -2. [An upload session is created and upload begins to the upload endpoint (currently `https://rupload.facebook.com/mapillary_public_uploads`)](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/upload_api_v4.py) -3. [The upload session is closed](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/upload_api_v4.py#L162) - -The problem is, after closing the upload session, there is no direct link to what sequences (IDs) or images (IDs) are created. This makes sense, because as noted earlier, the Mapillary server now begins the processing of images into Sequences. - -Nor is their a documented endpoint to track the status of an upload and what Sequences are created from it. - -The Mapillary API does provide the following responses at step 3 (from `https://graph.mapillary.com/finish_upload)`: - -``` -{'cluster_id': 'XXXXXXXXXXXXX'} -``` -The `cluster_id` is assume to be related to the Sequence in some way (more on that in a bit). We're not quite sure if it's the same as `sfm_cluster` in the docs (which references the [OpenSFM cluster processed by Mapillary](https://opensfm.org/)). - -This means whilst we can upload images, it appears very hard to tie an uploaded image/sequence uploaded to Mapillary, to corresponding published Mapillary image/sequence. - -I wanted to implement the ability to track Mapillary upload status and the ability for a user to open the corresponding Mapillary sequence (on mapillary.com). - -Therefore I started to look at potential workarounds to implement these two features... - -## Plan A (tl;dr, works, but lots of potential issues) - -To solve this, our first plan was to try an find the Mapillary sequence using information we had (the metadata of the images uploaded). We quickly learned the process to do this is fairly convoluted though. - -First, we can query the [coverage tiles endpoint](https://www.mapillary.com/developer/api-documentation/#coverage-tiles) to get sequence in the vicinity of the images in our uploaded sequences like so; - -```shell -https://tiles.mapillary.com/maps/vtp/mly1_public/2/{z}/{x}/{y} -``` -Where; - -* `{z}`: tile zoom level (always 14) -* `{x}`: x Cartesian coordinate -* `{y}`: y Cartesian coordinate - -p.s. If you're new to XYZ tiles, [this post](https://developers.planet.com/docs/planetschool/xyz-tiles-and-slippy-maps/) and [this post](https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames#X_and_Y) will prove very helpful. For even more on the subject, the [Mapbox Cookbook by Bill Kastanakis](https://www.packtpub.com/product/mapbox-cookbook/9781784397357) will prove another useful resource. - -To find the x,y co-ordinates of the tile, we use the middle image in the sequence (so we can hopefully capture the most images either side) and convert the latitude and longitude to x,y; - -``` -n = 2 ^ zoom -xtile = n * ((lon_deg + 180) / 360) -ytile = n * (1 - (log(tan(lat_rad) + sec(lat_rad)) / π)) / 2 -``` - -[A Python code snippet for this calculation is also included here](https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames#Lon..2Flat._to_tile_numbers_2). - -This request will then provide a response with zero or more sequences (`sequence_id`) found in that tile. - -If the response contains zero `sequence_id`s, the assumption being the sequence is still processing (or has failed). - -Assuming the response does contain one or more `sequence_id`s, using each `sequence_id` returned, we can then query the [Sequence endpoint](https://www.mapillary.com/developer/api-documentation/#sequence) for each returned `sequence_id`. - -``` -https://graph.mapillary.com/image_ids?sequence_id=XXX -``` - -Each response will contain the `image_ids` for the images in the Mapillary Sequence. The [Image endpoint can then be queried to get information about each image in that sequence](https://www.mapillary.com/developer/api-documentation/#image) - -``` -https://graph.mapillary.com/:image_id -``` - -This needs to be done for each image in the Mapillary sequence for best accuracy. - -The response of this endpoint includes `captured_at`, `altitude`, and `geometry` (a GeoJSON Point geometry object containing the latitude and longitude of the image). - -Using this the latitude+longitude+altitude+time of each uploaded image could be compared to these values returned for each image on Mapillary API. - -If a match exist, we can be fairly certain that the image on Mapillary is the same as the one from the upload. - -The response of the Image endpoint also contains `sequence_id` which means if a match exists we can then link each uploaded image to both a Mapillary image ID and a Sequence on Mapillary. - -Whilst this flow works (and is mostly documented), we decided that this approach was not viable; - -* It is excessive. The amount of requests needed to iterate through (potentially) thousands of images is problematic for both client and server at scale -* I have a suspicion Mapillary adjusts the co-ordinates of some images. In cases where this happens, this approach will fail. -* It assumes no other images have same x,y co-ordinates or time (which I can be 99% certain of, barring camera error where same location is reported). -* There is no information about the status of the Sequence upload. The assumption is that if the images don't match between Mapillary and the upload that the sequence is still uploading. Mapillary usually publishes sequences within 72 hours, so we could also assume if an image does not appear within 72 hours an error has occurred -- though this would be a massive assumption... and one I was not comfortable with. - -As such, we came up with a Plan B by watching the request of Mapillary Web. - -## Plan B (tl;dr, still hacky, but works, our current approach) - -This approach required using Chrome Developer tools to watch the requests the Mapillary Web app made to the Mapillary APIs. - -Mapillary API Chrome Dev tools - -One of the first things we noticed when on the user page and opening the sidebar in the Mapillary web app, e.g. https://www.mapillary.com/app/user/trekviewhq, was that a request to get new sequences accepted a username variable (and returned a user id in the response -- needed for other endpoints). - -``` -https://graph.mapillary.com/graphql?doc=query getNewSequences -``` - -For us, [and others](https://forum.mapillary.com/t/how-to-query-user-info-from-v4/5378), the lack of ability to get information for an authenticated user is problematic. [This did exist in v3](https://web.archive.org/web/20201106093440/https://www.mapillary.com/developer/api-documentation/). - -One advantage I have is being able to ask the user (me) for their username during the OAuth process (and keep our fingers crossed they enter it correctly -- because there is no way to validate it is correct, remember, no endpoint for user info!). - -Using the username we tried to query the `getNewSequences` using the `username` variable. - -Here comes the first issue. This endpoint, like all Mapillary API endpoint required authentication. We tried to use our applications `access_token`, but were denied. - -We then noticed Mapillary hardcoded their own `access_token` in all requests from the web app to the API. Using their `access_token` we were able to successfully get a response from this endpoint. - -This returns a response like so; - -```json - -{ - "data": { - "user_by_username": { - "id": "101176865463974", - "new_sequences": { - "sequence_keys": [ - "T1jwvlYndsWVzhuDHOSb7i", - "UmFYLgu9n15aBxXbiQH6vs", - "BsUTIV4RDXhidW7aM6Zvte", - "Y52dBvChfEaUlLt0TyGMmP", - "LNShFtI5X9EnqZc07dDRYg", - "5wiWDzQ1oBrlAeFb27gTER", - "eBcQxLG1sKJalMomPOYjfU", - "updPmsU57SQ9T3MlhHWaLV" - ], - "geojson": "{XXXXXXXXX}", - "__typename": "MLYNewSequencesData" - }, - "__typename": "User" - }, - "__typename": "Query" - }, - "extensions": { - "is_final": true - } -} -``` - -[Full example response for trekviewhq](https://gist.github.com/himynamesdave/c940d1eb5311b74e14014e5aed102853). - -The endpoint returns a `user_by_username` object which contains `id` (needed for many of the other endpoints). The trekviewhq user id is 101176865463974. - -One of the endpoints we identified in the web app using the `user_id` in requests was the `latestActivity` endpoint (also undocumented) which returns the uploads from the user sorted by most recent first (note, the response is paginated, you can use the `first` and `after` parameters to page through the response). - -``` -https://graph.mapillary.com/graphql?doc=query getLatestActivity -``` - -This request takes variables: - -* `id`; the user ID obtained at last step (for trekviewhq = 101176865463974) -* `first`; the number of items (`nodes`) to be returned (I think) -- the Mapillary web app uses `50` -* `after`; seems to be some sort of ID but no idea for what -- for example, when scrolling to load more results in the sidebar, a request was made with the value for this variable `AQHRAAK9wQE4i9s139Bh_DJev3-CV-L_o9SmWd6lHElM3SI2-BW5djTqY-dphpwUdoskjd_4nBTCS58-oz6ni8RSbA` - -In the response from this endpoint, a `node` object with a `cluster_id` is included (remember, a `cluster_id` is also included in the response of the upload). - -```json -{ - "data":{ - "fetch__User":{ - "id":"XXXXXXXXXXX", - "feed":{ - "page_info":{ - "start_cursor":null, - "end_cursor":null, - "has_next_page":false, - "has_previous_page":false - }, - "nodes":[ - { - "cluster_id":"XXXXXXXXXXX", - "type":"UPLOAD", - "thumb_url":null, - "item_count":null, - "image_id":null, - "status":"IN_PROGRESS", - "initial_processing_status":"IN_PROGRESS", - "anonymization_status":"IN_PROGRESS", - "tiler_status":"IN_PROGRESS", - "error_code":"UNSET", - "__typename":"ClusterLatestActivityItem" - } - ], - "__typename":"UserFeedConnection" - }, - "__typename":"User" - }, - "__typename":"Query" - }, - "extensions":{ - "is_final":true - } -} -``` - -[Here is a full sample response for trekviewhq](https://gist.github.com/himynamesdave/f968885b1b60fdbea33bd11e0dd67dbd). - -With this information we can compare each upload reported by the API (each `node`) for a user against the upload `cluster_id`s for that user (that we got when the upload was closed). If we get a match we can link the sequence on Mapillary to an upload. - -The `cluster_id` object also contains the upload state of the sequence, used to power one of the newer features in the Mapillary UI shows you the state of a sequence upload. - -Mapillary Web UI Sequence Status - -It shows four steps and their status; failed, success, or pending. - -1. Image ingestion -2. Image processing (`initial_processing_status`) -3. Map data processing (`anonymization_status`) -4. Map update (`tiler_status`) - -This information allows us to 1) link images uploaded by a user to an Image ID / Sequence ID on Mapillary, and 2) track the status of sequence uploaded to Mapillary. - -The response also contains a `error_code` property, allowing the ability to identify the reason for potential failures. - -If you're interested, [here is our proof-of-concept for this workflow](https://gist.github.com/himynamesdave/2e2048790096e4e41e845c839fc02717). - -It's still not perfect, but solves our use-case and so far is proving stable. My biggest worry is non of these endpoints are documented, and therefore assumed not supported (and could change at any time). - -## Dear Meta, Mapillary, - -If you don't plan to continue development on the API or its documentation, just let us know -- we can continue to use our workarounds. - -It would be incredibly helpful to have some clarity on what we can expect in the future (good or bad) so we can make design decisions accordingly. - -Yours sincerely, - -Trek View \ No newline at end of file diff --git a/_posts/2023-01-06-creating-360-street-level-maps-europe-ski-resorts-part-2.md b/_posts/2023-01-06-creating-360-street-level-maps-europe-ski-resorts-part-2.md deleted file mode 100644 index 85ab48d..0000000 --- a/_posts/2023-01-06-creating-360-street-level-maps-europe-ski-resorts-part-2.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -date: 2023-01-06 -title: "Creating 360 Tours of the Best Ski Resorts in France and Italy (Part 2)" -description: "I look back at my imagery from this season and show how you can do the same this year." -categories: treks -tags: [GoPro, MAX, ski, winter, Alps, mountain] -author_staff_member: dgreenwood -image: /assets/images/blog/2023-01-06/courmayeur-meta.png -featured_image: /assets/images/blog/2023-01-06/courmayeur-meta.png -layout: post -published: true -redirect_from: - - /blog/2023/creating-360-street-level-maps-europe-ski-resorts-part-2 - - /blog/2023/creating-360-street-level-maps-europe-ski-resorts-part-3 ---- - -**I look back at my imagery from this season and show how you can do the same this year.** - -Another ski season is over, for me anyway. - -It has been another year of bad snow here in Europe, though there were a few very good weeks. - -Here's where I've skied... - -## Serre Chevallier (France) - -Serre Chevallier - -[Explore it all here](https://www.mapillary.com/app/user/trekviewhq?lat=44.935685791896&lng=6.4898815332022&z=17&x=0.47695653858615583&y=0.5627130934789792&zoom=0&focus=photo&pKey=1625747594515652). - -## Courmayeur (Italy) - -Courmayeur - -[Explore it all here](https://www.mapillary.com/app/user/trekviewhq?lat=45.772465592161&lng=6.9122764561441&z=17&focus=map&pKey=1372695603514766&x=0.9047409299604711&y=0.5&zoom=0). - -## La Thuile (Italy) / La Rosiere (France) - -La Thuile and La Rosiere - -[Explore it all here](https://www.mapillary.com/app/user/trekviewhq?lat=45.64851260795024&lng=6.883980008276922&z=13.088877003786871&focus=map&pKey=144406351661676). - -## How can I do this? - -Hopefully this has inspired many of you to try to record your own footage this season and place it on the map. The good news; it's not as difficult as you might think. - -Here is everything you need to know; - -1. [Buy a GoPro MAX and helmet mount](/blog/announcing-trek-pack-v2) -2. [Set it up so that it performs well in cold weather](/blog/keeping-gopro-max-warm-extend-battery-life) -3. Record footage in video mode (5.2k @ 24 FPS) -4. [Break the footage up into frames (optional, not required if only for upload to Street View / Mapillary)](/blog/turn-360-video-into-timelapse-images-part-1) - -Need more inspiration? [Here is some more ski footage from my adventures last year](/blog/creating-360-street-level-maps-europe-ski-resorts-part-1). \ No newline at end of file diff --git a/_posts/2023-02-17-building-street-view-alternative-for-the-outdoors-part-1.md b/_posts/2023-02-17-building-street-view-alternative-for-the-outdoors-part-1.md new file mode 100644 index 0000000..8b268cf --- /dev/null +++ b/_posts/2023-02-17-building-street-view-alternative-for-the-outdoors-part-1.md @@ -0,0 +1,745 @@ +--- +date: 2023-02-17 +title: "Building a Street View Alternative for Exploration... with no money (part 1)" +description: "In this post I will walk you through my thinking about how such an app could be built, and my early experiments." +categories: developers +tags: [YouTube, Meta, Mapillary, Facebook, GoPro, OpenSFM] +author_staff_member: dgreenwood +image: /assets/images/blog/2023-02-17/mapillary-web-api-requests-meta.jpg +featured_image: /assets/images/blog/2023-02-03/mapillary-web-api-requests-meta.jpg +layout: post +published: true +redirect_from: + - +--- + +**In this post I will walk you through my thinking about how such an app could be built, and my early experiments.** + +Since the inception of Trek View I've wanted to build a map platform similar to Street View, but designed for adventurers. + +A Street View alternative that allowed you to share your own adventures captured in 360. + +A Street View alternative that allowed you to surface trails you're interested in so that they can be explored in 360, filtering by how the images were captured (ski, hike, etc.), the elevation (or elevation changes), etc. + +Over the last 4 years I've been doing a lot of research about how to implement the idea, much of which can be found on the blog. That research has made it clear the underlying infrastructure to build something like this is complex, but not impossible. + +With a renewed drive to get something built, the following series of posts will summarise what I've found. The final post will describe the design I've settled on based on the surfaced requirements. + +## What's wrong with Street View or Mapillary? + +Mostly nothing. + +The problem with these platforms is they were designed for street level images of roadways. They do a great job at that. + +Yes, they both work with trail imagery, users can upload photos and videos shot (mostly) anywhere. However, the user experience for those browsing the uploads is far from perfect. + +For me search is the biggest issue. Neither platform offers the ability to search on a specific criteria (e.g. elevation, type of trail, etc.). The assumption is you know where you want to drop into the imagery. For looking up what a store front looks like from an address, or if parking is easy, Mapillary and Street View are perfect. For trails, finding viewpoints or waymarkers is more important which you can't easily get an address or fixed point to search on. + +## What are the challenges with building your own street level image map? + +What seems like a fairly simple tool, a map with images you can drop into, unravels to be very complex (and expensive) once you get under the hood... + +### Data types/size + +My backup of GoPro imagery and video is more than 30Tb. + +To give a basic storage estimate using [Amazon S3 storage](https://aws.amazon.com/s3/pricing/) (I know there are cheaper options), it costs $0.023 per GB for storage. So $0.023 * 30000 = $690/mo! + +Many services also charge for bandwidth usage. For example, you pay for requests made against your S3 buckets and objects. Assuming a user views 30 or 40 images per session, the costs get even higher! + +Another issue with this is simply having a scalable solution for uploading this volume of imagery in a browser (or desktop app). Although this is a fairly trivial problem to solve. + +Finally, if you've ever used ffmpeg to process videos on your PC you've probably heard your fan start screaming. Processing videos is fairly intensive. To put videos on the map, you need to convert them to images which required ffmpeg. Therefore, costs to run an ffmpeg processing server also quickly adds up. + +For someone with a budget of less than $100/mo to run this, following the approach of hosting all the data myself is impossible. + +### Database storage + +This is where the complexity can come in. In Street View you have interconnected blue lines. You can jump between images seemlessly in the interface. + +In the backend, I am assuming Street View and Mapillary use a graph database to join points with relationships. This is fairly easy at a sequence level, that is for images in a timelapse or a single video uploaded by a user. Though how do you define how sequences close to each other are connected? This is defined in the queries. + +Similarly, before images are exposed on the map the metadata for them is processed to ensure accuracy. In the case of Street View, you might notice your images are snapped to existing features that appear on the map (like pathways)... although this is not always correct. + +With time, these queries and processing logic can be tuned, but working spacial data is tough (at least for someone that doesn't work full-time in this area). + +### User interface + +Viewing a single 360 in a panoramic viewer like [Panelleum is easy](https://pannellum.org/). + +Introducing navigation arrows to allow user to move between photos with seemless (and not jerky) transitions is much more difficult. + +Jumping between images on Street View and Mapillary often looks smooth, with a blur effect employed as the frame moves. This seemingly simple feature is actually a little more complex than it might first appear. + +In short, trying to build this from scratch to a level that would be acceptable for a user would not be easy. + +## A hacky plan... + +I'm used to working with limitations like this, and weirdly, quite enjoy it. + +Mapillary is a _fairly_ open tool in many regards; + +* You can upload images and videos with common metadata formats (CAMM and GPMD) using existing open-source tooling ([Mapillary Tools](https://github.com/mapillary/mapillary_tools)) +* It processes both images and video uploads and exposes the computed data server side for each sequence via their API +* There is no charge for accessing images via their API (unlike Street View which is eye-wateringly expensive) +* They offer an open-source panoramic browser ([MapillaryJS](https://mapillary.github.io/mapillary-js/) -- as used in Mapillary web) + +With this in mind, I toyed with the idea of using Mapillary as a backend as follows... + +1. allow user to upload photos or videos shot on GoPro via a Trek View web application + * design decision to be made: is it acceptable to ask user to upload to Mapillary directly via Mapillary Desktop Uploader? +2. the Mapillary processed metadata for each sequence uploaded is stored in Trek View web application +3. a user views images in Trek View web application (which are loaded from Mapillary servers) + +## The (big) risk of this plan... + +The obvious risk with this plan, and I hate it, is that Facebook (aka Mapillary) can kill this product at anytime. + +If Facebook shut-down Mapillary entierly or simply stop allowing users to upload or retrieve images for free my product is dead in the water. + +As this is a hobby project which I'm quite happy to use as a learning experience I am reluctantly happy to overlook this issue. However, I do expect to wake up one day for my map to be broken. To be clear, if this was a commercial project, I would not proceed any further with this approach (I'm putting this warning here for the many I have spoken to considering some form of competing commercial product to Street View or Mapillary. You have been warned!). + +## Researching if this is even possible + +With a plan in mind, now I had to figure out if such an approach was feasible... + +### How to handle uploads + +To keep things simple for an MVP I began with the view a user should upload to Mapillary using the existing Mapillary Desktop Uploader (which is very good, and would be time consuming to emulate). This would require them to have an additional account, for Mapillary, but it would save me the potentially messy issue of upload. + +That said, I wanted to first determine how this could be done if I did indeed decide to build the upload flow myself. + +Unfortunately, Mapillary does not currently have a documented endpoint for uploading photos or videos directly. They do however have a command line tool to handle photo and video uploads, [Mapillary Tools](https://github.com/mapillary/mapillary_tools). + +Install it as follows... + +```shell +mkdir mapillary_tools +cd mapillary_tools +python3 -m venv mapillary_tools_venv +source mapillary_tools_venv/bin/activate +pip3 install mapillary_tools +mapillary_tools --help +``` + +#### Uploading Photos using Mapillary tools + +Upload a timelapse sequence as follows; + +```shell +mapillary_tools process_and_upload \ + --user_name trekviewhq \ + --desc_path tmp/mapillary_description_file_1.json +``` + +A few things: + +* `--user_name` I pass my Mapillary username using the flag. The command line will ask me for a username and password to actually authenticate before uploading any files. +* `--desc_path` I want to store a copy of the description file Mapillary tools creates to help identify the actual sequence Mapillary creates on the server. [Here's the schema of the JSON structure for the description file](https://github.com/mapillary/mapillary_tools/blob/main/schema/image_description_schema.json). + + +Here's an example of image processing: + +```shell +mapillary_tools process_and_upload test_photos/tes-max-003g \ + --user_name trekviewhq \ + --desc_path tmp/mapillary_description_file_1.json +``` + +```txt +Extracting geotags from images: 100%|████| 418/418 [00:01<00:00, 290.19images/s] +Validating metadatas: 100%|███████████| 418/418 [00:00<00:00, 1226.21metadata/s] +2024-02-14 14:50:59,403 - INFO - Checking upload status for 418 metadatas +2024-02-14 14:50:59,429 - INFO - Check the description file for details: tmp/mapillary_description_file.json +2024-02-14 14:50:59,429 - INFO - 418 image(s) read in total +2024-02-14 14:50:59,429 - INFO - 418 image(s) are ready to be uploaded +Uploading IMAGE (1/2): 100%|███████████████| 1.11G/1.11G [01:24<00:00, 14.1MB/s] +Uploading IMAGE (2/2): 100%|█████████████████| 180M/180M [00:12<00:00, 15.0MB/s] +2024-02-14 14:52:43,150 - INFO - 2 IMAGE sequences uploaded +2024-02-14 14:52:43,150 - INFO - 1319.3M data in total +2024-02-14 14:52:43,151 - INFO - 1319.3M data uploaded +``` + +Which creates a `tmp/mapillary_description_file_1.json` as follows (I've removed many of the track-points in the printed output below for brevity in this post); + +```json +[ + { + "filename": "/Users/dgreenwood/Downloads/mapillary_tools/test_photos/tes-max-003g/GSAC5840.JPG", + "md5sum": "655077ac078951d8bb52c38e0f41fb31", + "filetype": "image", + "MAPLatitude": 50.8966219, + "MAPLongitude": -0.5591297, + "MAPCaptureTime": "2021_08_28_10_06_42_000", + "MAPAltitude": 77.42, + "MAPCompassHeading": { + "TrueHeading": 133.54, + "MagneticHeading": 133.54 + }, + "MAPSequenceUUID": "0", + "MAPDeviceMake": "GoPro", + "MAPDeviceModel": "GoPro Max", + "MAPOrientation": 1 + }, + { + "filename": "/Users/dgreenwood/Downloads/mapillary_tools/test_photos/tes-max-003g/GSAC5841.JPG", + "md5sum": "c1bb9c5f383cfc2e838f56b4e9e0a03f", + "filetype": "image", + "MAPLatitude": 50.8966029, + "MAPLongitude": -0.559098, + "MAPCaptureTime": "2021_08_28_10_06_44_000", + "MAPAltitude": 77.231, + "MAPCompassHeading": { + "TrueHeading": 136.336, + "MagneticHeading": 136.336 + }, + "MAPSequenceUUID": "0", + "MAPDeviceMake": "GoPro", + "MAPDeviceModel": "GoPro Max", + "MAPOrientation": 1 + }, + { + "filename": "/Users/dgreenwood/Downloads/mapillary_tools/test_photos/tes-max-003g/GSAC5842.JPG", + "md5sum": "1039fb7c265bf0a73f89cab21765077d", + "filetype": "image", + "MAPLatitude": 50.896584, + "MAPLongitude": -0.5590694, + "MAPCaptureTime": "2021_08_28_10_06_46_000", + "MAPAltitude": 76.932, + "MAPCompassHeading": { + "TrueHeading": 134.813, + "MagneticHeading": 134.813 + }, + "MAPSequenceUUID": "0", + "MAPDeviceMake": "GoPro", + "MAPDeviceModel": "GoPro Max", + "MAPOrientation": 1 + } +] +``` + +Most of it is fairly self-explanatory (again, [schema here](https://github.com/mapillary/mapillary_tools/blob/main/schema/image_description_schema.json)). + +I do however want to point out the `MAPSequenceUUID` property inside each track-point object, as this is very useful for my use case. + +From the schema; + +```json +"MAPSequenceUUID": { + "type": "string", + "description": "Arbitrary key for grouping images", + "pattern": "[a-zA-Z0-9_-]+" +} +``` + +Based on previous experience uploading photos to Mapillary, I know a sequence can only contain 500 images. Thus, if 600 images for a single timelapse, it will probably result in at least two Mapillary sequences. Of course, if I upload 5 distinct captures in one upload that are very far apart in distance or time then the actual sequences generated might be much higher. + +Back to the point, this is not defining the actual Sequence UUID Mapillary will generate on the server. + +In my sequence above of 418 images, 361 are marked with `"MAPSequenceUUID": "0"` and 57 with `"MAPSequenceUUID": "1"`. I'll come back to this later when we look at the processed images, just keep it in mind for now. + +Onto videos... + +#### Uploading Videos using Mapillary tools + +The upload process for videos is very similar to photos; + +```shell +mapillary_tools process_and_upload \ + --user_name trekviewhq \ + --desc_path tmp/mapillary_description_file.json +``` + +For example, to upload a single video: + +```shell +mapillary_tools process_and_upload test_vids/ESAD001v205vid7-trek_view_full_nadir-2_with-nadir.mp4 \ + --user_name trekviewhq \ + --desc_path tmp/mapillary_description_file_2.json +``` + +Below is an example of that file (I've removed many `MAPGPSTrack` objects for brevity in this post); + +```json +[ + { + "filename": "/Users/dgreenwood/Downloads/mapillary_tools/test_vids/ESAD001v205vid7-trek_view_full_nadir-2_with-nadir.mp4", + "md5sum": "e67f95ffe0c10ed700506cadae70089a", + "filetype": "gopro", + "MAPGPSTrack": [ + [ + 0, + -13.942725, + 28.5107049, + 549.857, + null + ], + [ + 0, + -13.9427281, + 28.5107056, + 550.076, + null + ], + [ + 200, + -13.9427339, + 28.5107044, + 550.055, + null + ] + ], + "MAPDeviceMake": "GoPro", + "MAPDeviceModel": "MAX" + } +] +``` + +You'll see the output is different to photos, but much of the data is similar. + +The `MAPGPSTrack` list contains; + +* a time offset of the track-point relative to the start of the video +* Longitude of the track point +* Latitude of the track point +* Altitude of the track point in meters +* Camera angle of the track point, in degrees (no present in my track as it appears this is not extracted from GPMD by Mapillary tools) + +For video uploads there is no specific mention of sequences in the description file. + +#### How Uploads actually work in Mapillary tools + +As I noted earlier, the Mapillary API docs doesn't mention upload endpoints at all, so of course I was intrigued; what is Mapillary Tools using to send the photos and videos to the Mapillary servers? + +As the code for Mapillary Tools is open, this wasn't too hard to figure out. [Here's the part of the code responsible for uploads](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/upload_api_v4.py). + +Whilst not documented [the actual upload flow can be deduced from mapillary_tools](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/upload_api_v4.py), which performs an upload of videos or images like so; + +1. [Images in the upload are placed into a .zip file](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/commands/zip.py). This step does not apply to videos +2. [An upload session is created (for authenticated user) and upload begins to the upload endpoint (currently `https://rupload.facebook.com/mapillary_public_uploads`)](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/upload_api_v4.py) +3. [The upload session is closed](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/upload_api_v4.py) + +The big security issue for baking Mapillary Tools into my own code is that [a user authenticates (to create a session) to Mapillary using their username and password](https://github.com/mapillary/mapillary_tools/blob/main/mapillary_tools/authenticate.py). When they do this on their own machine it poses no direct risk, only their machine sees the raw credentials. With security engineering being my day-job, I didn't want to go anywhere near handling user credentials in my own app. + +So then I wondered, could I use OAuth tokens generated for `https://graph.mapillary.com/token` as [publicly documented](https://www.mapillary.com/developer/api-documentation/#request-authorization) with `https://rupload.facebook.com/mapillary_public_uploads`. + +The answer appears to be, no, because I only ever received unauthorised errors trying this approach. + +### Ignoring authentication issues, this approach is still problematic + +For my software to work, I need to implement the ability to track Mapillary uploads through to the corresponding Mapillary sequence (on mapillary.com). + +The first problem is, after closing the upload session, there is no direct link to what sequences (IDs) or images (IDs) are created from uploads. + +This makes sense, because the Mapillary servers still needs to process the videos or images into actual Sequences. + +That said, there could be a key to link an upload to the final Sequences created from it (e.g. many sequences might have the same upload key). + +Looking a the Mapillary Tools code the Mapillary API does provide a response at step 3 (`https://graph.mapillary.com/finish_upload`) which includes a property called `cluster_id`. + +However, I'm not sure if this is useful at all for what I need. Some quick Googling shows `clusterID` in [MapillaryJS relates to the globally unique id of the SfM cluster to which the image belongs](https://mapillary.github.io/mapillary-js/api/classes/viewer.Image/#clusterid). Keep this in mind... I'll validate if this is in anyway useful later. + +### Using the Mapillary public API to link uploads to sequences + +We have three key bits of known info; + +1. The raw metadata in the photos/videos +2. The description JSON file created by Mapillary Tools (from the photos/videos metadata) +3. The Mapillary username (used to authenticate) + +Looking at the filters on the Mapillary API, it should be possible to find the sequences... + +```shell +GET https://graph.mapillary.com/images?creator_username=trekviewhq&fields=id,altitude,atomic_scale,camera_parameters,camera_type,captured_at,compass_angle,computed_altitude,computed_compass_angle,computed_geometry,computed_rotation,creator,exif_orientation,geometry,height,is_pano,make,model,width +``` + +Here's an example of the useful data that can be returned for an image (I have not included some `fields` not useful for this use-case;) + +```json +{ + "data": [ + { + "id": "1386976845208976", + "altitude": 48.419, + "atomic_scale": 0.86374780888523, + "camera_type": "spherical", + "captured_at": 1683629264000, + "compass_angle": 43.370713358692, + "computed_altitude": 2.7760077845305, + "computed_compass_angle": 165.86355589155, + "computed_geometry": { + "type": "Point", + "coordinates": [ + 4.1447180519741, + 39.865000210806 + ] + }, + "computed_rotation": [ + 0.31811377810602, + -1.8970785219694, + 2.3246601655456 + ], + "creator": { + "username": "trekviewhq", + "id": "101176865463974" + }, + "exif_orientation": 1, + "geometry": { + "type": "Point", + "coordinates": [ + 4.1446528999722, + 39.8650098 + ] + }, + "height": 2880, + "is_pano": true, + "make": "GoPro", + "model": "GoPro Max", + "width": 5760 + } +``` + +The good news is this endpoint also supports more filtering (than just `creator_username` used above). Those useful for my needs are; + +* `start_captured_at`: string: filter images captured after. Specify in the format. For example: "2022-08-16T16:42:46Z" +* `end_captured_at`: string: filter images captured before. Same format as "start_captured_at +* `make`: string, the manufacturer name of the camera device, can only be used when bbox parameter is used. Spaces are allowed. +* `model`: string, the model or product series name of the camera device, can only be used when bbox parameter is used. Spaces are allowed. + +So using these, lets see if I can find the sequence ID for the photo I uploaded by taking the values in the description file. + +In the description file my first photo has a `MAPCaptureTime` = `2021_08_28_10_06_42_000` and last photo `2021_08_28_10_20_36_000`. So we get `2021-08-28T10:06:42.000Z` and `2021-08-28T10:20:36.000Z` for `start_captured_at` and `end_captured_at` time respectively. + +Note on video files, the Mapillary description JSON produced by Mapillary does not include actual times of video GPS. This data will need to be pulled from the metadata of the video itself. The start time of the video can be pulled from the first GPS time, and the end time can be calculated by pulling out the last GPS time in the metadata. + +The `make` shown is `GoPro` and `model` `GoPro MAX` (this IS reported in the Mapillary description JSON) + +Which gives me the following query; + +```shell +GET 'https://graph.mapillary.com/images?creator_username=trekviewhq&make=GoPro&model=GoPro%20Max&start_captured_at=2021-08-28T10:20:36.000Z&end_captured_at=2021-08-28T10:06:42.000Z&fields=id%2Caltitude%2Catomic_scale%2Ccamera_parameters%2Ccamera_type%2Ccaptured_at%2Ccompass_angle%2Ccomputed_altitude%2Ccomputed_compass_angle%2Ccomputed_geometry%2Ccomputed_rotation%2Ccreator%2Cexif_orientation%2Cgeometry%2Cheight%2Cis_pano%2Cmake%2Cmodel%2Cwidth%2Csequence&access_token=REVOKED' +``` + +Returns 418 matches (the same number that were in my upload). + +Here's the first tow results; + +```json +{ + "data": [ + { + "id": "1447075489494186", + "altitude": 48.737, + "atomic_scale": 1, + "camera_type": "spherical", + "captured_at": 1630146036000, + "compass_angle": 4.008, + "computed_altitude": -1.5965052107349, + "computed_compass_angle": 7.4043872167977, + "computed_geometry": { + "type": "Point", + "coordinates": [ + -0.5463367458184, + 50.901142238256 + ] + }, + "computed_rotation": [ + 1.5680816608427, + -0.12745545191882, + 0.075540302347128 + ], + "creator": { + "username": "trekviewhq", + "id": "101176865463974" + }, + "exif_orientation": 1, + "geometry": { + "type": "Point", + "coordinates": [ + -0.5463302, + 50.9011354 + ] + }, + "height": 2880, + "is_pano": true, + "make": "GoPro", + "model": "GoPro Max", + "width": 5760, + "sequence": "d7DlxwUTp6JZaIWKHGtnMQ" + }, + { + "id": "1345201159514651", + "altitude": 48.77, + "atomic_scale": 1, + "camera_type": "spherical", + "captured_at": 1630146034000, + "compass_angle": 4.008, + "computed_altitude": -1.4189444249496, + "computed_compass_angle": 7.3760874326204, + "computed_geometry": { + "type": "Point", + "coordinates": [ + -0.54633846065914, + 50.901114286844 + ] + }, + "computed_rotation": [ + 1.5525942807035, + -0.098468494308371, + 0.10331052653854 + ], + "creator": { + "username": "trekviewhq", + "id": "101176865463974" + }, + "exif_orientation": 1, + "geometry": { + "type": "Point", + "coordinates": [ + -0.5463332, + 50.9011084 + ] + }, + "height": 2880, + "is_pano": true, + "make": "GoPro", + "model": "GoPro Max", + "width": 5760, + "sequence": "d7DlxwUTp6JZaIWKHGtnMQ" + } +``` + +Results appear to be sorted `captured_at` epoch time descending (put another way; the most recent photos first). + +In this case we get the same number of published photos as were included in the upload. Of course this might not always be true. It is completely possible for some photos to be discarded if they fail Mapillary's automated checks (e.g. are corrupt). + +The response also shows us 361 of the images belong to `"sequence": "s1lP92nCJwpxOaMEWRDX8b"` and 57 to `"sequence": "d7DlxwUTp6JZaIWKHGtnMQ"`. So this upload has created 2 sets of sequences. + +As an aside, going back to the confusion about `ClusterID` returned in the upload session response, I wanted to see what each images `sfm_cluster` was. + +```shell +GET 'https://graph.mapillary.com/images?creator_username=trekviewhq&make=GoPro&model=GoPro%20Max&start_captured_at=2021-08-28T10:20:36.000Z&end_captured_at=2021-08-28T10:06:42.000Z&fields=id%2Csfm_cluster&access_token=REVOKED' +``` + +Turns out the cluster ID of the response has nothing to do with SFM clusters as far as I can tell. There were XXX distinct `sfm_cluster.id` values as follows; + +* `928869945563125` -- 57 +* `783706370299876` -- 61 +* `408458911861505` -- 75 +* `1107885223743223` - 75 +* `709720021284571` -- 75 +* `1350423708991704` -- 75 + +In short `ClusterID` in upload != `sfm_cluster.id` returned via API, thus doesn't help us. Ah well. + +My API query assumes two things: + +1. we know the user which limits the images (which we do, b/c the assumption is they use Mapillary Tools) +2. the user does not have other photos shot at the same time. Now, this is not generally common; how can a user shoot many photos at the same time? True. Though keep in mind + * some accounts are used for organisations. As such, one account may upload photos shot in different places by different people at the same time, or + * a single person might shoot with two cameras, thus creating two sets of captures with the same time and similar GPS + +2.2 is hard to solve for using the API, and I'd argue is an edge case that can be ignored. + +2.1 is a little more common and possible to solve by improving our API request to include a bounding box `bbox`. + +A bounding box (usually shortened to bbox) is an area defined by two longitudes and two latitudes; + +``` +bbox = min Longitude , min Latitude , max Longitude , max Latitude +``` + +Using the `MAPLatitude` and `MAPLongitude` in the `mapillary_description_file_1.json` it's fairly trivial to create a bbox for the sequence path as follows; + +```python +# python3 path2bbox.py +import json + +# File paths +input_file_path = 'tmp/mapillary_description_file_1.json' + +# Read the JSON data from the file +with open(input_file_path, 'r') as file: + data = json.load(file) + +# Extract MAPLatitude and MAPLongitude and store as a list of points +points = [[item["MAPLatitude"], item["MAPLongitude"]] for item in data] + +def bounding_box_naive(points): + """returns a list containing the bottom left and the top right + points in the sequence + Here, we use min and max four times over the collection of points + """ + min_lat = min(point[1] for point in points) + min_lon = min(point[0] for point in points) + max_lat = max(point[1] for point in points) + max_lon = max(point[0] for point in points) + + return min_lon,min_lat,max_lon,max_lat + +# Calculate the bounding box +bbox = bounding_box_naive(points) + +# Print the bottom left and top right coordinates in the desired format +print(bbox) +``` + +Which prints; + +```json +50.8963246,-0.5591297,50.9011354,-0.5463302 +``` + +You can see this bounding box printed using this very handy website: http://bboxfinder.com/#50.896325,-0.559130,50.901135,-0.546330 + +Which gives us the following query; + +```shell +GET 'https://graph.mapillary.com/images?bbox=50.8963246,-0.5591297,50.9011354,-0.5463302&creator_username=trekviewhq&make=GoPro&model=GoPro%20Max&start_captured_at=2021-08-28T10:20:36.000Z&end_captured_at=2021-08-28T10:06:42.000Z&fields=id%2Caltitude%2Catomic_scale%2Ccamera_parameters%2Ccamera_type%2Ccaptured_at%2Ccompass_angle%2Ccomputed_altitude%2Ccomputed_compass_angle%2Ccomputed_geometry%2Ccomputed_rotation%2Ccreator%2Cexif_orientation%2Cgeometry%2Cheight%2Cis_pano%2Cmake%2Cmodel%2Cwidth%2Csequence&access_token=REVOKED' +``` + +Which returns the same results as before, however is more accurate in terms of getting only the images that were part of a specific upload. + +### The problem (and solution) with sequence states + +One thing I've ignored here is the time it takes from Mapillary to process and publish an upload. Sometimes this can take hours, other times, days. + +All of the above queries will only return data once the upload has been published. This is somewhat problematic, especially when it comes to failed uploads. + +If a video or photo sequence uploaded is never published it will never return results, but you will be unable to tell if the upload actually failed or whether it's still being published. + +In the Mapillary web app, you can see the state of an upload so this data must be exoposed somewhere... + +Mapillary API Web + +So I turned to Chrome Developer tools to watch the requests the Mapillary Web app made to the Mapillary APIs. + +Mapillary API Chrome Dev tools + +One of the first things we noticed when on the user page and opening the sidebar in the Mapillary web app, e.g. https://www.mapillary.com/app/user/trekviewhq, was that a request to get new sequences accepted a username variable (and returned a user id in the response -- needed for other endpoints). + +``` +https://graph.mapillary.com/graphql?doc=query getNewSequences($username: trekviewhq) +``` + +Using the username we tried to query the `getNewSequences` using the `username` variable. + +Here comes the first issue. This endpoint, like all Mapillary API endpoint required authentication. We tried to use our applications `access_token`, but were denied with an unauthorised response. + +We then noticed Mapillary hardcoded their own `access_token` in all requests from the web app to the API. Using their `access_token` we were able to successfully get a response from this endpoint. + +This returns a response like so; + +```json + +{ + "data": { + "user_by_username": { + "id": "101176865463974", + "new_sequences": { + "sequence_keys": [ + "T1jwvlYndsWVzhuDHOSb7i", + "UmFYLgu9n15aBxXbiQH6vs", + "BsUTIV4RDXhidW7aM6Zvte", + "Y52dBvChfEaUlLt0TyGMmP", + "LNShFtI5X9EnqZc07dDRYg", + "5wiWDzQ1oBrlAeFb27gTER", + "eBcQxLG1sKJalMomPOYjfU", + "updPmsU57SQ9T3MlhHWaLV" + ], + "geojson": "{XXXXXXXXX}", + "__typename": "MLYNewSequencesData" + }, + "__typename": "User" + }, + "__typename": "Query" + }, + "extensions": { + "is_final": true + } +} +``` + +[Full example response for trekviewhq](https://gist.github.com/himynamesdave/c940d1eb5311b74e14014e5aed102853). + +The endpoint returns a `user_by_username` object which contains `id` (needed for many of the other endpoints). The trekviewhq user id is `101176865463974`. + +One of the endpoints we identified in the web app using the `user_id` in requests was the `latestActivity` endpoint (also undocumented) which returns the uploads from the user sorted by most recent first (note, the response is paginated, you can use the `first` and `after` parameters to page through the response). + +``` +https://graph.mapillary.com/graphql?doc=query getLatestActivity($id: 101176865463974, $first: 50, $after: AQHRAAK9wQE4i9s139Bh_DJev3-CV-L_o9SmWd6lHElM3SI2-BW5djTqY-dphpwUdoskjd_4nBTCS58-oz6ni8RSbA) +``` + +This request takes variables: + +* `id`; the user ID obtained at last step (for trekviewhq = 101176865463974) +* `first`; the number of items (`nodes`) to be returned (I think) -- the Mapillary web app uses `50` +* `after`; seems to be some sort of ID but no idea for what -- for example, when scrolling to load more results in the sidebar, a request was made with the value for this variable `AQHRAAK9wQE4i9s139Bh_DJev3-CV-L_o9SmWd6lHElM3SI2-BW5djTqY-dphpwUdoskjd_4nBTCS58-oz6ni8RSbA` + +In the response from this endpoint, a `node` object with a `cluster_id` is included. This is the same `cluster_id` is also included in the response of the upload -- we now have a key to join upload and sequence in Mapillary! + +```json +{ + "data":{ + "fetch__User":{ + "id":"XXXXXXXXXXX", + "feed":{ + "page_info":{ + "start_cursor":null, + "end_cursor":null, + "has_next_page":false, + "has_previous_page":false + }, + "nodes":[ + { + "cluster_id":"XXXXXXXXXXX", + "type":"UPLOAD", + "thumb_url":null, + "item_count":null, + "image_id":null, + "status":"IN_PROGRESS", + "initial_processing_status":"IN_PROGRESS", + "anonymization_status":"IN_PROGRESS", + "tiler_status":"IN_PROGRESS", + "error_code":"UNSET", + "__typename":"ClusterLatestActivityItem" + } + ], + "__typename":"UserFeedConnection" + }, + "__typename":"User" + }, + "__typename":"Query" + }, + "extensions":{ + "is_final":true + } +} +``` + +[Here is a full sample response for trekviewhq](https://gist.github.com/himynamesdave/f968885b1b60fdbea33bd11e0dd67dbd). + +With this information we can compare each upload reported by the API (each `node`) for a user against the upload `cluster_id`s for that user (that we got when the upload was closed). If we get a match we can link the sequence on Mapillary to an upload. + +The `cluster_id` object also contains the upload state of the sequence (as shown in the Mapillary UI screenshot earlier in this post). + +It shows four steps and their status; failed, success, or pending. If all four pass, the sequence gets published. + +1. Image ingestion +2. Image processing (`initial_processing_status`) +3. Map data processing (`anonymization_status`) +4. Map update (`tiler_status`) + +This information allows us to 1) link images uploaded by a user to an Image ID / Sequence ID on Mapillary, and 2) track the status of sequence uploaded to Mapillary. + +The response also contains a `error_code` property, allowing the ability to identify the reason for potential failures. + +## In summary... + +Mapillary Tools and the Mapillary API allow us to upload files, find the uploads using the Mapillary API endpoints, store the Mapillary metadata in Trek View, and then finally render the points/sequences on the map (rendering the actual images from the server). + +However, if you've read this far you will probably be thinking; it's a very disjointed approach. My biggest worry is the use of the undocumented Mapillary APIs. + +As such, I decided, as per my original inclination, it would be better not to handle any uploads via my app, at least for the MVP. + +My plan, allow a user to import their sequences to Mapillary once they've been published. Said plan is still coming together, but lookout for my next post in two weeks where I'll divulge everything! \ No newline at end of file diff --git a/_posts/2023-03-02-building-street-view-alternative-for-the-outdoors-part-2.md b/_posts/2023-03-02-building-street-view-alternative-for-the-outdoors-part-2.md new file mode 100644 index 0000000..d10986e --- /dev/null +++ b/_posts/2023-03-02-building-street-view-alternative-for-the-outdoors-part-2.md @@ -0,0 +1,19 @@ +--- +date: 2023-03-02 +title: "Building a Street View Alternative for the Great Outdoors" +description: "" +categories: developers +tags: [] +author_staff_member: dgreenwood +image: /assets/images/blog/2023-03-02/ +featured_image: /assets/images/blog/2023-03-02/ +layout: post +published: false +redirect_from: + - +--- + + +OK + + diff --git a/_posts/2023-08-25-open-sourcing-gopro2gsv.md b/_posts/2023-08-25-open-sourcing-gopro2gsv.md deleted file mode 100644 index 2f1066b..0000000 --- a/_posts/2023-08-25-open-sourcing-gopro2gsv.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -date: 2023-08-25 -title: "Open-Sourcing gopro2gsv" -description: "Our internal tool that processes images and videos shot on GoPro 360 cameras and uploads them to Google Street View is now available for everyone to use and modify." -categories: developers -tags: [Google, Street View] -author_staff_member: dgreenwood -image: /assets/images/blog/2023-08-25/ -featured_image: /assets/images/blog/2023-08-25/ -layout: post -published: false -redirect_from: - - ---- - -**Our internal tool that processes images and videos shot on GoPro 360 cameras and uploads them to Google Street View is now available for everyone to use and modify.** - -Trek View started as a project with a simple aim; shoot 360 footage and upload it to Google Street View. - -Not much has really changed over the last 4 years. - -From Trek View's inception, I've attempted to build various tools around Street View, including my ongoing attempt at creating an "off-road Street View". - -In that time the type of images available on Street View and the way they are uploaded has also changed. - -In 2020, Street View only accepted photo uploads (which was somewhat problematic). Street View allowed - - -Then the brialliant Stafford Marquidt showed me the then in beta - -Now those API methods are generally available to anyone. As such I've now decided to release (and open-source) gopro2gsv built on said video upload methods. - -gopro2gsv takes 360 timelapse photos and videos shot on GoPro cameras (currently the Fusion and MAX) and uploads them to Google Street View. You can also use gopro2gsv to optionally add your own nadir to the uploaded content. - -## Videos shot on GoPro - -Currently gopro2gsv works with - - - - -## An insight into our workflow - -Often we will shoot multiple timelapses with thousands of photos, or hour of video footage (often chaptered by the camera into many videos). - -To aid us automate this process we diff --git a/_posts/2023-09-20-climbing-meru.md b/_posts/2023-09-20-climbing-meru.md deleted file mode 100644 index f0437d5..0000000 --- a/_posts/2023-09-20-climbing-meru.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -date: 2023-09-22 -title: "Climb Mount Meru Virtually" -description: "We're back in Tanzania, this time for Kilimanjaro's little brother." -categories: treks -tags: [Mount Meru, Mount Kilimanjaro] -author_staff_member: dgreenwood -image: /assets/images/blog/2023-09-22/P1040574.JPG -featured_image: /assets/images/blog/2023-09-22/P1040574.JPG -layout: post -published: true -redirect_from: - - ---- - -**We're back in Tanzania, this time for Kilimanjaro's little brother.** - -A few years ago we set out on an adventure to map many of the routes up Kilimanjaro, Africa's highest mountain. - -During that expedition, on many occasions we heard about Mount Meru, Kilimanjaro's little brother. - -On a clear day, Meru can be seen from the summit of Kilimanjaro, 70 kilometers to the West, standing at 4,566 meters. - -Mount Meru is Tanzania's second-highest peak and is classed as a dormant volcano, last erupting about 100 years ago (Kilimanjaro is an extinct volcano). - -What really perked our interest in Meru was the fact many classed it as a more technical climb compared to the routes up Kilimanjaro, which are well established and don't require any scrambling. - -Mount Meru Fig Tree - -And so, afew years later, we set out to capture 360 footage of Mount Meru, with support from the [Greg Adventures team](https://gregadventures.com/) led by our professional guide, Saladini Mshana. - -## The Plan - -We set out again with the Trek Pack v2, and a two external powerpacks. [It was exactly the same setup as we used for Kilimanjaro](/blog/climbing-kilimanjaro-part-1-the-kit/). - -Mount Meru Routes - -Meru can be climbed in three days, stopping at the Mirikamba Hut on day one and Saddle Hut at day two. Both days are fairly easy walks, taking around 5 hours each day. On day two, an afternoon hike up to Little Meru (3,820 meters) for acclimatisation is advised. - -On day three, summit attempts usually start at around 0100 to reach Meru's summit in time for sunrise. Day three also involves a long hike down to the base, taking a slightly different route on the final part of the descent after Miriakamba hut. - -Like Kilimanjaro, the environment changes on dramatically on the way up. Starting in the lush tropical forests of Arusha National Park where a whole host of wildlife can be seen (and is the reason why a park ranger with a gun is required on all hikes) to the cold, dry, desert like conditions at the peak. - -If you're considering the hike, here are some of the key parts to it, all of which are available to explore virtually on Street View. - -## Momela Gate - - - -## Fig tree arch - - - -## Miriakamba Hut - - - -## Saddle Hut - - - -## Little Meru (3,820 meters) - - - -## Rhino Point - - - -## Meru's Volcanic Cone - - - -## The Summit (4,566 meters) - - - -Mount Meru Summit - -Mount Meru Kilimanjaro Sunrise - -## How can I do this? - -Hopefully this has inspired many of you to try to record your own footage this season and place it on the map. The good news; it's not as difficult as you might think. - -Here is everything you need to know; - -1. [Build your own Trek Pack](/trek-pack) -3. Record the footage of your adventure in video mode -3. Upload it to Google Street View \ No newline at end of file diff --git a/assets/images/blog/2021-05-28/amsterdam.jpeg b/assets/images/blog/2021-05-28/amsterdam.jpeg deleted file mode 100644 index b8d864d..0000000 Binary files a/assets/images/blog/2021-05-28/amsterdam.jpeg and /dev/null differ diff --git a/assets/images/blog/2021-05-28/bristol.jpeg b/assets/images/blog/2021-05-28/bristol.jpeg deleted file mode 100644 index f6ea0c9..0000000 Binary files a/assets/images/blog/2021-05-28/bristol.jpeg and /dev/null differ diff --git a/assets/images/blog/2021-05-28/iowa.jpeg b/assets/images/blog/2021-05-28/iowa.jpeg deleted file mode 100644 index aa58a26..0000000 Binary files a/assets/images/blog/2021-05-28/iowa.jpeg and /dev/null differ diff --git a/assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-meta.jpg b/assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-meta.jpg deleted file mode 100644 index e13a972..0000000 Binary files a/assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-meta.jpg and /dev/null differ diff --git a/assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-sm.jpg b/assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-sm.jpg deleted file mode 100644 index 3d74055..0000000 Binary files a/assets/images/blog/2021-05-28/map-the-paths-devonshire-tunnel-sm.jpg and /dev/null differ diff --git a/assets/images/blog/2022-03-25/chamonix-map.png b/assets/images/blog/2022-03-25/chamonix-map.png deleted file mode 100644 index 0366948..0000000 Binary files a/assets/images/blog/2022-03-25/chamonix-map.png and /dev/null differ diff --git a/assets/images/blog/2022-03-25/la-plagne-map.png b/assets/images/blog/2022-03-25/la-plagne-map.png deleted file mode 100644 index fa66508..0000000 Binary files a/assets/images/blog/2022-03-25/la-plagne-map.png and /dev/null differ diff --git a/assets/images/blog/2022-03-25/tignes-map.png b/assets/images/blog/2022-03-25/tignes-map.png deleted file mode 100644 index f977cae..0000000 Binary files a/assets/images/blog/2022-03-25/tignes-map.png and /dev/null differ diff --git a/assets/images/blog/2022-03-25/tignes-street-level-view.png b/assets/images/blog/2022-03-25/tignes-street-level-view.png deleted file mode 100644 index 7c49a7d..0000000 Binary files a/assets/images/blog/2022-03-25/tignes-street-level-view.png and /dev/null differ diff --git a/assets/images/blog/2022-03-25/val-thorens-map.png b/assets/images/blog/2022-03-25/val-thorens-map.png deleted file mode 100644 index 58a8a07..0000000 Binary files a/assets/images/blog/2022-03-25/val-thorens-map.png and /dev/null differ diff --git a/assets/images/blog/2023-01-06/courmayeur-meta.png b/assets/images/blog/2023-01-06/courmayeur-meta.png deleted file mode 100644 index c98ad9f..0000000 Binary files a/assets/images/blog/2023-01-06/courmayeur-meta.png and /dev/null differ diff --git a/assets/images/blog/2023-01-06/courmayeur.png b/assets/images/blog/2023-01-06/courmayeur.png deleted file mode 100644 index f459def..0000000 Binary files a/assets/images/blog/2023-01-06/courmayeur.png and /dev/null differ diff --git a/assets/images/blog/2023-01-06/la-thuile-la-rosiere.png b/assets/images/blog/2023-01-06/la-thuile-la-rosiere.png deleted file mode 100644 index b9b7992..0000000 Binary files a/assets/images/blog/2023-01-06/la-thuile-la-rosiere.png and /dev/null differ diff --git a/assets/images/blog/2023-01-06/serre-chevallier.png b/assets/images/blog/2023-01-06/serre-chevallier.png deleted file mode 100644 index 7d1da6c..0000000 Binary files a/assets/images/blog/2023-01-06/serre-chevallier.png and /dev/null differ diff --git a/assets/images/blog/2022-08-12/mapillary-image-pipeline.jpg b/assets/images/blog/2023-02-17/mapillary-image-pipeline.jpg similarity index 100% rename from assets/images/blog/2022-08-12/mapillary-image-pipeline.jpg rename to assets/images/blog/2023-02-17/mapillary-image-pipeline.jpg diff --git a/assets/images/blog/2022-08-12/mapillary-web-api-requests-meta.jpg b/assets/images/blog/2023-02-17/mapillary-web-api-requests-meta.jpg similarity index 100% rename from assets/images/blog/2022-08-12/mapillary-web-api-requests-meta.jpg rename to assets/images/blog/2023-02-17/mapillary-web-api-requests-meta.jpg diff --git a/assets/images/blog/2022-08-12/mapillary-web-api-requests-sm.jpg b/assets/images/blog/2023-02-17/mapillary-web-api-requests-sm.jpg similarity index 100% rename from assets/images/blog/2022-08-12/mapillary-web-api-requests-sm.jpg rename to assets/images/blog/2023-02-17/mapillary-web-api-requests-sm.jpg diff --git a/assets/images/blog/2022-08-12/trek-view-mapillary-app.jpg b/assets/images/blog/2023-02-17/trek-view-mapillary-app.jpg similarity index 100% rename from assets/images/blog/2022-08-12/trek-view-mapillary-app.jpg rename to assets/images/blog/2023-02-17/trek-view-mapillary-app.jpg diff --git a/assets/images/blog/2023-09-22/P1040540.JPG b/assets/images/blog/2023-09-22/P1040540.JPG deleted file mode 100755 index a776d82..0000000 Binary files a/assets/images/blog/2023-09-22/P1040540.JPG and /dev/null differ diff --git a/assets/images/blog/2023-09-22/P1040574.JPG b/assets/images/blog/2023-09-22/P1040574.JPG deleted file mode 100755 index a2189b9..0000000 Binary files a/assets/images/blog/2023-09-22/P1040574.JPG and /dev/null differ diff --git a/assets/images/blog/2023-09-22/P1040575.JPG b/assets/images/blog/2023-09-22/P1040575.JPG deleted file mode 100755 index 920619f..0000000 Binary files a/assets/images/blog/2023-09-22/P1040575.JPG and /dev/null differ diff --git a/assets/images/blog/2023-09-22/meru-route-map.jpeg b/assets/images/blog/2023-09-22/meru-route-map.jpeg deleted file mode 100644 index 8a6fd9a..0000000 Binary files a/assets/images/blog/2023-09-22/meru-route-map.jpeg and /dev/null differ