Shared library for OSM tags -> lane specification

[dabreegster]

One piece of the A/B Street importing process looks at an OSM way's tags and calculates a list of lanes in left-to-right order, with a few simple details about each lane. The idea has come up a few times to share this piece of code among other projects that also process lane-level details.

See the threads below for details (and feel free to start new sub-threads!). The main questions I have for the stakeholders listed below are:

1. Would this be useful to you / your project?
2. Would you help contribute to it?

Stakeholders:

• @westnordost (StreetComplete)
• @matkoniecz
• @enzet (Map Machine)
• @BjornRasmussen (JOSM lanes plugin)
• @tordans (part of the https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Radwege#Tagging-Beispiele effort)
• @d-wasserman (OSM Derived Shared-ROW Data d-wasserman/shared-row#11)
• @mvl22 (CycleStreets, and context from https://github.com/a-b-street/abstreet/issues/330#issuecomment-767105579)
• @kfarr (3D Street)

[dabreegster]

API

The API would just have a function that transforms a string->string dictionary of key/values for a single OSM way into an ordered list of lane specifications. So as a very simple example, for something like https://www.openstreetmap.org/way/428294122 with lanes=2, oneway=yes, sidewalk=both, cycleway:left=lane in a right-handed driving country, the output would be something like

[
  {lane_type = sidewalk, width = 2m, dir = backwards},
  {lane_type = bike lane, width = 2.5m, dir = forwards},
  {lane_type = general purpose, width = 3m, dir = forwards},
  {lane_type = general purpose, width = 3m, dir = forwards},
  {lane_type = sidewalk, width = 2m, dir = forwards},
]

Lane specs

What's in a lane spec? The A/B Street definition currently looks like this:

• lane type (general-purpose, bus-only, bike lane, sidewalk, center turn lane, parallel street parking, and different types of buffers/separators like stripes, flex posts, planters, curbs)
• width of the lane
• direction: forwards or backwards

But I'd like to extend this to be more useful:

• include turn:lanes restrictions
• for lanes allowing movement, list types of vehicles/traffic allowed. A bus lane may allow taxis, high-occupancy vehicles, bicyclists.
• indicate if bidirectional movement is allowed for some traffic types
• Possibly include time restrictions on lane usage

Unlike OSM, I don't want free-form tagging here. I want to define an explicit specification, so software like A/B Street can handle all cases captured in the spec. At the same time, I realize a schema could get complex very quick, so I want to strike some balance. So for instance, time restrictions might not make v1 of this schema. Alternatively, software consuming this API can just ignore things it doesn't care about.

Localization

The function would probably also take in some kind of configuration, at least describing whether traffic drives on the left or right hand side, and things like if cyclists are generally allowed to use bus lanes.

Only one way at a time

This function would only look at a single way's tags. If a street car track, cyclepath, or footway are mapped as a separate way that's parallel or partly overlapping, this API wouldn't try to handle any of that. Only what can be inferred from the given way's tags.

Assuming defaults?

For the A/B Street use case, I want widths of each lane. Usually that's not explicitly tagged, but sometimes it is (the Berlin community has been doing an awesome micromapping effort here). One option is that the API makes reasonable guesses (perhaps driven by the localization config passed in) about lane widths. The other is that unknown information is just omitted / indicated as missing in the output.

[tordans]

Only one way at a time

I think we should reconsider this part. Mapping practice in OSM is already leaning towards mapping foot and bike traffic separate from cars. And when done properly, that is a good thing.

When I think of this effort, I think of all the lanes between two rows of houses. Basically what https://streetmix.net/ shows.

So we should think about was to

• allow for an input of multiple OSM ways
• consolidate multiple OSM ways into one output file

[tordans]

Assuming defaults?

OSM has the idea of defaults. Sometimes regional/per country defaults. Like access and lanes and such.
So IMO we will need to make assumptions part of the output.

However, we should look for a way to indicate that fact in the data. Maybe something like

{
  "lanes": [
    "lane": [
      "width": {
        "width": 2,
        "unit": "m",
        "source": "assumed default" /* or 'width:carridsway value on osm way#<ID>' or 'calculated based on width:carrigeway minus something' (when calculation was possible to derive the width) */
      }
    ],
  ],
}

[dabreegster]

Treating all of the lanes in between two rows of houses as one logical road is like, the holy grail in my opinion! It's a hard problem.

allow for an input of multiple OSM ways

So it would be up to the caller of the library to figure out which parallel ways logically belong together. What if the main road references cycleway=sidepath and the other way isn't provided? And how will the library figure out if separate ways are geometrically located to the left or right of the main road?

we should look for a way to indicate that fact in the data

I like this source tag a lot. It should probably belong to most of the attributes of each lane -- inferring turn:lanes just by lane count is another good case where we could apply a heuristic, but a consumer of the library could choose to ignore the guess, because it's labeled a guess.

"unit": "m",

I would be in favor of canonicalizing units. If the source data lists feet, convert to meters. If we go with a JSON output format, we could still explicitly list the units to be clear, or document it on the width field.

[dabreegster]

Implementation

Output

The easy answer is JSON, probably with a schema. Alternatively, we could use protocol buffers or something similar with cross-language support.

Language(s)

I have an initial implementation of this API: https://github.com/a-b-street/abstreet/blob/master/map_model/src/make/initial/lane_specs.rs
(It has some A/B Street specific assumptions baked in that would be cleaned up, of course.)
This implementation is in Rust. It can be compiled to WASM and called from Javascript/Typescript. It should also be reasonable to call it from C, Java, etc through some kind of foreign-function interface. Also there could be a simple command-line tool built for multiple platforms that reads in a JSON file and outputs another one.

But I think having multiple implementations in different languages would be fine too. The Rust implementation doesn't use any language features that should be surprising, but getting the development environment set up could still be a barrier to people. I think maintaining an implementation in multiple languages should not be too tedious, because the code structure is straightforward to port. And because of testing -- see below.

Tests

Even if we didn't want to share an implementation, I'd at least like to share a set of test cases to agree on the transformation process. I have some examples written:

abstreet/map_model/src/make/initial/lane_specs.rs

Line 395 in cf3d00f

(

Each test case would list the OSM key/values, then the expected lane spec output. The test cases could be checked in as a JSON file in a git repo somewhere, and any implementation could check itself against them.

[enzet]

How do you see the implementation? Should it be a separate project with a collection of data files (JSON files, presumably) and several code files in different languages?

[dabreegster]

How do you see the implementation?

Yes, I would create a new git repo containing test cases (probably in JSON) and implementations in at least 2 languages. I assume you'd vote for Python as a second?

[enzet]

Looks good to me!

Yes, I'm ready to support Python version. As well as Java/Kotlin version if it is needed.

[dabreegster]

Goals

Why bother trying to share this kind of code and make a schema for lanes?

• There are many equivalent ways of tagging things like cycleway orientation and side of the road. Interpreting the combinations gets confusing. Quick test, can you visualize what cycleway:left=opposite_track means? (https://www.openstreetmap.org/way/369623526) I'm sure the A/B Street implementation gets some of these cases wrong. Selfishly I'd like help from the community to transform this into a format that I can understand and work with more easily.
• I'd like to add support for details that the Berlin community has been working on: https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Radwege#Tagging-Beispiele. Or more accurately, I'd like to encourage somebody else to add that support and hand me better lane specifications, so I can focus on the rendering/semantics in A/B Street. :) (Also, this wiki made me realize lane specs should maybe include a surface or smoothness type, and info on coloring/markings.)
• In some cases, the entire road width can be figured out from other GIS sources. It'd be interesting to add support for calculating the likely split of that total width amongst the lanes mapped in OSM.
• This library could be a step towards getting lane visualization (and maybe editing) into ID, so more people can easily validate tagging.

[westnordost]

You asked whether it would be useful for StreetComplete: Currently not and also probably not in the long-term. The app is too simple and does not allow users to edit roads in such a "holistic" way that includes all lanes. Also, for an editor, OSM tags -> lane spec is just half the rent, lane spec -> OSM tags being the other half.

Selfishly I'd like help from the community to transform this into a format that I can understand and work with more easily.

I can answer any cycleway-tagging related questions as I also wrote a parser with extensive tests for that.
(Implementation here: https://github.com/streetcomplete/StreetComplete/blob/master/app/src/main/java/de/westnordost/streetcomplete/quests/cycleway/CyclewayParser.kt )

[dabreegster]

Noted, thanks!

for an editor, OSM tags -> lane spec is just half the rent, lane spec -> OSM tags being the other half.

I forgot to mention that, but it'd be a possible next step. The same unit test suite could be helpful, and just round-tripping and making sure nothing changes.

I can answer any cycleway-tagging related questions as I also wrote a parser with extensive tests for that.

Thank you for the offer! The test suite is indeed quite exhaustive; excellent reference: https://github.com/streetcomplete/StreetComplete/blob/master/app/src/test/java/de/westnordost/streetcomplete/quests/cycleway/CyclewayParserKtTest.kt

[d-wasserman]

I think the transformer code to create what you suggest would have value honestly outside this project if just to reference. I am interested in reviewing, but would have limited bandwidth to contribute.

[dabreegster]

Got it! To refresh my memory, here's the shared-row lane specification: https://github.com/d-wasserman/shared-row/blob/main/specification/MarkdownTables/Slice.md#slice-types
Lots more detail about lane types and further attributes of each.

[enzet]

Regarding Map Machine, it will be great to have such an API. I'm going to support lane tags but as for now, I've covered just lane number and lane widths.

[tordans]

FYI, there will be an update of the Straßenraumkarte by @SupaplexOSM soon and I am working on a (english) blogpost about the tagging involved. That will show some of the processing decisions Alex' made for displaying the lanes in all it's details. Current version at https://supaplexosm.github.io/strassenraumkarte-neukoelln/?map=micromap#20/52.47400/13.44061

[mvl22]

@tordans Use of area:highway is very interesting.

This is very cool work, well done.

Have you seen our talk at SOTM2019?
https://www.cyclestreets.org/news/2019/09/22/sotm2019/

I'd be interested to discuss this further. Also would be interested to try to add this to junctions here in Cambridge, UK, if you want some more data to test on.

[tordans]

Have you seen our talk at SOTM2019?
https://www.cyclestreets.org/news/2019/09/22/sotm2019/

I think I was even in the room :-). Will have to re-watch it though.

[tordans]

area:highway

Right now, Alex @SupaplexOSM (who is doing all the Mapping+Processing+Mapping work) is a big fan; but we consider it a local experiment for this specific project. Personally, I am still unsure if now is the right time to push this schema further, since it will only work with really good editor integration and a lot of new documentation and know how sharing. — And for this, we would need a lot more interesting use cases that only work with this schema, like Alex map. — osm2lanes might actually become such a project :-).

[mvl22]

The use of areas to indicate junction extents, in same way in our view has a clear benefit to routing: better overall routing directions, and dealing with signal delay counts, as discussed in our talk. Whether area:highway is the specific answer or something similar, I'm not sure.

[enzet]

Wow! This work is kinda next level of road mapping! Very inspiring.

[tordans]

I would create a new git repo containing test cases (probably in JSON)
#789 (reply in thread)

I think this would be a great first step. We can collect OSM Way data in a key-value (txt?) format and the desired JSON output next to it. A few complex situations from #789 (comment) come to mind, that will make for nice test cases.

I am wondering …

• Should we create some kind of visualization / page / output of those test files, just to make it easier to work with them
• How much documentation references do we want to add to those files; like links to pages like https://wiki.openstreetmap.org/wiki/DE:Tag:traffic_sign%3DDE:240 that describe specific tagging quite explicitly.
• Should the test files have meta tags, like a name and description. From the same example wiki page above, that would be a "Gemeinsamer Geh- und Radweg".

[enzet]

Is it OK if i copy some tests (as JSON files) from here into the repository?

[dabreegster]

Absolutely, go ahead!

[enzet]

Great! Thanks!

[enzet]

Well, I just finished supporting tests (except the last one) in Python and Kotlin.

[enzet]

Regarding the last test, I've created an issue.

[tordans]

Would this be useful to you / your project?
Would you help contribute to it?

Right now, I am very interested in everything that helps explain and standardize the tagging. The actual shared lane "file" is less interesting for now, since we don't have any use cases for it yet.

However, the two things can go hand in hand: If we where to process areas of OSM based on this schema and visualize it as a (vector tile) map, it would help a lot to improve the tagging in general (make things visible, showing interpreted data).

Right now, a more pressing issue for me (for this general topic) is, to create something like https://streetmix.net/ which outputs OSM tagging recommendations. This would help a lot to get better tagging on the road.

[dabreegster]

If we where to process areas of OSM based on this schema and visualize it as a (vector tile) map

This was one of the goals behind https://a-b-street.github.io/docs//software/osm_viewer.html. The biggest usability gap is that the web version only works for areas that I import, so the coverage is very limited. The downloaded version can import any area, but that's lots of steps before seeing anything. I have thought about a web-based import process, where someone zooms in on an area, presses a button, waits possibly up to a minute or so, and sees the results. It could use Overpass to get the latest data for the area, and run a limited version of the A/B Street map importer, all locally in the browser. But the latency is probably still unacceptable.

something like https://streetmix.net/ which outputs OSM tagging recommendations

I would love to see this. I've followed openstreetmap/iD#387 a bit. A partial start to the UI could be what's in A/B Street today: https://a-b-street.github.io/docs/project/retrospective/index.html#road-editor
But this presents a very limited number of possible changes.

So it sounds like for this shared library to be useful, being able to transform the list of lane specifications back into OSM tags would also be needed. Where this gets difficult is following the existing tagging style of a way. There's multiple methods to tag cycle lanes, for example, and if somebody makes an adjustment, we'd either have to remove all the existing lane-related tags on the way and insert new ones, or try to reconcile with the existing tags.

[tordans]

I just ran across https://osm-tiles.james-ross.co.uk/?map=19/51.5050/-0.0899&layer=standard&layer=all again. A project by @twpol at https://github.com/twpol/osm-tiles. James, this discussion an the Repo at https://github.com/a-b-street/osm2lanes could also be interesting for you and your project.

[enzet]

Thanks! The project is really nice and interesting. Maybe we can add C# implementation as well.

[twpol]

Thanks for the tag, this is indeed an interesting discussion - and the Straßenraumkarte looks amazing

I'd love to run my tags->lanes code against a test suite, as I'm sure it'll throw up some interesting issues :)