Skip to content

Fix "Hole lies outside shell" for reversed islands - #53

Merged
joto merged 1 commit into
osmcode:masterfrom
wangi:hole-lies-outside-shell
Sep 8, 2026
Merged

Fix "Hole lies outside shell" for reversed islands#53
joto merged 1 commit into
osmcode:masterfrom
wangi:hole-lies-outside-shell

Conversation

@wangi

@wangi wangi commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Fixes #41.

The problem

create_polygons() calls organizePolygons() with METHOD=ONLY_CCW, so ring direction alone decides shell vs. hole, and a hole is assigned to a shell using only a bounding box test. An island whose coastline was mapped the wrong way round becomes a hole candidate, and if it happens to lie inside the bounding box of a larger landmass (but outside the landmass itself) it is attached to it as a hole. GEOS then reports

Warning 1: Hole lies outside shell at or near point ...

and the whole polygon — possibly a whole continent — is invalid and ends up in the error_lines table, or is dropped entirely.

The existing "Fixing coastlines going the wrong way" step runs after the polygons have been created and only looks at exterior rings, so it never caught this case and reported "Turned 0 polygons around".

The fix

When a polygon is invalid, each interior ring is checked against the exterior ring. The check tests the first point of the interior ring with OGRLinearRing::isPointInRing() rather than a GEOS operation, because it has to work on invalid geometries too. Rings that are outside their shell are taken out of the polygon, turned around into land polygons of their own, and reported in the error_lines table with the error direction (the same error the existing direction fix uses). They are counted in num_rings_turned_around.

While at it, invalid polygons that are not part of a multipolygon are now repaired in the same way as those that are (report an invalid error line and try Buffer(0)), instead of just being dropped.

Tests

  • invalid-direction-island-in-bbox — reversed island inside the bounding box of an L-shaped landmass (single polygon code path).
  • invalid-direction-island-in-bbox-of-many — the same with a second, correctly mapped island, so organizePolygons() returns a multipolygon (the code path hit by the case in Reversed coastline results in "Hole lies outside shell" output #41).

valid-inland-sea-with-island needed updating: the island in the inland sea is mapped hole-style and so ends up nested inside the hole formed by the sea ("Holes are nested"). That polygon is now repaired instead of discarded, so the land polygon survives and the result is a warning rather than an error. The island itself is still dropped and reported as questionable; that nested-hole limitation is unchanged.

@wangi

wangi commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

I have also tested this against the files referenced on #41, which now gives following output:

/opt/opengeofiction/osmcoastline/bin/osmcoastline --verbose --srs=3857 --overwrite --output-lines --output-polygons=both --output-rings --max-points=1000 --output-database ogf-coastlines-split.db ogf-coastline-data.osm.pbf
[ 0:00] Started osmcoastline version 2.5.0 / libosmium version 2.23.0
[ 0:00] Using SRS 3857 for output. (Change with the --srs/s option.)
[ 0:00] Writing to output database 'ogf-coastlines-split.db'. (Was set with the --output-database/-o option.)
[ 0:00] Removing database output file (if it exists) (because you told me to with --overwrite/-f).
[ 0:00] Will create geometry index. (If you do not want an index use --no-index/-i.)
[ 0:00] Reading from file 'ogf-coastline-data.osm.pbf'.
[ 0:00] Reading ways (1st pass through input file)...
[ 0:01]   There are 0 nodes where the coastline is not closed.
[ 0:01]   There are 38567 coastline rings (36907 from a single closed way and 1660 others).
[ 0:01] Memory used: current: 1494 MBytes, peak: 1494 MBytes
[ 0:01] Reading nodes (2nd pass through input file)...
[ 0:02] Checking for missing locations...
[ 0:02]   All locations are there.
[ 0:02] Memory used: current: 1767 MBytes, peak: 1767 MBytes
[ 0:02] Check line segments for intersections and overlaps...
[ 0:03] Trying to close Antarctica ring...
[ 0:03]   Did not find open Antarctica ring.
[ 0:03] Close broken rings... (Use --close-distance/-c 0 if you do not want this.)
[ 0:03]   Closing if distance between nodes smaller than 1. (Set this with --close-distance/-c.)
[ 0:03]   Closed 0 rings. This left 0 nodes where the coastline could not be closed.
[ 0:03] Writing out rings... (Because you gave the --output-rings/-r option.)
[ 0:05] Create polygons...
Warning 1: Hole lies outside shell at or near point 128.80197369999999 35.702867900000001
[ 0:07] Fixing coastlines going the wrong way...
[ 0:07]   Turned 1 polygons around.
[ 0:07] Transforming polygons to EPSG 3857...
[ 0:07] Writing coastlines as lines... (Because you used --output-lines/-l)
[ 0:08] Not performing check for questionable input data, because it only works in EPSG:4326...
[ 0:08] Split polygons with more than 1000 points... (Use --max-points/-m to change this. Set to 0 not to split at all.)
[ 0:08]   Using overlap of 10 (Set this with --bbox-overlap/-b).
[ 0:23] Checking and making polygons valid...
[ 0:24] Writing out land polygons...
[ 0:25] Writing out water polygons...
[ 1:40] Memory used: current: 1767 MBytes, peak: 1767 MBytes
[ 1:40] Committing database transactions...
[ 1:40] All done.
[ 1:40] Memory used: current: 1767 MBytes, peak: 1767 MBytes
There were 1 warnings.
There were 0 errors.

Also tested as part of the complete OpenGeofiction coastline process.

@joto

joto commented Sep 5, 2026

Copy link
Copy Markdown
Member

Thanks for looking into this and for the PR. I had to bring the CI up to date first, can you please rebase the PR on top of the current master?

On the whole this looks pretty promising, but I am a bit concerned about last existing test with the changed outcome. In the old version it created three error lines marked "questionable" which is osmcoastline's way of throwing its hands into the air and saying: "Something looks fishy here, but I don't know what is happening, a human needs to look at this." The new code generates a correct land polygon, but it also puts the outermost ring as "invalid" into the errors table, which is wrong. So we have a regression here. Can you look into that?

@wangi
wangi force-pushed the hole-lies-outside-shell branch from 151ca86 to 3d577f8 Compare September 6, 2026 16:01
@wangi

wangi commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master.

You are right about the invalid error line, and looking into it showed that the outer ring was not the problem at all: in that test the island in the inland sea is mapped the wrong way round too, so organizePolygons() makes it a hole inside the hole formed by the inland sea ("Holes are nested"). That is the same bug as #41, just one level deeper — a hole inside a hole can only be land, so it is a ring going the wrong way, not something GEOS should have to repair.

So instead of falling through to Buffer(0) and blaming the outer ring, interior rings are now checked for both cases: outside the exterior ring, and inside another hole. Either way the ring is taken out of the polygon, turned around and added as a land polygon of its own, reported as direction.

For valid-inland-sea-with-island that now gives the correct result, with no invalid line:

land_polygons:
  POLYGON((1.05 1.99, 1.14 1.99, 1.23 1.98, 1.25 1.97, 1.21 1.92, 1.08 1.92, 1.04 1.97, 1.05 1.99),
          (1.1 1.97, 1.12 1.94, 1.16 1.94, 1.17 1.97, 1.1 1.97))
  POLYGON((1.13 1.96, 1.14 1.96, 1.14 1.95, 1.13 1.96))
error_lines:
  direction    LINESTRING(1.13 1.96, 1.14 1.96, 1.14 1.95, 1.13 1.96)
  questionable LINESTRING(1.1 1.97, 1.17 1.97, 1.16 1.94, 1.12 1.94, 1.1 1.97)

The island is reported as direction (which is what is actually wrong with it) and the inland sea is still reported as questionable, so nothing that needs a human is swallowed.

Two notes on the implementation:

  • The containment tests use OGRLinearRing::isPointInRing() rather than a GEOS operation, because they have to run on geometries that GEOS already considers invalid. Only the first point of each ring is tested, so rings that are only partly outside are not touched — those show up as intersections and are reported elsewhere.
  • All of this only runs for polygons that are invalid, and envelopes are precomputed, so there is no cost for the normal case. I re-ran the full OGF world coastline through it: 0 warnings, 0 errors, and no measurable difference in runtime.

@wangi

wangi commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Ran the data from #41 (https://data.opengeofiction.net/extracts/osmcoastline-issue41/ogf-coastline-data.osm.pbf) through both current master (2c6dd17) and this branch, same command line as in the issue:

master this branch
land_polygons 48658 48659
water_polygons 2923 2923
error_lines 1 × invalid, 639225 points, bbox 24.78,15.67 .. 152.14,79.42 1 × direction, 9 points, bbox 128.801,35.699 .. 128.804,35.703
runtime 1:42 1:43

That invalid line on master is the whole continent, which is what made this hard to track down in the first place. It is now replaced by the 9 point ring that is actually wrong — at 128.80197369 35.7028679, exactly the point GEOS names in its warning — and the island itself comes back as land instead of being swallowed by the Buffer(0) repair (the extra land polygon).

Both runs report 1 warning and 0 errors, and the runtime is unchanged.

Comment thread src/osmcoastline.cpp Outdated
return false;
}

return ring->isPointInRing(&point, FALSE) != FALSE;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a) This is C++ not C code and
b) != FALSE??

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tidied that up now.

organizePolygons() decides which rings are holes based on their
direction only and uses just a bounding box test when assigning a hole
to a polygon. A ring whose coastline was mapped the wrong way round can
therefore end up as a hole of a polygon it is not inside of, or as a
hole nested inside another hole. GEOS then reports "Hole lies outside
shell" or "Holes are nested" and the whole polygon, possibly a whole
continent, is invalid and ends up in the error_lines table. The
existing direction fix runs after the polygons have been created and
only looks at exterior rings, so it never caught these cases.

Those rings are not holes at all but land mapped the wrong way round.
They are now taken out of the polygon, turned around into land polygons
of their own and reported in the error_lines table with the error
"direction".

While at it, invalid polygons that are not part of a multipolygon are
now repaired in the same way as those that are, instead of being
dropped.

Fixes osmcode#41

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wangi
wangi force-pushed the hole-lies-outside-shell branch from cd8414f to 79a14a7 Compare September 7, 2026 14:27
@joto
joto merged commit 73c8ffc into osmcode:master Sep 8, 2026
26 checks passed
@joto

joto commented Sep 8, 2026

Copy link
Copy Markdown
Member

Thanks for the PR. I am not 100% happy, because it seems kind of backwards to me to fix the output of organizePolygons() instead of either fixing that function at the source or maybe just doing the whole thing ourselves. But it is what it is, it fixes a class of annoying errors we had for a long time and that's good. So we'll use it until something better comes along.

@wangi

wangi commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks Jochen, appreciate the merge. As you say it's working around the root issue, rather than fixing it at source - never the ideal situation. I would like to return to it in the future, once I clear my plate of a few other things on the go just now.

@wangi
wangi deleted the hole-lies-outside-shell branch September 8, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Reversed coastline results in "Hole lies outside shell" output

2 participants