Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tile ArcGis FeatureServer requests #7370

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open

Conversation

nf-s
Copy link
Contributor

@nf-s nf-s commented Dec 13, 2024

Tile ArcGis FeatureServer requests

Fixes #3661 #7331 #6840

Merge after #7144

Changes

Main breaking changes

  • Add tiling support to ArcGisFeatureServerCatalogItem - this will be enabled by default if the server supports tiling and unsupported marker/point styles aren't used. See ArcGisFeatureServerCatalogTraits tileRequests. When enabled, pbf tiles will be fetched and drawn using ProtomapsImageryProvider

    • This can be disabled by setting tileRequests to false
    • For point features, only the following PointSymbolTraits are available - fill, stroke, radius (which uses height). Custom markers (markers other than point or circle) are not supported.

Other changes

  • Add request parameter to ArcGisImageServerImageryProvider.buildImageResource - this enables Cesium to manage requests
  • Decrease protomaps tile buffer to 32 pixels (from 64) to increase performance
  • Change ProtomapsImageryProvider to use a "soft" minimum level, so no tiles will be created below the minimumZoom provided.
  • Move ProtomapsImageryProvider.pickFeatures GeoJSON logic to inside ProtomapsGeojsonSource.pickFeatures.
  • Fix FeatureInfoUrlTemplateMixin reactivity warnings
  • Move GeojsonMixin protomaps paint/label rules to tableStyleToProtomaps.
    • Also create getStyleReactiveDependencies that can be used to track (and react to) table styling traits
  • Add dash to OutlineStyleTraits (only supported by line features), and outlineStyle to LegendTraits.
  • Add MinMaxLevelMixin to ArcGisFeatureServerCatalogItem - only applied when tiling requests
  • Tweaked TableStyleMap and TableColorMap conditions to handle empty TableColumns (to support styling ArcGisFeatureServerCatalogItem when tiling requests)
  • Fix bug with expanding "Developer Details" in the error modal
  • Fix regression bug from Upgrade protomaps leaflet and remove mvt imagery provider #7144 (not released), GeoJson MultiPolygon features were being dropped

Todo

  • Get table styling working (with esri style)
  • Check all properties that are relevant to tiling (like is pbf supported, is quantization supported, etc)
  • Fix issues along tile border
  • Optimise number of features can we filter out null features in the request?
  • Feature picking
  • Test more layers
  • Write unit tests
    • New ProtomapsArcGisPbfSource
    • Added request to image server
    • tableStyleToProtomaps new properties - dash/color/style/width + point features
  • Make sure requests are cancelled when moving cesium camera around This is harder than expected, so will leave unless it seems necessary later
    • This is a problem for larger layers, need to at least cancel requests when layers are turned off or removed from the wokrbench This is a big issue, but I can't find an easy way to do this. Not for this release, will make a ticket. This ties into larger issues of large network requests crashing Terria. After testing many layers, I haven't been able to crash Terria, but sometimes it takes a while for layer requests to finish, even if you have removed it from the workbench
  • Fix performance regression with geojson
  • Add support for marker icons... Not for this release, will make a ticket
  • Add min/max scale support
  • Fix outline legend
  • Add outline dashed styles
  • Add MinMaxLevelMixin mixin
  • Fix bug where tiles are reloaded everytime the style changes (eg edit style) leaving this bug due to lack of time - will create a ticket
  • Fix share bug with minmaxlevel see Tile ArcGis FeatureServer requests #7370 (comment) - leaving this bug due to lack of time - will create a ticket

Test me

Checklist

  • There are unit tests to verify my changes are correct or unit tests aren't applicable (if so, write quick reason why unit tests don't exist)
  • I've updated relevant documentation in doc/.
  • I've updated CHANGES.md with what I changed.
  • I've provided instructions in the PR description on how to test this PR.

@nf-s nf-s changed the base branch from main to upgrade-protomaps-leaflet December 13, 2024 07:12
@nf-s nf-s marked this pull request as draft December 13, 2024 07:13
@zoran995
Copy link
Collaborator

zoran995 commented Dec 14, 2024

It might be of interest to look into the layer mentioned in #6840. That layer is failing with

{
"code": 400,
"message": "",
"details": [
"'Invalid field: objectid' parameter is invalid"
]
}

@CLAassistant
Copy link

CLAassistant commented Dec 24, 2024

CLA assistant check
All committers have signed the CLA.

Base automatically changed from upgrade-protomaps-leaflet to main February 4, 2025 00:40
@nf-s
Copy link
Contributor Author

nf-s commented Feb 4, 2025

There is a bug I haven't been able to fix. It occurs in the 3D (cesium) viewer, with layers that have a minimum zoom level, when you create a share link with a picked feature

You get this error

image image

Comment on lines +144 to +148
(colorColumn &&
!colorTraits.mapType &&
colorColumn.type === TableColumnType.scalar) ||
colorTraits.mapType === "continuous" ||
colorTraits.mapType === "bin"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loosen condition here, as we need to allow Discrete/Continuous color maps when colorColumn is undefined (eg empty)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: is there a scope to make the intent more clear for the condition? One possibility might be to assign first to a temporarily variable like: const isScalarColorColumn = colorColumn?.type === TableColumnType.scalar.

@nf-s nf-s marked this pull request as ready for review February 6, 2025 01:31
): GeomType | null => {
switch (type) {
case "Point":
case "MultiPoint":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the input for this function ever come from the code updated in #7432, or is that a separate code path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most likely yes, but depends on a few other factors

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just merged your PR in

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if your yes means that the input comes from the other code or if it is a separate code path, but thank you for merging the PR.

const geom = f.geometry as any as [number, number][][];
transformedGeom = geom.map((g1) =>
g1.map((g2) => {
const x = g2[0];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this function body identical to the function body in the else-branch on line 271?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I think it is fine. IMO generally not worth refactoring duplication out unless there are more then 2 instances

Comment on lines +227 to +228
let transformedGeom: Point[][] = [];
let numVertices = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think these two need to be initialized (looks like they are assigned in all code paths below).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking no, but they make sense as default values, so i'm going to leave them there

@na9da na9da self-assigned this Feb 11, 2025
Copy link
Collaborator

@na9da na9da left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nf-s - this is an awesome feature improvement. Great work on the implementation 🎉

I have left some comments in the code review plus a couple of other observations:

  • I noticed in some cases, the column values differ between protomap and geojson rendering modes. This affects the symbology. Not sure if it is easily fixable though.
    protomap:
    image
    geojson:
    image

  • Not something from this PR but I think the second parameter here should be this._featureServer. This for example affects the behaviour when splitting an item for compare.

`WARNING: Terria only supports ArcGisFeatureService UniqueValueRenderers with a single field (\`field1\`), not multiple fields (\`field2\` or \`field3\`)`
);
}

if (uniqueValueRenderer.visualVariables?.length) {
console.warn(
`WARNING: Terria does not support visual variables in ArcGisFeatureService SimpleRenderers`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small typo: should this say UniqueValueRenderers instead of SimpleRenderers? Same for classbreaks

@@ -315,6 +388,12 @@ class FeatureServerStratum extends LoadableStratum(
const simpleRenderer = renderer as SimpleRenderer;
const symbol = simpleRenderer.symbol;

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: is it worth splitting styling and associated functions into a separate Stratum and file?

Comment on lines +191 to +198
supportedQueryFormats?: string;
maxRecordCount?: number;
tileMaxRecordCount?: number;
maxRecordCountFactor?: number;
supportsCoordinatesQuantization?: boolean;
supportsTilesAndBasicQueriesMode?: boolean;
objectIdField?: string;
fields?: Field[];
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like for the other types, is there a spec link for FeatureServer we can point to in comment?

Comment on lines +97 to +98
// Not sure how to handle request here - as we are making multiple requests
// request: request
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it work if we clone the request? AFAICT, for imagery layers, it only sets some flags at this stage in the code path. Cancellation might work even without passing request as it seems to create a new Request by default.

abstract mapValueToColor(
value: string | number | null | undefined
): Readonly<Color>;
abstract mapValueToColor(value: JsonValue | undefined): Readonly<Color>;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doubt: is this change required? Most implementations of ColorMap assumes the param to be string | .... Also the JsonValue imported here comes from protomap-leaflet

MappableMixin.isMixedInto(catalogItem)
? catalogItem.showStringIfPropertyValueIsNull
: undefined
currentTime,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doubt: Should we do currentTime ?? JulianDate.now() instead so that it is always current and not the time when the closure was generated.

Comment on lines +1318 to +1327
export function isLine(json: any): json is Feature<LineString> {
return (
isJsonObject(json, false) &&
json.type === "Feature" &&
isJsonObject(json.geometry, false) &&
json.geometry.type === "LineString"
);
}

export function isMultiLineString(json: any): json is Feature<MultiLineString> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point maybe we should probably extract these functions to a new file. I have found them useful in other contexts.

Comment on lines +144 to +148
(colorColumn &&
!colorTraits.mapType &&
colorColumn.type === TableColumnType.scalar) ||
colorTraits.mapType === "continuous" ||
colorTraits.mapType === "bin"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: is there a scope to make the intent more clear for the condition? One possibility might be to assign first to a temporarily variable like: const isScalarColorColumn = colorColumn?.type === TableColumnType.scalar.

Comment on lines +457 to +458
"imageryProvider" in d &&
d.imageryProvider instanceof ProtomapsImageryProvider
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: There's also ImageryParts.is() for this test

@na9da
Copy link
Collaborator

na9da commented Feb 14, 2025

On the cesium error, I think it is happening when the imageryProvider gets destroyed while it is being loaded. This happens when the trait change inside requestImage triggers a chain of events that results in our Cesium.ts removing it from the scene.

One fix we could do at our end is to wrap these setTrait calls within runLater - which gives cesium a chance to finish what it is doing before the imagery provider gets destroyed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Look into tiling ArcGis feature service requests
5 participants