P1 capabilities and migration
The P1 implementation adds a restricted pass-through engine, explicit CRS transformations, approved schemas, observed source identities, optional workers and monitoring assets. Local native tests are development evidence. ArcGIS Enterprise 12.1/Linux staging, real remote accounts and container execution remain release gates.
Enable and inspect
Use the example configuration, then run:
node scripts/cdf-cli.js validate-config -f config/examples/pass-through.json
node scripts/cdf-cli.js preflight -f config/examples/pass-through.json -d default
CDF_CONFIG=config/examples/pass-through.json npm start
Relative paths resolve from the application root. Set CDF_API_TOKEN for production or any non-loopback listener. Existing configurations keep full-fetch mode and one worker. Set queryMode: "pass-through" per source to let getData(req) process supported ArcGIS parameters. The diagnostic query route is available for either mode; it returns GeoJSON and CDF metadata, not a standalone ArcGIS REST FeatureServer.
GET /api/sources/{name}/schema returns native columns, mapped fields, schema fingerprint and approved version. GET /api/sources/{name}/query accepts the following parameters; duplicate and unknown keys are errors. All endpoints require the diagnostic API's bearer token when configured.
| Parameter | Contract |
|---|---|
where |
Up to 8 KiB; comparisons = <> != < <= > >=, boolean AND/OR/NOT, parentheses, IN/NOT IN (100 entries), BETWEEN/NOT BETWEEN, LIKE/NOT LIKE, IS NULL/IS NOT NULL. Simple schema names, finite numbers, booleans, NULL and single-quoted strings with doubled apostrophes. No functions, comments, arithmetic or subqueries. Max 1024 tokens and 32 nested predicates. |
objectIds |
Comma-separated positive 32-bit IDs, up to 1000 entries. Combined with other predicates. |
outFields |
* or up to 64 unique schema fields; the ID is always included. Geometry is separate. |
orderByFields |
Up to 16 schema names with optional ASC/DESC; ID is appended as a deterministic tie-breaker. |
resultRecordCount, resultOffset |
1 to configured maxRecordCount; offset 0 to 1000000. Pages fetch one extra row and report exceededTransferLimit both at top level and in metadata. |
returnCountOnly, returnExtentOnly |
true/false; may be combined. Native aggregates cover all matching records, independently of paging. Empty extent is null. Extent responses also include count. |
returnIdsOnly |
All matching IDs, independent of maxRecordCount; fails above 100000 IDs or 8 MiB instead of truncating. Cannot combine with paging, statistics, count or extent. Narrow larger requests with a filter. |
returnGeometry |
true/false; false returns null geometry with the source geometry type in metadata. |
geometry, inSR |
GeoJSON or ArcGIS point/multipoint/envelope/paths/rings. JSON strings and REST comma-separated envelopes/explicit points accepted. Limit 64 KiB/10000 positions. Intersects only; polygon topology must be valid. Rings are grouped by nesting, including holes and separate parts. |
geometryType |
Optional ArcGIS esriGeometryPoint/Multipoint/Envelope/Polyline/Polygon; must match the supplied geometry. inSR/outSR also accept a bounded JSON WKID spatial-reference object. |
spatialRel |
Only esriSpatialRelIntersects. |
outSR |
4326, 3857, 102100/102113 aliases, or the source's configured EPSG. Applies natively to extents. Feature GeoJSON remains WGS84 (inputCrs: 4326); the CDF runtime handles client projection. |
outStatistics |
JSON array of 1–16 {statisticType,onStatisticField,outStatisticFieldName} objects. count, sum, min, max, avg, stddev, var; numeric native fields for functions except count; count(*) supported. Bounded result groups. Results are tabular GeoJSON features with null geometry. Values use double precision; exact large integer/decimal attributes are strings, but aggregates may round. |
groupByFieldsForStatistics |
Schema fields, only with statistics. Groups sort deterministically. Statistics reject field selection, ordering, IDs and offsets. Singleton stddev/var are null. |
Boolean values must be true/false. f, token and cacheHint are accepted as CDF transport hints; they do not affect diagnostic output or authorization. Unsupported operations (distance, time, having, distinct, quantization, generalization, editing and arbitrary spatial relations) fail explicitly. The CDF adapter reports filtersApplied for operations actually handled; feature output projection is delegated to CDF. Validate counts, IDs, field metadata, geometry omission, statistics and projection against the real staging runtime before enabling a public service. The engine supports a subset of GeoServices, not every query parameter.
Pass-through source qualification scans native aggregates before the first query for a source identity: IDs must be globally unique positive 32-bit integers; geometry must be valid, within WGS84 bounds after transformation, and from one supported family. This catches errors across pages. Qualification is reused for unchanged observed identities; TTL mode requalifies after cacheTTL. Discovery and deadline limits still apply. Large sources must complete qualification within the agreed initialization/query budgets.
CRS and attributes
crs accepts an EPSG integer. Every source other than 4326 must explicitly set axisOrder: "xy"; transforms use DuckDB ST_Transform with always_xy. Only the configured local EPSG plus 4326/3857 are accepted from requests. Conflicting internal and explicit input references fail. Geometry is emitted in WGS84. GeoParquet metadata, when present, must match the declared CRS; null/unknown CRS metadata fails. Missing metadata requires operator attestation. Z coordinates set hasZ metadata; CDF Z output requires JSON (PBF is not qualified). Measured M coordinates are rejected; convert them upstream. Local grid accuracy and rendering require production qualification.
Mapped metadata defines all attributes. ID is integer; small native integers map to integer; float/double to double; boolean to smallinteger (0/1); dates/timestamps to UTC epoch milliseconds. Exact BIGINT/HUGEINT/unsigned 32-bit integers and decimals map to string to prevent JSON precision loss. UUID is string. Unsupported nested/array/BLOB/time attributes must be flattened or cast upstream. Up to 256 source fields are accepted. Full-fetch preserves existing attribute representations unless an approved schema is enabled; pass-through always uses the mapped contract.
Approve schema: {version,policy,fields,fingerprint?} using columns from the schema endpoint or preflight report. Fields specify native name and exact type and must include ID and geometry. Strict policy rejects any added, missing or changed field; additive permits extra supported fields while requiring all approved types to remain exact. A fingerprint additionally pins the entire layout, so additive growth also requires updating a pinned fingerprint. Approval is configuration owned by operators; there is no automatic migration or remote approval API. Runtime checks happen before cache lookup, and preflight reports drift. Revalidate CDF service metadata and consumers before approving a new version.
Freshness and replicas
| Source freshness | Behavior |
|---|---|
absent or {mode:"ttl"} |
Existing cacheTTL behavior. No claim that the origin is unchanged. |
{mode:"stat"} |
One local file; checks device/inode/size/mtime/ctime before and after reads and cache hits. Globs/directories/remote URIs rejected. |
{mode:"version",version:"snapshot-42",modifiedAt:"2026-10-02T00:00:00Z"} |
Immutable operator snapshot; replicas sharing location/version derive identical identity keys. Publish a new location/version and roll out all replicas together. Optional timestamp supplies source age. |
{mode:"http",timeoutMs:5000} |
Public HTTPS GeoParquet URL; HEAD requires a strong quoted ETag, rejects redirects/outages/missing or weak tags, checks before and after reads and cache hits. Optional Last-Modified supplies age. Node HEAD does not inherit DuckDB secrets, so authenticated sources require immutable version mode. |
Cache keys include source, hashed identity and schema; observed identity changes invalidate that source's entries. Failed checks never serve stale entries. Changed reads return SOURCE_CHANGED and are not cached. Metadata includes mode, hashed version, checkedAt, fetchedAt, cacheAgeMs, cacheHit and optional modifiedAt. IDs/ETags/paths are not disclosed as identity values. Native external-file caching is disabled to keep application identity checks authoritative.
Before/after observations are not transactions: a mutable origin can change and revert between checks, or remote range reads can span versions. Publish immutable snapshots for production consistency. A manually reused version cannot detect origin mutations. Local stat keys are useful within one host, not a shared identity across hosts. ETag semantics must be qualified against the actual origin; version keys provide consistency only while replicas read the same immutable bytes. Existing CDF response caching is a separate layer and must use compatible invalidation settings.
Workers and evidence
database.poolSize defaults to 1, supports 1–8, and requires at least that many configured threads. Total memoryLimit and threads are divided (rounded down); each pooled worker requires at least 64MB. Separate native instances and generated per-model/per-worker scratch directories isolate interruption and spills, including when no tempDirectory is configured. tempDirectory is the scratch parent (OS temp by default). Owned directories are removed after all native handles close; cleanup failures block reuse and retain handles/scratch for safe shutdown or supervisor recovery. The shared queue still bounds all pending work. Cancellation/deadlines retain only the affected worker until native work settles and abort pending source identity HTTP requests. CloudFS requires poolSize 1 due to process-scoped credentials. Use dedicated processes for CloudFS isolation.
Run the fixture comparison or supply a representative configuration:
npm run test:pool -- -q 40 -o pool-report.json
node scripts/pool-check.js -f config/production.json -d parcels -q 100 -c 4 -o pool-report.json
The comparison tests pools 1/2/4 where the thread budget permits, in fresh child processes with identical total budgets and disabled result caches. Each worker has a 120-second supervisor deadline. It measures a 100-feature page plus discovery/identity work, throughput, p95, process RSS/CPU and sampled native resources. It does not cover every aggregate/filter workload. Reports fail on query errors; operators must evaluate ratios against their own SLOs. Configure at least 64MB per tested worker; CloudFS configurations are rejected for multi-worker comparisons.
Node 22 fixture evidence and Node 24 fixture evidence record 25000 CSV rows, 40 queries/scenario, concurrency 4 and a total 512MB/4-thread budget. Both runs improved throughput with multiple workers and increased RSS. They justify an opt-in implementation; one worker remains the production default until repeated Linux/remote/staging measurements meet agreed targets. Each scenario uses a fresh process so RSS comparisons exclude retained allocations from prior scenarios; sampling still may miss short peaks.
Monitoring
Enable monitoring.enabled in Helm only with Prometheus Operator CRDs; match monitoring.labels to the Prometheus selectors. Scrapes use the existing API token Secret through ServiceMonitor authorization. Enable monitoring.alerts.enabled after choosing p95/error/RSS/spill targets; source-age alerts stay off until source update expectations are agreed. Rendering without the required CRDs fails explicitly. Verify the token Secret access, ServiceMonitor discovery and alert routing in staging.
Import the Grafana dashboard using the Classic JSON import flow and select the Prometheus datasource and job. It shows requests, latency, queue/deadlines, RSS, cache, buffer/spill/scratch observations and source/resource observation ages. The JSON and Helm assets are locally parsed/rendered; live Grafana/Prometheus import and delivery remain platform checks. See Prometheus Operator authorization and Grafana JSON models.
Native resource gauges are sampled through the exclusive worker after successful data requests (at most once/second) and on authenticated metrics scrapes. Other workers retain their last observation and timestamp. These are current samples, not guaranteed query peaks; short-lived spill can be missed. Use OS/disk monitoring and query profiling for peak scratch qualification. Failed sampling increments a counter. Source age is modification age when known, not cache age or proof of business timeliness. Source checks occur on reads/preflight, so compare observation timestamps before trusting an old age gauge. TTL sources and ETags without Last-Modified have no source-age series.
See design decisions, operations and recovery and ArcGIS staging certification. External P0 release gates continue to apply.