P1 query and scale design

Target: ArcGIS Enterprise 12.1 on Linux. Full-fetch remains the default. Pass-through is explicit per source and must be certified in staging before release.

flowchart LR
  Client --> Validation[Query grammar and schema allowlist]
  Validation --> Queue[Bounded admission and deadlines]
  Queue --> Worker[Exclusive native worker]
  Worker --> Version[Source identity before and after reads]
  Version --> DuckDB[Bound SQL and spatial transformations]
  DuckDB --> Response[Bounded GeoJSON or aggregate result]
  Worker --> Metrics[Resource and source observations]

The query grammar accepts boolean combinations of comparisons, IN, BETWEEN, LIKE and NULL tests. It binds literals and obtains identifiers from DESCRIBE. No client SQL, functions, subqueries or comments are accepted. Paging has an ID tie-breaker. Count, extent and statistics operate in DuckDB without fetching the complete dataset; returned rows and serialized results remain bounded. Unsupported GeoServices operations fail explicitly.

Geometry uses explicit XY axes. Sources declare EPSG codes; filters accept GeoJSON and ArcGIS point, multipoint, envelope, paths and rings. DuckDB transforms source and input geometry to WGS84. GeoJSON responses stay WGS84; CDF handles client output projection. Extent output can be transformed explicitly. Arbitrary PROJ strings are outside the contract. Local CRS accuracy depends on DuckDB's bundled PROJ data and requires operator qualification.

Approved schema contracts contain a version, field names/native types and a strict or additive policy. A fingerprint can additionally pin the exact source layout. Runtime discovery checks contracts before cache lookup. Exact 64-bit integers and decimals are exposed as strings; unsupported compound attributes are rejected in pass-through mode. Full-fetch attributes keep their existing representation.

Cache identity includes the source and an observed identity. TTL is the compatibility default; local stat, immutable operator version and HTTPS strong ETag modes are opt-in. Identity is checked before and after reads, including cache hits. Mutable globs require an immutable snapshot/version. No distributed invalidation service is required when replicas share the same immutable version or strong ETag. TTL alone is not a snapshot guarantee.

Pool size defaults to one. Opt-in workers own independent DuckDB instances; total configured memory and threads are divided among them. A global queue limits admission. Timeout or cancellation retains only that worker until native work settles. CloudFS permits one worker because credentials have process scope. Installation occurs through the primary worker before other workers load extensions. Enable pooling only after representative capacity measurements prove benefit within the same total resource budget.

Prometheus scrapes sample native buffer/spill gauges through the bounded queue and expose observation timestamps. Source age uses observed modification times, never inferred data timeliness. Dashboards and example alerts require deployment-specific thresholds and authentication. Horizontal replicas need shared source versions and independent caches; source updates are operator-owned immutable snapshots.

Tradeoffs: per-request discovery and source checks cost I/O but detect drift before serving cached data. Independent instances give cancellation isolation but duplicate metadata and buffers. Revisit this design for catalog snapshot transactions, larger response limits or distributed job execution. ArcGIS staging, Linux images and real remote accounts remain release gates.

Contracts follow the CDF pass-through guide, CDF metadata reference and DuckDB spatial functions.