Recommended roadmap
Reviewed October 2, 2026. Priorities reflect the actual bounded DuckDB core, ArcGIS full-fetch model, and diagnostic API. Advanced modules are prototypes. Effort estimates are relative, not delivery commitments.
Current delivery status
The approved first sprint is implemented in the core and locally tested. Target: ArcGIS Enterprise 12.1 on Linux. External certification remains open.
| Priority item | Delivered now | Remaining evidence |
|---|---|---|
| Source preflight | Uncached bounded scan; global IDs, schema fingerprint/drift, geometry/GeoParquet CRS, row/byte limits; CLI JSON reports | Operator CRS/field attestation and immutable production snapshots |
| Deadlines and cancellation | Bounded queue wait, initialization, execution and total deadlines; AbortSignal/client disconnects; safe native drain/recovery | Real cloud/native cooperation and supervisor restart tests |
| Observability | Queue/deadline/latency/RSS/cache metrics; P1 native spill/source-age samples, alert rules and dashboard assets | Live scrape/dashboard/alert-routing validation and agreed production thresholds |
| Capacity and freshness baseline | Load CLI with SLO gates; 12 fixture scenarios; overflow and TTL update checks; recorded development evidence | Production SLO agreement, Linux/cloud/ArcGIS soak and recovery qualification |
| Linux/container verification | Node 22.14.0/24.13.0 matrix; constrained read-only image smoke/scan on branches/MRs; evidence artifacts and protected publication | Actual successful pipeline/image digest; local Docker daemon is unavailable |
| ArcGIS 12.1 package certification | Correct SDK scaffold/export procedure, model adapter example and acceptance checklist | Linux SDK, genuine .cdpk and staging registration/client/query/recovery evidence |
Use operations and ArcGIS 12.1 Linux certification for the next release checks. The P1 build below is now implemented and locally tested. One worker remains the default; enable pooling only after representative measurements. Certify opt-in pass-through before releasing sources beyond full-fetch limits. CloudFS activation gates apply only to deployments enabling that adapter.
P0 — release gates
| Recommendation | Acceptance criteria | Effort |
|---|---|---|
| Certify the ArcGIS 12.1 Linux CDF package | Matching scaffold/runtime/native compatibility; real registration, service creation, IDs, fields, counts, spatial queries, client rendering, restarts, and overflow behavior | Medium |
| Qualify Linux/container deployment | Image build/scan, native reads, HTTP/auth probes, read-only filesystem, graceful signals; immutable digest recorded | Small–medium |
| Publish/qualify CloudFS for the pinned engine (when enabled) | Signed DuckDB 1.5.6 Windows/Linux builds; reviewed revision; native test suite without runtime overrides; real provider permission/expiry/rotation/egress/outage/load tests | Medium |
| Qualify real cloud sources | HTTPS/S3 permissions, outages, redirects/egress, credential rotation, source CRS/schema/IDs | Medium |
| Set capacity/freshness targets | Define latency/error/staleness and source-size limits; load/soak tests measure RSS, CPU, spill, queue, and recoverability | Medium |
| Complete platform controls | TLS, network policy, rate limits, secret/monitoring authentication, recovery runbook, and deployment ownership | Medium |
Assign GIS administrators to CDF certification, platform operators to deployment/recovery, and maintainers to correctness/capacity. Record evidence against exact revisions, engine versions, image digests, and source snapshots.
P1 — correctness and scale
The approved P1 scope is implemented. See capabilities and migration and design decisions. These are locally tested capabilities with explicit limits; P0 production gates remain open.
| Capability | Delivered | Remaining qualification/enhancement |
|---|---|---|
| ArcGIS pass-through query engine | Schema-allowlisted bound expression grammar; global ID/geometry qualification; fields, ID predicates, deterministic pages, complete bounded ID responses, filtered counts/projected extents and grouped statistics; opt-in getData adapter and query/schema routes | Compare subset and filtersApplied against the real 12.1 runtime; qualify metadata/projection/statistics and unsupported-parameter behavior. Add further GeoServices operations only with contracts and tests. |
| CRS and ArcGIS geometry support | EPSG source declarations, explicit XY transforms, WGS84 output, 3857/local input/extent support, points/multipoints/envelopes/paths/nested rings and topology/reference validation | Qualify local transformation grids/accuracy and actual ArcGIS rendering/axis conventions. More formats and broader spatial operations remain P2/P3. |
| Schema evolution and field compatibility | Versioned strict/additive native-field contracts, optional pinned fingerprint, runtime/preflight drift checks, complete CDF field metadata; safe integer/decimal strings, booleans and UTC dates | Certify mapped fields/client compatibility and approve production versions. Automated migrations/catalog schemas remain future work. |
| Measured connection pooling | Opt-in 1–8 exclusive native instances with divided total memory/threads, scratch separation, single installer, global admission and cancellation isolation; 1/2/4 fixture comparisons on Node 22/24; single worker default | Repeat isolated-process representative Linux/cloud/ArcGIS load/soak/SLO runs before increasing production workers. CloudFS stays one worker per process. |
| Data-aware cache freshness | TTL/stat/immutable-version/public-HTTPS strong-ETag identities, source invalidation, schema-aware bounded keys, before/after checks and freshness metadata; native external-file cache disabled | Qualify real origin semantics and CDF response caching. Immutable snapshots are required for consistency; observations are not transactions. Distributed mutation broadcasts/catalog transactions remain future work. |
| Production monitoring integration | Native buffer/spill/scratch samples with timestamps and failure counter; observed source modification/identity metrics; authenticated opt-in ServiceMonitor, configurable alert rules and Grafana dashboard | Import/scrape/routing tests on the actual platform and agreed thresholds. Current gauges can miss short spill peaks; supplement with host/disk/query profiling. |
Recorded fixture measurements demonstrate throughput improvements with added workers and higher RSS. They are development evidence, not production SLO certification.
P2 — connectors and workflows
| Feature | Acceptance criteria | Dependencies | Effort |
|---|---|---|---|
| Complete CloudFS connectors | Opt-in CSV/JSON/GeoParquet adapter and native VFS fixture implemented; qualify real SharePoint/OneDrive/Drive/Dropbox/TLS-agent accounts, refresh/rotation, glob semantics and cancellation | Compatible build (P0), dedicated workers | Medium |
| Fix/qualify CloudFS TLS-agent routing | Backend must recognize vfs+tls; verify certificate/hostname validation and denial cases; current build 4ca1432 only qualifies loopback vfs fixtures | Reviewed upstream fix and signed compatible build | Small–medium |
| Secure CloudFS SFTP | Upstream known-host/fingerprint verification, key rotation/rejection tests, Windows key-path parsing and interruption; no bypass flag | Upstream security fix and reviewed signed builds | Medium |
| CloudFS lakehouse/write workflows | Delta/Iceberg readers, catalogs and snapshots; separately authorized COPY TO/export jobs with read/write credentials, destination policy, atomicity and recovery tests | Qualified connectors/catalogs, explicit write API | Medium–large |
| Azure Blob/ADLS | Verified extension reads, managed identity/secret references, permission/outage/rotation tests | Cloud qualification | Medium |
| PostgreSQL/PostGIS | Correct read-only extension/connection APIs, geometry conversion, schema discovery, safe filters, pooling/outages | Schema/query layer | Medium |
| Iceberg/Delta | Real catalogs/tables; snapshots, evolution, deletes, auth, geometry encoding, extension-version compatibility | Connector contracts | Medium–large |
| More file formats | Qualify GeoPackage/Shapefile/FlatGeobuf/coordinate CSV; drivers, encodings, multi-layer files, invalid inputs, CRS | CRS/schema support | Medium |
| Stream ingestion/replay | Kafka/Redis/WebSocket adapters, durable IDs/checkpoints, deduplication, backpressure, validation, reconnect/replay | Durable state/schema | Large |
| Durable ETL jobs | Persistent jobs, idempotent batches, bounded workers, retries/dead-letter handling, schedules/restart recovery | Durable state/connectors | Large |
P3 — new product features
| Feature | User value and acceptance criteria | Dependencies | Effort |
|---|---|---|---|
| Configuration/source-preview UI | Validate and preview schema/data/connectivity with actionable errors; management authorization/audit; no secret disclosure | Stable admin API | Medium |
| Identity and tenant isolation | OIDC/ArcGIS identity, per-source policies, tenant-aware cache keys, durable sessions where needed, cross-tenant negative tests | Authorization design | Large |
| GraphQL/live subscriptions | Real resolver contracts, per-operation authorization, depth/cost limits, durable delivery, disconnect cleanup | Query/identity/stream foundations | Large |
| Spatial analytics/graph tools | Unit-correct projected/geodesic operations, reproducible statistics, qualified graph extensions, bounded cost | CRS/query foundations | Medium–large |
| Safe natural-language assistance | Preview/explain allowlisted plans; validate syntax/schema, enforce user permissions/resource limits, explicit approval for writes | Parser/identity | Large |
| Connector/plugin SDK | Versioned interfaces, lifecycle ownership, capability boundaries, compatibility checks, real sample integration tests | Stable adapters | Medium–large |
| Reproducible releases | SBOM/provenance, signed CDF/container artifacts, native-platform matrix, compatibility manifest, preflight tooling | Qualified release process | Medium |
Delivery policy
Each enhancement needs a user scenario, compatibility scope, security/resource boundary, measurable real integration tests, updated operational docs, and migration notes. Do not advertise or enable a feature solely because a module/flag exists. Ship small qualified capabilities incrementally.