Source preflight, capacity and operations

Run source qualification on the same Linux runtime, permissions, immutable data snapshot, and resource limits that will serve production. Readiness only probes the database. The CLI reports are local administrative tools.

Preflight before every source/configuration release

node scripts/cdf-cli.js validate-config -f config/local.json
node scripts/cdf-cli.js preflight -f config/local.json -o preflight-report.json
node scripts/cdf-cli.js preflight -f config/local.json -d default --max-scan-rows 100000 --expect-schema <previous-sha256>

By default every configured source is scanned without using the query cache. Reports include discovered column names/types, a SHA-256 schema fingerprint, total/scanned rows, geometry family, null geometries, and estimated serialized bytes. They contain no source paths, credentials, SQL, or row values. Column names remain visible; handle reports as operational metadata.

Exit code 1 means qualification failed. Resolve the issues before publication:

The scan cap defaults to 100,000 rows and accepts 1–1,000,000. A partial scan never passes. Full-fetch eligibility requires all scanned rows to pass, at most maxRecordCount features, and both native rows and the GeoJSON collection within 8 MiB. Memory also depends on native chunks, property sizes, and the ID set; the cap is not a hard process-memory limit. GeoParquet metadata inspection is capped at 1,000 entries.

Preflight and service reads are separate operations. External files are not locked by a database transaction. Matching counts/fingerprints cannot detect every concurrent replacement. Release immutable/quiescent snapshots. Schema fingerprints detect column/type/configuration changes, not content changes. Coordinate range checks cannot establish CRS, axis order, ArcGIS attribute compatibility or future ID stability. Native preflight/query qualification now checks topology; actual client compatibility still needs staging evidence. Attest source CRS when metadata is absent; certify fields and clients on ArcGIS.

Representative load and freshness targets

node scripts/cdf-cli.js load-test -f config/local.json -d default --full-fetch -q 1000 -c 4 --cache disabled --max-duration-ms 60000 --max-p95-ms 500 --max-error-rate 0 -o load-report.json
node scripts/cdf-cli.js load-test -f config/local.json -d default --full-fetch -q 1000 -c 4 --cache warm -o warm-report.json
npm run test:capacity -- -o capacity-report.json

The 500 ms target above is an example, not an agreed production SLO. Set latency, allowed errors, update staleness, source-size/byte limits, and recovery time with GIS/platform owners. Repeat under real cloud latency, outage, rotation, CPU/memory constraints, and multiple replicas.

load-test reports attempted/successful/failed requests, bounded error codes, p50/p95/p99, successful-request p95, throughput, CPU time, sampled RSS, cache usage, and queue state. Limits: 10,000 requests, 256 workers, and a maximum one-hour run deadline (default 60 seconds). A deadline stops admission and cancels outstanding requests; incomplete runs fail. The default allowed error fraction is zero. Errors are included in overall latency; inspect success latency and error rate together.

warm primes one identical query; cold starts with an empty enabled cache; disabled measures native reads. Initialization is completed before timing. Full-fetch uses the complete configured record limit; use --limit only for diagnostic pages. Increasing caller concurrency exercises the configured exclusive workers and global queue; one worker remains the default. This is an in-process model test; it excludes HTTP/TLS/ArcGIS serialization overhead and does not replace an external soak test. RSS sampling is approximate and may miss short peaks; native spill gauges are sampled and may miss peaks; supplement with host/provider monitoring.

The capacity script generates 10/1,000/2,000-row CSVs and runs 12 combinations of one/four workers and disabled/warmed cache, checks 2,001-row overflow rejection, then updates a file to verify TTL expiry. Temporary fixtures are removed after native shutdown. It does not contact external providers.

Recorded Windows Node 22 fixture evidence: all 12 scenarios passed with 20 requests each and zero errors. Across this run p95 ranged from 0.12–86.42 ms and sampled process RSS from 80.36–150.23 MiB. Scenarios ran sequentially in one process, so RSS includes earlier native allocations. The 100 ms TTL check retained the prior value before expiry and observed the update after 120 ms. These are development observations, not Linux/ArcGIS capacity commitments.

Deadlines and cancellation

Defaults: queue wait 15 seconds, initialization 30 seconds, execution 15 seconds, and total admitted-operation lifetime 30 seconds. The first applicable deadline wins. Queue expiry immediately releases the waiting slot. Caller errors use QUEUE_TIMEOUT, INITIALIZATION_TIMEOUT, QUERY_TIMEOUT, or REQUEST_TIMEOUT (504); full admission returns BUSY (503).

An AbortSignal can cancel model queries and preflight. The standalone server forwards client disconnects. Queued cancellations never execute; active cancellations interrupt only their own native operation. REQUEST_CANCELLED is code 499 for direct callers; disconnected clients receive no response.

After a caller deadline, initialization/native work retains ownership until it settles. Later requests cannot reuse that connection. Initialization that has not returned a handle cannot be interrupted. Deadlines and cancellation depend on event-loop/native cooperation; they are not hard CPU or network termination. Direct connect() is a low-level lifecycle method without an admission deadline; use checkReadiness() for bounded startup. An uncooperative worker needs supervisor termination/restart.

Monitoring and recovery

Authenticated /metrics exposes arcgis_queue_pending/waiting/active/high_water, arcgis_queue_failures_total with bounded reason labels, and queue_wait/execution/request_duration_seconds histograms. It also exposes process RSS/heap, cache bytes/entries/hits/misses, request/error counters and uptime. Histograms use cumulative fixed buckets in seconds. Queue pending includes timed-out work still holding native ownership. Metrics and cache are per process and reset on restart.

Suggested alerts: sustained admission saturation, deadline/error rate above the agreed SLO, p95 deterioration, RSS approaching the container limit, readiness failures, restart bursts, and source age above the staleness target. Readiness alone cannot establish data freshness. TTL is not an end-to-end freshness guarantee: ArcGIS/proxy caches and upstream publication delays also apply.

For a stuck worker, preserve sanitized metrics, source snapshot/version, package/image digest, and timestamps. Remove it from service, allow drain, then have the supervisor terminate/restart if native work does not settle. SIGTERM has a 30-second server shutdown deadline; allow at least 35 seconds in the platform. Recheck readiness, preflight, representative features, and ArcGIS clients. Roll back the package/config/source together using the deployment runbook.

Use P1 operations for schema approvals, source versions, pool comparisons, authenticated ServiceMonitor/Grafana assets and alert tuning. SOURCE_CHANGED/SCHEMA_CHANGED require a stable snapshot or reviewed approval; SOURCE_IDENTITY_UNAVAILABLE fails closed. Never bypass freshness checks to recover an unavailable mutable source.