DuckDB ArcGIS Custom Data Feed Provider

A read-only DuckDB provider model for ArcGIS Custom Data Feeds, with a standalone diagnostic API.

Status: hardened core with local automated tests. Production release requires the environment checks in the review report. No live ArcGIS deployment is certified. Advanced modules in src/experimental/ are prototypes and are excluded from the distributable package and container.

Run locally

Use Node.js 22.14+ (22 or 24) and npm.

npm ci
npm run build
npm start

The included CSV has three WGS84 point features with stable integer IDs. DuckDB installs its official spatial extension on first local use; extension-repository access is required.

curl http://127.0.0.1:3000/ready
curl "http://127.0.0.1:3000/api/sources/default/features?limit=2"
node scripts/cdf-cli.js preflight -d default
node scripts/cdf-cli.js load-test -d default --full-fetch -q 100 -c 4

Unauthenticated local use is allowed only on loopback. Set CDF_API_TOKEN to a random token of at least 32 characters for production or a non-loopback bind. Send Authorization: Bearer <token>. Terminate TLS at a controlled proxy.

Supported scope

Capability	Implementation and limits
Local JSON records	WKT geometry through read_json_auto
CloudFS (opt-in)	SharePoint/OneDrive/Google Drive/Dropbox/agent CSV, JSON, GeoParquet; compatible signed 1.5.6 build and real-provider qualification pending; SFTP blocked
Local CSV	WKT geometry through read_csv_auto
Local GeoParquet	Native geometry or explicit WKB through read_parquet
Local geospatial files	GDAL ST_Read; tested with GeoJSON, other formats require qualification
HTTPS/S3 GeoParquet	Reader/extensions wired; endpoints and credentials need environment tests
Geometry and identity	Explicit EPSG/XY projection to WGS84; ArcGIS points/envelopes/paths/rings; nullable geometry and globally unique positive 32-bit IDs
ArcGIS model	Default full-fetch; opt-in restricted pass-through with paging/counts/extents/IDs/statistics and explicit CDF fields
Diagnostic API	Source listing/schema, safe query endpoint, pagination/intersects, health/readiness and native/source metrics
Resource controls	Bounded queue/version-aware cache/results, exclusive workers, native cancellation, shared total engine budget and draining close

The default model uses full-fetch: the CDF runtime queries the complete bounded collection. Set source queryMode: "pass-through" for larger datasets and the supported query subset. See P1 contracts, configuration and migration. Diagnostic routes return GeoJSON/CDF metadata and are separate from ArcGIS FeatureServer routes.

Azure, PostGIS, Iceberg, Delta, Kafka, Redis, graph queries, GraphQL, WebSocket, multi-tenancy, ETL, plugins, and AI assistance remain experimental. Earlier documentation described proposed functionality rather than a deployed end-to-end implementation.

Configuration

Default: config/config.json; select another file with CDF_CONFIG, or pass configPath / config to the model constructor. Missing/invalid configuration and unresolved placeholders stop startup. Relative data paths resolve against the application root.

{
  "dataSources": {
    "default": {
      "type": "csv", "path": "data/sample.csv",
      "idField": "id", "geometryColumn": "geom",
      "geometryEncoding": "wkt", "crs": 4326
    }
  }
}

Older configurations need migration: ID/geometry fields are required, TTL uses milliseconds, and enabled unsupported integrations fail validation.

Documentation and checks

npm run build runs lint, tests, and default configuration validation. npm run test:coverage measures supported runtime code, excluding prototypes. Tests use real DuckDB spatial reads and lifecycle fault injection. Check dependencies with npm audit and inspect distribution with npm pack --dry-run. CI validates Helm and blocks dependency/image security findings.

Local checks do not establish ArcGIS/native-platform compatibility, cloud authorization, or load capacity. These remain release gates. Licensed under MIT.