CloudFS integration

CloudFS is a community filesystem extension. This provider now has an opt-in read adapter, credential references, and native qualification tests.

Compatibility status — October 2, 2026

Production activation is blocked on a compatible signed build. The app keeps @duckdb/node-api 1.5.6-r.1 / DuckDB 1.5.6. The community repository returned 404 for CloudFS on 1.5.6 for Windows AMD64 and Linux AMD64. A signed CloudFS build is available for DuckDB 1.5.4 on both platforms; its installed extension_version is 4ca1432.

Do not downgrade the production engine or disable signature/version checks to load that binary. DuckDB 1.5.6 contains newer security and correctness fixes. Obtain a reviewed, signed build for the pinned engine and deployment platform, then complete the qualification below.

The reviewed community descriptor selects source revision 4ca1432dfa01dffae0b167ba96b44720f0e39d83. CloudFS identifies that build as 4ca1432. The app checks this value before creating credentials. A newer build requires review, tests, and an explicit cloudfs.extensionVersion update; installation does not certify the build.

Provider and format scope

CSV uses read_csv_auto; JSON uses read_json_auto; GeoParquet uses read_parquet. Asterisk globs (* and recursive **) are passed as bound locations to these readers. URI query strings/fragments are forbidden, so question-mark globs are not accepted; bracket/richer matching is not qualified. Existing limits, ID validation, WGS84 geometry, sorting, spatial filters, cache bounds, and full-fetch overflow behavior apply across files. Every matched file must have compatible schema/geometry and globally unique IDs.

JSON means tabular records with an explicit geometry column, for example:

[{"id":1,"name":"Point","geom":"POINT (-74 40)"}]

It does not flatten arbitrary GeoJSON FeatureCollections or create geometry from longitude/latitude automatically. JSON/CSV default to geometryEncoding=wkt; set WKB/native explicitly when appropriate. GeoJSON through GDAL remains a local-file reader.

Upstream advertises Delta, Iceberg, and COPY TO. Those capabilities require additional table readers/extensions/catalogs or a write workflow; they are not exposed by this read-only app. Their acceptance criteria are in roadmap.md.

Configuration and credentials

Start with the example. It contains illustrative locations and environment variable names, with no secret values. Configuration validation succeeds without credentials; connection initialization requires credentials for the providers actually used.

Minimal example, to activate only after a matching build is available:

{
  "dataSources": {
    "default": {
      "type": "geoparquet",
      "url": "spfs://tenant.sharepoint.com/sites/GIS/Shared Documents/points.parquet",
      "idField": "id",
      "geometryColumn": "geom",
      "crs": 4326
    }
  },
  "cloudfs": {
    "enabled": true,
    "extensionVersion": "4ca1432",
    "credentials": {
      "sharepoint": {
        "provider": "token",
        "tokenEnv": "CDF_SHAREPOINT_ACCESS_TOKEN"
      }
    }
  },
  "database": {
    "installExtensions": false,
    "extensionDirectory": "/app/extensions"
  }
}

Set tokenEnv to the name of a secret-manager-injected environment variable. The token must be a provider API access token with read access to the selected files, not the diagnostic API's CDF_API_TOKEN. The adapter creates CREATE TEMPORARY SECRET statements and never writes persistent DuckDB secrets. Token literals are escaped; initialization and source-read errors omit credential contents.

Static tokens have no refresh workflow in this adapter. Acquire them externally, rotate before expiry, and restart the worker. Interactive OAuth and application client_credentials are not supported by the reviewed extension build's adapter. Main-branch upstream documentation can describe APIs absent from the distributed revision.

For Google service accounts, use:

"gdrive": {
  "provider": "service_account",
  "keyJsonEnv": "CDF_GOOGLE_SERVICE_ACCOUNT_JSON"
}

The environment variable must contain the JSON credential document, not a filename. Keep keys outside config/source control; qualify permissions, token refresh, and rotation with the real account.

Isolation and transport

The reviewed secret registry and extension utilities keep process-global state. The adapter permits one active CloudFS model per process and one credential profile per provider. Multiple sources may share that profile. Use dedicated worker processes for separate accounts/trust domains; replica processes can each run one model. Do not combine this adapter with other CloudFS loaders in the same process.

The reviewed secret registry also does not declare credential redaction keys. There is no raw SQL/secret-inspection endpoint in this app. Do not log statements, inspect DuckDB secrets in shared tooling, expose the native connection to untrusted code, or allow persistent secret creation.

The reviewed VFS backend parses vfs+tls:// but inherits scheme routing that only accepts vfs://. The adapter rejects vfs+tls for build 4ca1432. A newer reviewed build must fix routing and pass TLS/certificate/authorization tests before setting an updated extensionVersion. A controlled local TLS relay with certificate verification is a possible platform workaround; it is not built into this app.

Use vfs+tls:// for remote agents once the routing fix is qualified. Plain vfs:// is accepted only for literal loopback addresses 127.0.0.1 and ::1. Configure agent authentication, read-only roots, TLS certificates, and egress explicitly. The SFTP backend proceeds from SSH handshake to authentication without host-key validation, so SFTP is blocked here.

Community extensions execute native code with worker privileges. Review and scan the exact source/build, keep configuration operator-controlled, and restrict egress including redirects. Provider-side native I/O cancellation and caching are not certified by the core query timeout/cache limits.

Installation and qualification

The default image installs official extensions only. After a compatible, reviewed CloudFS build is published:

docker build --build-arg WITH_CLOUDFS=true --build-arg CLOUDFS_EXTENSION_VERSION=4ca1432 -t duckdb-arcgis-provider:cloudfs .

This currently fails on 1.5.6 instead of silently using an incompatible binary. Match cloudfs.extensionVersion to the reviewed image build. Use database.installExtensions=false in production and inject credential env vars through container/Kubernetes secrets. The chart accepts providerSecretEnv entries with name, secretName, and key; these reference existing Secrets without inline values. For example, name=CDF_SHAREPOINT_ACCESS_TOKEN, secretName=cloudfs-read, key=sharepoint-token. Runtime/auth variable names are reserved. Credential changes require worker restart.

For a development host with a compatible engine/build, the installer also accepts WITH_CLOUDFS=true and DUCKDB_EXTENSION_DIRECTORY. Runtime installation when database.installExtensions=true explicitly uses INSTALL cloudfs FROM community. LOAD still requires a version/platform-compatible signed extension.

node scripts/cdf-cli.js validate-config -f config/examples/cloudfs.json
npm run test:cloudfs

The optional native suite uses an authenticated local HTTP agent fixture and real CloudFS CSV/JSON/GeoParquet/glob reads, spatial predicates, pagination, and temporary secrets. It requires repository access or preinstalled extensions and runs in its own process. On the pinned runtime it currently fails with CLOUDFS_UNAVAILABLE. Normal npm run build includes CloudFS validation/lifecycle/security tests; it does not pretend native qualification passed.

For maintainers reproducing the upstream 1.5.4 build, CLOUDFS_TEST_NODE_API can point to an isolated Node API installation for this test suite only. Never use that override for deployment; neither app dependencies nor runtime selection support it. Run only the native suite in that dedicated process to avoid loading two native DuckDB versions together.

Before release, run the suite without overrides on the supported engine/platform, then test every real provider's permission denial, token expiry/rotation, outages, TLS, redirects, schema/CRS/IDs, glob ordering, large files, freshness, memory, and cancellation. A local agent fixture is not SharePoint/OneDrive/Google/Dropbox or production-agent certification.