Storage connections and transfers
The Data page connects a workspace to buckets you already own and moves data between them and the workspace in bulk. A transfer runs on the platform, not on a runtime, so nothing has to be running while it copies.
Connections
Add connection on the Connections tab asks for a provider first:
| Provider | Sign-in | Notes |
|---|---|---|
| Amazon S3 | IAM role (recommended) or access keys | Your own bucket. Validation detects the bucket's real region. |
| Public S3 bucket | Anonymous | A bucket that allows anonymous reads, such as an open dataset. |
| Cloudflare R2 | Access keys | S3-compatible. |
| Backblaze B2 | Access keys | S3-compatible. |
| Wasabi | Access keys | S3-compatible. |
| DigitalOcean Spaces | Access keys | S3-compatible; the container is a Space. |
| Other S3-compatible | Access keys or anonymous | MinIO, Ceph, or any S3 API over HTTPS; you give the endpoint URL. |
| Google Cloud Storage | Workload Identity Federation (recommended) or a service-account key | |
| Azure Blob Storage | Container SAS token | A container in a storage account. |
Then:
- Name (how it appears in the workspace), the bucket, Space or container, and an optional Prefix: the root folder inside the bucket that the connection is limited to.
- Access mode: read-only, which allows imports into the workspace, or read-write, which also allows exports. Anonymous connections are always read-only.
- Authentication for the provider. For an IAM role the Setup guide gives you the connector's role ARN, an External ID, and the trust and permissions policies to paste into your AWS account. For Workload Identity Federation it asks for the project number, pool and provider ids, and an optional service account to impersonate.
Credentials are write-only: once saved they are never shown again, and Edit lets you supply new ones. Each connection has Validate (checks the credentials and the bucket), Browse (lists what the connection can see), Setup guide and Delete. Viewers see connections and can browse them; adding, editing and deleting need an editor.
Transfers
New transfer on the Transfers tab, or Import from cloud… and Export to… on the Files page, starts a copy:
- Direction: import into the workspace, or export from it. Exporting needs a read-write connection.
- Connection, a remote path relative to the connection's prefix, and the workspace folder.
- Existing files: Skip existing (only copy what the destination does not have; re-runs resume), Update if newer (also replace files whose source copy is newer), or Overwrite.
- Verification: Size (fast) or Checksum (slower).
- Include patterns and Exclude patterns to copy part of a tree, an optional Bandwidth limit (Mbps), and a Runner size (large runners copy many small files faster).
- Estimate (whole path) counts files and bytes before you commit.
A transfer copies. It never deletes anything from your bucket or your workspace. Progress shows on the Transfers tab with Transfer details for the full record, and a transfer can be cancelled while it is queued or running.
Limits and billing
- One transfer executes per workspace at a time, with up to 5 queued.
- A single transfer may run for at most 24 hours.
- An import is checked against your storage pool's free space before it starts; a dataset larger than the space left is refused up front. To work with data larger than your pool, read it from the bucket directly from a cell instead, with the credentials in workspace secrets.
- Transfers are free. Imported data counts against your pool like any other file.
Related
- Files: where imported data lands.
- Packages in your runtime: what belongs in the workspace and what belongs on the runtime's scratch disk.
- Limits: every number on one page.