Imports

An import is a tracked run rather than a fire-and-forget upload.

ImportRun records what was attempted, what succeeded, and how far it got.

That distinction is the whole design. A dropped upload leaves you guessing; a run leaves a row you can read afterwards, re-run, and download the failures from.

The four sources

ImportSource is a closed set:

shopify
csv
reviews
collabs

• shopify brings in the catalog, over the Shopify Admin OAuth connection.
• csv brings in uploaded rows.
• reviews brings in existing reviews from another platform.
• collabs brings in partner records.

The set is closed on purpose. A declared-but-unimplemented source reads as a capability the store does not have, so a union member is added with its adapter, never before it.

There is no URL import adapter. Pulling a product from a URL is a separate AI-assisted creation path, not an import run, and it does not produce an ImportRun.

A run

Each run carries its scope and mode, running counts, a cursor, and a downloadable error report.

The row is:

source
sourceAccountId
entityScope
mode
status
cursor
checkpoint
counts
startedAt
completedAt
errorReport

sourceAccountId is what the run was reading from — a Shopify shop id, or a generated batch id for an uploaded file.

mode is full or incremental.

status moves through:

1. pending — the run row exists, nothing has been read yet
2. running — startedAt is set
3. completed or failed — completedAt is set

paused is the fourth status the schema allows.

A run that imported nothing and collected errors is recorded as failed rather than completed. That is what the importFailed operator task keys on.

counts accumulates rather than being overwritten. Each batch adds its imported, updated, skipped and errors to the running total, so the number on the card is the number so far, not the number in the last batch.

The cursor

cursor records the last processed page token, written after every batch alongside that batch's counts rather than once at the end.

It is a record of progress, not a checkpoint to restart from. There is no resume mode:

// "resume" removed — recovery is idempotent re-run (externalId dedupe), never a cursor.

Recovery is re-running the import. Shopify imports are bulk-JSONL and have no cursor to resume from at all, so the developer card offers a Re-run rather than a Continue.

That is safe because of the source-record mapping below: a re-run matches what it already created by external id and updates those rows instead of inserting duplicates. Idempotency is what makes a failed import recoverable, not a checkpoint.

Source records

ImportSourceRecord maps an external identifier to what was created locally.

The mapping is keyed by four columns together:

source
sourceAccountId
entityType
externalId

That is a unique constraint, not just an index. It is what makes a re-import an update rather than a second copy, and what lets you trace a product back to the row it came from.

Each record also stores the internal id, a checksum of the raw data for change detection, and when it was last imported.

Not every source keys on the same thing. Collabs key their idempotency by email, and review CSV rows carry no external ids at all — the Review table's own unique constraints guard those instead.

Errors

Failures are collected per row while the run streams, then written to the run as a report at the end.

Each row error is typed:

validation
duplicate
reference
api
unknown

The report carries the total count, the errors themselves, a breakdown by type, and whether the collector hit its cap. The cap is 1,000 errors — past that, the run keeps importing but stops recording new failures.

Where a new import fits

If the data comes from an external platform over an API, it needs an adapter. ImportAdapter is the contract: testConnection and getShopInfo for the connection, then fetchProducts and fetchCustomers as async generators that yield batches and a next cursor. Collections, orders and entity counts are optional.

If the data arrives as a file an operator uploads, it does not need an adapter. The in-app importers under server/admin/import/ create their own run, qualify the source label so the card names the actual pipeline, and drive the same runtime.

Either way, three things are not optional:

1. Create the run before you read anything
2. Write counts and a cursor as you go
3. Finish with completeImportRun or failImportRun, so the row has an end state

The question that decides the shape is:

Can this import be run twice safely? If the answer depends on an external identifier, that identifier belongs in ImportSourceRecord. If there is no external identifier, the uniqueness has to come from the target table.

Related