Imports

Imports bring products, customers, reviews and partner records into Litestore from another platform or a file. Read this page when you need to understand an import that failed, or when you add a new kind of import.

Litestore tracks every import as a run. It does not simply accept an upload and forget about it.

An ImportRun row records what the import tried to do, what succeeded and how far it got. If an upload just drops, you are left guessing. With a run, you can read the row afterwards, run the import again and download the rows that failed.

The four sources

ImportSource is a closed set:

shopify
csv
reviews
collabs

• shopify imports the catalog through the Shopify Admin OAuth connection.
• csv imports rows from an uploaded file.
• reviews imports existing reviews from another platform.
• collabs imports partner records.

The list is closed on purpose. If a source were listed before it worked, the store would appear to support something it doesn't. A new source joins the union together with its adapter, never before.

There is no adapter for importing from a URL. Creating a product from a URL is a separate path that uses AI, and it does not create an ImportRun.

A run

Each run records what it imports and how, keeps running counts and a cursor, and ends with an error report you can download.

The row has these fields:

source
sourceAccountId
entityScope
mode
status
cursor
checkpoint
counts
startedAt
completedAt
errorReport

sourceAccountId identifies what the run read from. For Shopify it is the shop id, and for an uploaded file it is a generated batch id.

mode is either full or incremental.

status moves through these values:

1. pending means the run row exists but nothing has been read yet.
2. running means the import has started and startedAt is set.
3. completed or failed means the import has ended and completedAt is set.

The schema also allows a fourth status, paused.

If a run imported nothing and collected errors, Litestore marks it failed instead of completed. The importFailed operator task looks for exactly that.

Each batch adds to counts instead of overwriting it. Litestore adds the batch's imported, updated, skipped and errors to the running total. The number on the card is the total so far, not the number from the last batch.

The cursor

cursor stores the token of the last page the run processed. Litestore writes it after every batch, together with that batch's counts, not just once at the end.

The cursor shows how far a run got. You cannot restart a run from it, because there is no resume mode:

// "resume" removed — recovery is idempotent re-run (externalId dedupe), never a cursor.

To recover from a failed import, you run it again. Shopify imports use bulk JSONL and have no cursor to resume from, so the developer card offers Re-run instead of Continue.

Running again is safe because of the source records described below. A second run finds the rows it already created by their external id and updates them instead of inserting duplicates. You can recover a failed import because running it twice is safe, not because it saved a checkpoint.

Source records

ImportSourceRecord links an identifier from the source platform to the record Litestore created from it.

Four columns together identify each link:

source
sourceAccountId
entityType
externalId

These four columns form a unique constraint, not just an index. Because of it, importing again updates the existing record instead of creating a second copy. It also lets you trace a product back to the source row it came from.

Each record also stores the internal id, a checksum of the raw data to detect changes, and the time it was last imported.

Not every source matches records the same way. Collabs match by email. Review CSV rows have no external ids at all, so the Review table's own unique constraints prevent duplicates instead.

Errors

While the run reads data, Litestore collects a failure for each bad row. At the end, it writes them all to the run as a report.

Each row error has one of these types:

validation
duplicate
reference
api
unknown

The report contains the total number of errors, the errors themselves, a count for each type, and whether the collector reached its limit. The limit is 1,000 errors. After that, the run keeps importing but stops recording new failures.

Where a new import fits

If the data comes from another platform's API, you need to write an adapter that implements ImportAdapter. Its testConnection and getShopInfo methods handle the connection. Its fetchProducts and fetchCustomers methods are async generators that yield batches along with the next cursor. Fetching collections, orders and entity counts is optional.

If an operator uploads the data as a file, you don't need an adapter. The importers under server/admin/import/ create their own run and use the same runtime. They give the run a specific source label, so the card names the pipeline that actually ran.

In both cases, every import must follow these steps:

1. Create the run before you read anything.
2. Write counts and a cursor after each batch.
3. Finish with completeImportRun or failImportRun, so the row always records how the run ended.

The most important design question is this:

Can this import be run twice safely? If the answer depends on an external identifier, store that identifier in ImportSourceRecord. If the source has no external identifier, the target table has to enforce uniqueness.

Related