Imports

Imports bring products, customers, reviews and partner records into Litestore from another platform or a file. Read this page when you need to investigate an import that failed, or when you add a new kind of import.

Litestore records every import as a run in the database.

An ImportRun row records the rows the import tried to write, the ones that succeeded and how far it got. After an import, read the row, run the import again or download the rows that failed.

Sources

ImportSource is a closed set.

The list is closed on purpose. If a source were listed before it worked, the store would appear to support something it doesn't. A new source joins the union together with its adapter, never before.

There is no adapter for importing from a URL. Creating a product from a URL is a separate path that uses AI, and it does not create an ImportRun.

Import runs

Each run records its source and mode, keeps running counts and a cursor, and ends with a downloadable error report.

The row has these fields:

source
sourceAccountId
entityScope
mode
status
cursor
checkpoint
counts
startedAt
completedAt
errorReport

sourceAccountId identifies what the run read from. For Shopify it is the shop id, and for an uploaded file it is a generated batch id. mode is either full or incremental.

status moves through these values.

The schema also allows a fourth status, paused.

If a run imported nothing and collected errors, Litestore marks it failed instead of completed, so the Import runs card on Settings → Developer shows it as a failure.

Each batch adds to counts instead of overwriting it. Litestore adds the batch's imported, updated, skipped and errors to the running total. The number on the card is the total so far, not the number from the last batch.

Cursor

cursor stores the token of the last page the run processed. Litestore writes it after every batch, together with that batch's counts, not just once at the end.

The cursor shows how far a run got. You cannot restart a run from it, because there is no resume mode:

// "resume" removed — recovery is idempotent re-run (externalId dedupe), never a cursor.

To recover from a failed import, you run it again. Shopify imports use bulk JSONL and have no cursor to resume from, so the developer card offers Re-run instead of Continue.

Running again is safe because of the source records described below. A second run finds the rows it already created by their external id and updates them instead of inserting duplicates. You recover a failed import by running it again, which is safe because a second run updates the rows the first one created.

Source records

ImportSourceRecord links an identifier from the source platform to the record Litestore created from it. Four columns together identify each link:

source
sourceAccountId
entityType
externalId

These four columns form a unique constraint, not just an index. Because of it, importing again updates the existing record instead of creating a second copy. It also traces each product back to the source row it came from.

Each record also stores the internal id, a checksum of the raw data to detect changes, and the time it was last imported.

Not every source matches records the same way. Collabs match by email. Review CSV rows have no external ids at all, so the Review table's own unique constraints prevent duplicates instead.

Errors

While the run reads data, Litestore collects a failure for each bad row. At the end, it writes them all to the run as a report. Each row error has one of these types:

validation
duplicate
reference
api
unknown

The report contains the total number of errors, the errors themselves, a count for each type, and whether the collector reached its limit. The limit is 1,000 errors. After that, the run keeps importing but stops recording new failures.

Adding imports

In both cases, every import must follow these steps.

The most important design question is this: can this import be run twice safely? If the answer depends on an external identifier, store that identifier in ImportSourceRecord. If the source has no external identifier, the target table has to enforce uniqueness.

Related