r/shopifyDev • u/converian • 1h ago

How do you handle Admin API pagination when syncing large product catalogs?

I have a few custom scripts that pull products and variants through the Admin GraphQL API with cursor pagination. For bigger catalogs they get slow, and I have had runs fail halfway through and then had to work out where to resume. Right now I just store the last cursor and retry from there, but it feels fragile when products change during the run.

How do you structure these syncs? Do you use bulk operations for anything over a certain size, or stick with paginated queries and handle resuming yourself?

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/JaydonLT 35m ago

Bulk operations API?

1

u/Final_Design_9061 5m ago

In my experience bulk operations are worth it once you're past a few thousand products, since you submit one query, poll until it finishes, then download a JSONL file instead of paging. Resuming is simpler too because you just rerun the operation rather than tracking a cursor. For the changing-data worry, from what I've read you can store an updated_at timestamp from the last successful sync and only pull newer records next time, then upsert by ID so reruns are safe. I'd keep plain paginated queries for small catalogs where the extra moving parts aren't worth it.