Skip to content

Data & automation

Data pipelines & inventory synchronization

Scheduled scrapers that keep a Shopify store’s inventory in sync with six vendors that offer no APIs.

Client
TBA, USA
Services
Custom software development, Data engineering
Date
June 2021
Data pipeline overview

TBA is a one-stop online store for home improvement, running on Shopify.

The challenge

The store had to keep inventory in sync and create new products from several vendors, none of which offered an API. Every product also had to match Shopify’s data requirements.

What we built

We built a scraper for each of the six partner vendors, collecting descriptions, pricing and images so the store always reflects what the vendors actually have.

The sync jobs run on a schedule or on demand, each deployed on AWS Lambda for reliability, and automated email logs record every change.