USI-Bot
USI (Universal Scraper & Integrator) is a specific-purpose web scraping system engineered for reliability, precision, and automation. Using JavaScript and Puppeteer, it extracts and caches relevant data from selected online sources, processes the content to reduce noise and normalize structure, and stores the refined results in a PostgreSQL database. USI also features automated report generation and an interactive dashboard for data visualization, while its Shopify GraphQL integration enables direct synchronization of cleaned data to storefronts — closing the loop between extraction and deployment.
Timeline
4 months
Team
2 full-stack developers
Key Features
Puppeteer-based scraping engine
PostgreSQL-backed ETL pipeline
CSV report generation & dashboard analytics
Shopify GraphQL integration
Technologies Used
Node.js,PuppeteerPostgreSQLShopify GraphQL APIDocker
Challenges
Handling dynamic, JavaScript-heavy websites protected by anti-bot measures
Maintaining clean, normalized data across multiple sources
Automating large-scale data transformations and visual reporting
Integrating processed data directly into Shopify with minimal delay
Solutions
Built Puppeteer-based crawling scripts for real-browser scraping and caching
Implemented data cleaning and noise reduction pipelines
Automated data loading into PostgreSQL and CSV report generation
Developed Shopify GraphQL integration for seamless data sync
Results & Impact
Accurately extracted and processed targeted data from multiple sources
educed manual intervention by 90% via full automation
Enabled real-time Shopify updates and streamlined data workflows
Delivered consistent, structured CSV reports and dashboard insights
More work
- DealMeal: AI Deal FinderAI-powered food deals app with DealRanker chatbot for instant savings. Discover the best food deals from multiple sources in one smart platform.
- Sailor: AI Style & SaveAI-powered fashion deals app with Captain AI style assistant. Get personalized recommendations and exclusive discounts from top brands.
- QutlineQutline is a unified, mobile-first restaurant growth platform that connects ordering, menu management, kitchen workflows, and analytics into one seamless ecosystem — empowering restaurants to operate smarter and deliver exceptional dining experiences.
- Rhymo - Persian Rhyme Finder AppRhymo is a sophisticated Persian poetry and rhyme-finding mobile application that helps users discover rhyming words and phrases in Persian/Farsi language using advanced phonetic and traditional rhyme algorithms.
- ETL BotETL Bot is a high-performance data automation tool that crawls Shopify stores, extracts and processes product data, and loads it into a MongoDB database — all managed through an intuitive Vue/NuxtJS admin dashboard.
- DetailmaxxDetailmaxx is a mobile-focused platform that connects car wash owners, employees, and customers through real-time scheduling, operations tracking, and financial analytics — empowering teams to run smarter, more efficient businesses.
- Shopify.rsShopify.rs is a Rust-based automation system designed to process massive datasets and synchronize them with Shopify, ensuring fast, reliable, and scalable data and image uploads.
- CollegemateCollegemate is a modern web platform connecting students and mentors through interactive scheduling, video conferencing, and real-time analytics — built to create engaging, responsive learning experiences across devices.