True Talent Solutions Partners (via agency) · Engineering · Live in: anywhere in the world
Posted 2026-11-02 · Verified 2026-11-04
About the Role Join a growing data infrastructure team building large-scale web data systems for AI research and model development. You will develop reliable crawling and data acquisition infrastructure across diverse websites, helping research teams create datasets for training and evaluating advanced models. What You'll Do Build and maintain web crawlers across social, travel, multilingual, and other domains. Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day. Adapt crawling methods to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites. Build pipelines to clean, deduplicate, filter, and normalize web data at... Read the full ad