apply is seeking a Scraping Tech Lead Engineer to serve as the technical leader for a high-load web scraping platform. You will drive architectural decisions and shape the future of the scraping infrastructure, which supports both cloud and on-premises deployments.
What You'll Do
- Design architecture for the data-plane, including high-parallel crawlers, session management, anti-bot bypass, and CAPTCHA solving.
- Design architecture for the control-plane, covering APIs, authorization, billing, quotas, and monitoring.
- Design infrastructure using microservices, Docker/K8s, CI/CD, and observability tools.
- Lead development: participate in hiring backend, browser, DevOps, and QA engineers; establish development processes; oversee code quality.
- Conduct anti-bot research and bypass, handling browser and network-level fingerprinting and bypassing protections like Cloudflare and Akamai.
- Manage DevOps/SRE: set up CI/CD and Helm charts, configure scaling, and organize monitoring with Prometheus, Grafana, and Loki.
What We're Looking For
- 5+ years of engineering experience in high-load backend or data platforms.
- Extensive, hands-on web scraping experience.
- Excellent understanding of HTTP, TLS, HTTP/2, proxies, and the browser stack.
- Experience with async Python and Go, Redis, queues, and message brokers.
- Advanced level expertise with DevOps tools: Kubernetes, Helm, Prometheus, and CI/CD.
- Fluent English for documentation and team communication.
- Strong decision-making skills, mentoring abilities, and accountability for results.
Nice to Have
- Experience analyzing and bypassing bot protection systems.
Technical Stack
- Python, Go, Redis
- Kubernetes, Helm, Prometheus, Grafana, Loki, Docker
Team & Environment
You will lead the web scraping development team.
Benefits & Compensation
- Participate in developing the company's key product from scratch.
- High autonomy and influence on architecture and team decisions.
- Budget for conferences, courses, and work tools.
- Competitive compensation package.
- Remote work flexibility.
Work Mode
This is a global, remote position.
apply is an equal opportunity employer.
