Apply is hiring a Scraping Engineer to serve as Tech Lead for a new, high-load web scraping platform. You will drive architectural decisions and shape the future of this key product, built from scratch at a rapidly growing company.
What You'll Do
- Design architecture for the data plane, including a high-parallel crawler, session management, anti-bot bypass, and CAPTCHA solving.
- Design the control plane, covering API, authorization, billing, quotas, and monitoring.
- Design infrastructure using microservices, Docker/Kubernetes, CI/CD, and observability.
- Lead team building by participating in hiring backend, browser, DevOps, and QA engineers.
- Establish development and release processes and oversee code quality and technical decisions.
- Conduct anti-bot research and bypass, focusing on browser and network-level fingerprinting (JA3, TLS, CDP) for systems like Cloudflare, Akamai, and others.
- Handle DevOps/SRE tasks including CI/CD setup, Helm charts, scaling configuration, and monitoring organization with Prometheus, Grafana, Loki and SLA processes.
What We're Looking For
- 5+ years of engineering experience in high-load backend or data platforms.
- Extensive, hands-on web scraping experience.
- Excellent understanding of HTTP, TLS, HTTP/2, proxies, and the browser stack (CDP, WebRTC).
- Experience with async Python and Go, Redis, queues, and message brokers.
- Advanced level with DevOps tools: Kubernetes, Helm, Prometheus, and CI/CD systems.
- Fluent English for documentation and team communication.
- Strong decision-making skills, mentoring abilities, and accountability for results.
Nice to Have
- Experience analyzing and bypassing specific bot protection systems.
Technical Stack
- Languages: Python, Go
- Databases/Caching: Redis
- Infrastructure & Observability: Kubernetes, Helm, Prometheus, Grafana, Loki, Docker
Team & Environment
You will lead the web scraping development team, driving its technical direction and growth.
Benefits & Compensation
- Participate in developing the company's key product from the ground up.
- High autonomy and influence on architecture and team decisions.
- Budget for conferences, courses, and work tools.
- Competitive compensation package.
- Remote work flexibility.
Work Mode
This is a remote position.
