For companies:
For professionals:
For everyone:
Responsibilities
About the Role
We are looking for a senior engineer to join our lean team and take ownership of the core of our large-scale scraping engine, which powers over 1 billion requests each month. You will play a critical role in shaping our technology and addressing complex challenges related to browser automation, distributed systems, proxy infrastructure, and performance optimization.
This role is ideal for someone who thrives with autonomy, prefers building over meetings, and enjoys solving technical problems that emerge only at scale.
What This Role Is Not
Although we are a web scraping company, this is not a data extraction job. We are not seeking developers who write ad-hoc scrapers using requests or BeautifulSoup. Instead, this role focuses on building and scaling the infrastructure that supports scraping at massive scale. This includes working on Python internals, concurrency, distributed systems, and performance engineering, rather than writing and maintaining scripts to fetch pages or extract data.
What You’ll Do
* Design, build, and optimize our Python-based distributed scraping engine.
* Profile, debug, and resolve performance bottlenecks and production-only bugs.
* Develop new product features and ensure reliable performance under heavy load.
* Collaborate across backend, DevOps, and tooling as needed, adapting to different responsibilities when required.
* Improve code quality through testing, documentation, and refactoring.
* Share knowledge, mentor teammates, and contribute to strengthening our engineering culture.
* Participate in hiring and help grow the team.
What Makes You a Good Fit
* You prioritize engineering and prefer coding over meetings.
* You thrive in a small, fast-moving team where you own features end to end.
* You view failure as feedback and iterate quickly.
* You actively improve systems and push for reliability and performance at scale.
Requirements
* 5+ years of professional Python experience in production systems.
* Deep understanding of Python internals, including memory management, GIL, and asyncio.
* Strong expertise in concurrent programming, including multiprocessing, threading, and async I/O.
* Solid knowledge of HTTP and networking fundamentals.
* Comfortable working in Linux and Unix environments.
* Proven ability to debug complex, large-scale production issues with minimal guidance.
* Strong communication skills and the ability to write clear documentation.
Nice to Have
* Experience with Golang.
* Experience with large-scale web scraping in production environments.
* Familiarity with Kubernetes internals and container orchestration at scale.
Our Tech Stack
* Python 3.13+.
* Redis, MongoDB, MariaDB, RabbitMQ.
* Kubernetes, Helm.
* Linux (x86).
This role is ideal for engineers who want to own critical systems, influence technology decisions, and solve challenges at scale without unnecessary management layers or meetings.
Indefinitely open
tune NOT FOR YOU? IMPROVE YOUR RESULTS