You have opted out of job matches in .
To undo this, go to the 'Skills and Interests' section of your preferences.
Posted 7 months ago
Requirements and responsibilities
We are seeking a Mid / Senior Web Crawling + Backend Engineer to lead the development and optimization of large-scale data acquisition and processing systems. The candidate must have expertise in advanced web crawling techniques and proven experience managing high-volume unstructured data pipelines. This role is essential to our mission of identifying and preventing rental fraud through comprehensive data collection and analysis. The position is remote and requires daily collaboration with the development team.
Key Responsibilities
* Design, develop, and maintain advanced web crawling systems capable of processing millions of property listings and rental advertisements.
* Implement techniques such as JavaScript rendering, anti-bot detection evasion, and distributed crawling architectures.
* Build and optimize backend systems for processing, validating, and storing large volumes of unstructured data from multiple sources.
* Develop data validation models to ensure quality and integrity in the fraud detection pipeline.
* Architect scalable workflows for both real-time and batch data processing requirements.
* Implement monitoring, alerting, and recovery systems for critical data acquisition processes.
* Collaborate with cross-functional teams to translate business requirements into technical solutions.
* Optimize system performance and costs while ensuring high availability and reliability.
* Stay updated with emerging web scraping technologies, anti-detection methods, and best practices in data processing.
Key Requirements
* 4+ years of experience in backend development with focus on web crawling and data processing.
* Expertise in web scraping frameworks and libraries (Scrapy, Selenium, Playwright, BeautifulSoup, or similar).
* Strong knowledge of web technologies including HTML parsing, JavaScript execution, session management, and proxy rotation.
* Experience with anti-detection techniques, CAPTCHA solving, and handling dynamic content.
* Proficiency in backend technologies such as Python, Node.js, or Java for data-intensive applications.
* Experience working with unstructured data formats (JSON, XML, HTML) and data transformation pipelines.
* Knowledge of data validation, cleaning, and quality assurance methodologies.
* Experience with database design and management for high-volume data storage (MongoDB, PostgreSQL, or similar).
* Ability to build and deploy systems processing millions of records efficiently.
* Problem-solving skills and experience debugging distributed systems.
* Fluency in English communication.
* Ability to work effectively in a remote team environment.
Optionally, you can add more information later (benefits, pre-screening questions, etc.)