Role OverviewWe are looking for an ETL / Data Pipeline Engineer to join our data infrastructure team. This role operates on an overnight schedule aligned with US Central Time, providing critical coverage for automated scraping and ingestion systems that run during US business hours and overnight windows. You will help manage and monitor our existing fleet of web scrapers, build new data collection pipelines for regulatory and energy market sources, and contribute to the broader evolution of our ETL architecture.This is a hands-on engineering role. You will work closely with our Houston-based data and analytics teams to ensure data flows reliably from source to production databases, and that our clients receive accurate, timely intelligence every morning.Key ResponsibilitiesOvernight Operations & Data QualityMonitor and manage overnight scraper and ingestion runs, triaging failures and applying fixes in real time to minimize data gaps before US market openVerify data completeness and quality across all automated feeds, flagging anomalies and coordinating with the Houston team on persistent issuesMaintain run logs, error documentation, and escalation notes for seamless async handoffsNew & Expanded Data CollectionBuild and maintain scrapers, parsers, and ingestion pipelines across a growing set of energy market data domains, including but not limited to:Pipeline operator portals, electronic bulletin boards, and related filings (notices, maintenance, capacity, gas quality, customer indices)Government and regulatory agency databases at the federal, state, and provincial level across North AmericaInternational energy data sources covering European, Canadian, and Mexican supply, demand, and power marketsEmissions and environmental reporting systemsFinancial and corporate filings, including public company disclosures and production reportingGeospatial and mapping data related to production, infrastructure, and market geographyETL Architecture & EnhancementContribute to the design and build-out of our broader ETL infrastructure, including scheduling, orchestration, and error handlingWrite transformation logic to clean, normalize, and load raw data into PostgreSQL staging and production tablesOptimize existing pipelines for performance, reliability, and cost efficiencyHelp build monitoring dashboards and alerting for pipeline health and data freshnessDocument data lineage, schema changes, and pipeline dependenciesTechnical EnvironmentLanguages: Python (primary), SQL, Bash scriptingDatabase: PostgreSQL, Mongo and SnowflakeInfrastructure: AWS (EC2, S3, Lambda), DockerScraping: Selenium, Playwright, BeautifulSoup, Scrapy, or similar frameworksOrchestration: cron, Airflow, or equivalent scheduling toolsVersion Control: Git / GitHubCommunication: Slack, with async handoffs to US-based teamRequired Qualifications3+ years of experience building and maintaining ETL pipelines or data engineering systemsStrong Python skills with experience in web scraping, data parsing, and automationProficiency in SQL and experience working with relational databases (PostgreSQL preferred)Experience with headless browsers, anti-bot mitigation, and scraping resilience patternsComfortable working independently during overnight hours with minimal real-time supervisionStrong debugging instincts and ability to triage pipeline failures quicklyClear written communication in English for async collaboration and documentationPreferred QualificationsExperience with energy, commodities, or financial data pipelinesFamiliarity with FERC-regulated pipeline data, EIA reporting, or utility/regulatory filingsExperience with PostgreSQL-specific features (partitioning, materialized views, JSONB, pg_cron, logical replication)Familiarity with infrastructure-as-code (Terraform, CloudFormation) or containerized deploymentsPrior experience working on a distributed team across time zonesWhat We OfferCompetitive compensation and benefits with room to grow as the team scalesDirect impact on the data infrastructure behind a leading natural gas intelligence platformA small, senior team where your work is visible and valued from day oneFlexible remote work with clear async workflowsExposure to the North American energy markets and commodity data at scale
5 months ago