As a Data Pipeline Engineer, you will take ownership of the infrastructure that powers our core data products. You'll be responsible for designing, scaling, and securing the systems that move and store critical data across the organization. This role blends hands-on technical execution with strategic thinking to continuously improve platform reliability, performance, and cost efficiency.
Key Responsibilities
- Manage and optimize AWS-based data infrastructure including EC2, RDS, VPC, S3, and MWAA, using Terraform for infrastructure-as-code and ensuring secure, compliant configurations.
- Monitor, troubleshoot, and enhance the stability of Airflow-managed ETL/ELT pipelines that feed analytics and machine learning workloads in Databricks.
- Design and implement new data ingestion workflows from operational sources such as CRM, telephony, and transaction systems to support key business objectives.
- Administer PostgreSQL databases in RDS, handling replica management, network configuration, and performance tuning.
- Ensure infrastructure meets SOC-2 compliance standards through proper access controls, SSO integration, and secure networking practices including VPN management via Pritunl.
- Collaborate with data engineers and analysts to align infrastructure capabilities with data modeling and transformation needs.
- Lead strategic improvements to reduce complexity, streamline orchestration, and guide long-term platform evolution.
Qualifications
You bring deep experience in cloud data platforms and infrastructure automation. Proficiency with Databricks, Airflow (especially MWAA), and AWS services is essential. Strong scripting skills in Python, including PySpark, are required, along with hands-on management of RDS and VPC peering setups. Experience with Unity Catalog, security groups, and SSO configuration strengthens your application. You communicate clearly across technical domains and can influence design decisions at both team and system levels.
Work Environment
This is a remote position open to candidates in the U.S., offering flexibility and full support for distributed work. You’ll operate with significant autonomy while contributing to a technology-driven culture focused on transparency, efficiency, and customer impact.
