Responsibilities
- Design, develop, and support robust ELT and ETL data workflows that handle both real-time streaming and batch processing of large, complex data sets into Google BigQuery.
- Oversee the administration, performance monitoring, and version upgrades of the Apache Airflow and Cloud Composer environments.
- Develop well-structured, reusable Python-based DAGs, create custom operators and hooks, and enhance workflow execution efficiency.
- Ensure reliable operation and performance of core Google Cloud Platform data services including BigQuery, Cloud Storage, Cloud Functions/Run, Pub/Sub, and IAM configurations.
- Apply advanced optimization techniques in BigQuery such as partitioning, clustering, slot allocation management, and query cost reduction strategies.
- Build efficient Python-based utilities, API clients, and automation scripts for data retrieval, schema validation, and integration with external data sources.
- Implement comprehensive data quality controls, continuous monitoring, and automated alerting systems to prevent data loss and reduce processing delays.
- Manage data infrastructure using code-defined templates and support continuous integration and deployment processes through Git-based workflows.
- Participate in system design evaluations and guide less experienced team members in data warehouse design principles and engineering standards.
Work Arrangement
Hybrid
Other
- Applicant must be a U.S. Citizen as required by federal government contracting regulations.
- This organization operates as an equal opportunity and affirmative action employer.
- The company upholds a pay transparency policy and will not penalize employees for discussing or disclosing their compensation or that of coworkers under certain conditions.
- Accommodations will be made for qualified individuals with disabilities in compliance with relevant laws and regulations.
