Responsibilities
- Design and manage reliable data workflows using Azure Databricks, Confluent, Delta Live Tables, Spark, and Delta Lake for trading and market data processing.
- Independently analyze current data pipelines and implement enhancements to improve scalability, continuity, and performance.
- Provide operational support for production pipelines, including monitoring jobs, resolving incidents, and optimizing system performance.
- Manage Databricks environments, including cluster setup, job orchestration, access policies, and Unity Catalog configuration.
- Develop and maintain CI/CD workflows in GitLab to automate testing, deployment, and version control of data engineering code.
- Enforce coding standards such as modular architecture, peer reviews, and thorough documentation through GitLab processes.
- Work closely with team members, analysts, and architects to define data needs and implement effective solutions.
- Create reusable tools and frameworks to streamline development and maintain consistency across platforms.
- Engage in Agile rituals like sprint planning, daily stand-ups, and retrospectives to support team efficiency and process improvement.
Work Arrangement
Remote (Worldwide)
Other
The company is committed to equal opportunity employment and actively welcomes applications from qualified individuals of all backgrounds, including differences in gender, age, religion, disability, sexual orientation, gender identity, marital status, race, color, nationality, and ethnic origin.
