Responsibilities
- Partner with cross-functional teams to interpret technical data concepts for non-technical stakeholders and ensure data initiatives support business objectives.
- Collaborate within an agile delivery team on innovative data projects that convert raw data into strategic insights driving organisational change.
- Leverage cloud data platforms including Microsoft Fabric, Azure Data Factory, Azure Synapse, Databricks, and Power BI to build end-to-end, scalable data systems.
- Design and deploy efficient ETL and ELT workflows to integrate and transform data from diverse sources into reliable, high-quality datasets.
- Create and maintain advanced data models using dimensional modelling techniques to enable effective analytics and reporting.
- Enforce data governance, security, and compliance standards using tools such as Azure Purview, Unity Catalog, and Apache Atlas.
- Maintain data integrity through rigorous quality assurance and continuous optimisation of data pipelines and queries.
- Build intuitive and interactive Power BI dashboards that deliver actionable insights to business stakeholders.
- Monitor system performance and implement tuning strategies to improve speed, reliability, and functionality of data solutions.
- Stay current with evolving data engineering trends and integrate emerging tools, languages, and methodologies into daily practice.
Requirements
- Advanced proficiency in SQL and Python for solving complex data challenges.
- Solid understanding of Data Lakehouse architecture principles and implementation.
- Proven experience working with Spark-based platforms such as Azure Synapse, Databricks, or on-premise Spark clusters using PySpark or Spark SQL.
- Demonstrated ability to design and implement scalable ETL and ELT pipelines, primarily in Azure environments using Azure Data Factory and Spark.
- Skilled in writing high-performance, cost-efficient queries for optimal data retrieval.
- Experience developing insightful and interactive Power BI dashboards to support business decision-making.
- Strong capability in dimensional modelling to build effective, business-aligned data structures.
- Familiarity with CI/CD practices, including automated testing, builds, and deployments in data engineering workflows.
- Ability to interpret business requirements and convert them into clear technical specifications.
- Strong communication skills with a focus on simplifying technical concepts for non-technical audiences.
- Commitment to ongoing learning and development in data engineering technologies and best practices.
Nice to Have
- Familiarity with Microsoft Fabric and its core capabilities.
- Experience managing large-scale, high-performance data systems handling over one billion records or terabyte-level databases.
- Knowledge of Delta Tables or Apache Iceberg for efficient large-scale data lake management.
- Experience using data governance tools such as Azure Purview, Unity Catalog, or Apache Atlas.
- Exposure to real-time data technologies including Kafka, Azure Event Hub, and Spark Streaming.
- Understanding of SOLID principles in object-oriented programming.
- Familiarity with agile methodologies such as SCRUM to support iterative development.
- Awareness of emerging technologies like GenAI, RAG, and Microsoft Copilot, along with relevant cloud or data science certifications.
- Experience preparing and managing data for data science, AI, and machine learning applications.
- Background working with public sector clients and understanding of their unique requirements.
Compensation
Not specified
Work Arrangement
Hybrid
Work Arrangement
Hybrid working available in London, Sheffield, and Bristol.
Other
- This role requires Security Clearance.
- Candidates must complete a Baseline Personnel Security Standard during onboarding.
- Evidence required for security clearance is outlined on Gov.UK.
- Employment may be delayed or denied if the candidate does not meet clearance requirements.
- Role involves travel to client sites and company offices.
- This is a permanent position.
Not specified
