Data Engineering Technical Lead, Hybrid working 1 day office, Big data, SQL, Apache Airflow
Salary: £80,000 to £90,000
My leading client is looking for a Data Engineering Technical lead, who is responsible for technical design of data solutions and platforms within the IT estate, adhering to enterprise standards and best practice.
The role will involve working closely with the enterprise architects, technical architects, solution architects, business analysts, engineers, and project managers to deliver IT change that meets the needs of projects and programmes within the enterprise portfolio.
The technical Skills below:
- Solid understanding of data concepts and data modelling principles; applied across traditional warehousing, OLTP and also modern data lakes.
- 5+ years of solid experience in designing and building solutions on a big data distributed system - data lake, data warehouse, or lakehouse.
- Strong software development background, with previous hands-on experience of applying modern development and release engineering practices (TDD, BDD, CICD etc.).
- Proficient in Apache Spark or any other distributed data programming frameworks (e.g. Flink, Arrow, MapR).
- Strong SQL background - comfortable writing efficient SQL (Transact-SQL, Hive -HQL) to meet the requirement.
- Commercial experience of building and architecting scalable Extract Load Transform (ELT) data pipeline implementations at varying frequencies - batch, near-real-time and real-time, from a myriad of source types - relational, non-relational, API, streaming (e.g. Kafka topic)
- Experience with Apache Airflow/Spring DataFlow, Control-M, or a similar orchestration tool in a Production setting.
- Exposure to working with large datasets (transforming, cleansing, loading) on a distributed computing environment.
- Experience working with low-latency data requirements where the speed of decision making dictates the speed of data delivery to the consumer.
- Strong understanding of Data Management principles (security and data privacy) and how they can be applied to Data Engineering processes/solutions, e.g. access management, data privacy, handling of sensitive data (e.g. GDPR)
- Experience working with a combination of cloud-based big data technologies (HDFS, Blob Storage, Spark, Kafka, Delta, Hive, Oozie, NiFi).
If this opportunity sounds like it could be of interest, please send me your CV ASAP.
