Accessibility Links

Big Data Engineer

  • Salary: £440 - £475 per annum
  • Job type: Contract
  • Location: Ipswich, Suffolk
  • Sector: Other
  • Date posted: 08/09/2017
  • Job reference: J373816A

We're really sorry, but it looks like this job has already been filled.

Register your CV with us, see our latest jobs or use the search below.

We're recruiting for a Big Data Engineer to join our world leading insurance client in Ipswich on an initial 3 month contract.

The core responsibility of the candidate will be to design, develop and automate data pipelines extending the data platforms to meet the needs and requirements of the Business stakeholders.

You will be the go-to person within the team for Python development, Spark, Hadoop ecosystem, Cloudera specific tools. You will be working alongside data scientists to develop, automate and support data pipelines on Cloudera Hadoop and Spark ecosystem implementations.

You must have good knowledge of:

  • Testing automation in a Big data environment and develop and support the MS BI ETL implementations in Data Services as well as DB design.
  • Good knowledge of testing in an MS BI ETL data environment.
  • Knowledge of coding and deployment practice and be the Champion in core Agile principles and practices; organise and facilitate daily meetings, retrospectives, design and planning sessions; promote team self-organisation through empowerment; reporting on time box/release progress and team metrics.

Essential knowledge:

  • Proficient in Python and SPARK development
  • Well experienced working on Hadoop environments with various suite of services [Oozie, HIVE, Impala, ELK, Kafka, Storm, Sqoop etc]
  • Experience working with various formats - XML, JSON, AVRO, PARQUET etc.
  • Strong experience in Data Ingestion, processing and performance tuning on Big Data platforms
  • Proficient in ETL and Data Warehousing using Microsoft BI Stack

Desirable knowledge:

  • Knowledge of Cloud solutions (Azure or AWS)
  • Knowledge of CI and CD.

Essential skills & abilities

  • Pyspark, Yarn, Shell Scripting
  • MS SQL Server 2008 r2/2012/2014 product stack
  • Situational Awareness: ability to identify risks/issues, mitigate them as they arise and understand how and when to escalate to senior management.
  • Data Modelling and solution design on different data platforms.

Interested candidates should submit their CVs in the first instance to harriet.valentine @

Similar jobs