Data Engineer
London
Remote(1 day a week might be required in the future)
£600 p/d
Key requirements are a refreshed front-end, an API which can support data filtering and aggregation and a highly-performance approach to surfacing the data. This scope does not look at replacing the interpretation steps.
Essential skills and experience
Data Architect / Data Engineer - understand the different types of data that need to be brought into the system storage and access , work with the API team as to how it needs to be accessed and how to make the data available. Develop the Data Model, Design out performance related issues.
Desirable skills and experience
Current AWS certification
Deep understanding and experience in data processing, streaming and querying tools on AWS
Deep understanding and experience in developing solutions using AWS services such as Lamdba, Glue, SQS, SNS, Redshift, etc.
Develop object-oriented code using Python, besides PySpark, SQL and other languages
Experienced in implementing Redshift on AWS
Experience building pipelines and orchestration of workflows in an enterprise environment
Experience on Streaming technologies both OnPrem/Cloud
Experience implementing batch processing using AWS, Lambda & Data Pipeline
Experience in optimizing the cost for service being utilized
Experience in EMR/MSK/ElasticSearch/Neptune
Perform ongoing monitoring, automation and refinement of data engineering solutions
Experience working with an on-shore / off-shore model
