PySpark Developer (SC Cleared)

  • Location

    Newport, Wales

  • Sector:


  • Job type:


  • Salary:

    £440 - £540 per day

  • Contact:

    Harriet Hughes

  • Contact email:

  • Job ref:


  • Published:

    11 months ago

  • Expiry date:


  • Client:


PySpark Developer (SC Cleared), PySpark, Python, Spark, Amazon Web Services, AWS, Cloudera, Hive, Impala SQL, ETL, Integration, Jenkins, GIT, Version Control, JIRA, Development Lifecycle, Data Management, SC Cleared, SC Clearance, Security Cleared, Security Clearance, Government, Newport.

Our client based in Newport are looking for PySpark Developers (SC cleared) to join an existing project team and to work with an established product backlog to delivery system capability to a BAU team (responsible for publication of new/improved trade and investment data).

With the UK due to leave the European Union in 2020, the focus on trade and investment statistics has never been higher. There is a need to develop and transition new and existing processing systems. Their platform is based on both PySpark by Cloudera and Amazon Web Services technology.

Key Skills/Experience:

  • Strong experience in development of relational database systems using the Cloudera stack (particularly PySpark, Hive and Impala SQL)
  • Ability to produce technical documentation to support development of production systems
  • Experience in the development of Extract Transform and Load (ETL) pipelines
  • Experience developing continuous integration solutions (particularly in Jenkins)
  • Experience in version control and software review using GIT
  • Proficient in use of Jira for work item management
  • Experience of working across the full development lifecycle
  • Awareness of good data security procedures and adherence to secure information management policies
  • Ability to work collaboratively with members in other technical roles, including Business Analysts and Methodologists
  • Experience of data management