Senior Data Engineer

  • Location

    London, England

  • Sector:

    Data

  • Job type:

    Contract

  • Salary:

    £600.00 - £675.00 per day

  • Contact:

    James Jarvis

  • Contact email:

    james.jarvis@xcede.co.uk

  • Job ref:

    HQ00030883_1590573711

  • Published:

    about 1 month ago

  • Duration:

    6 Months

  • Expiry date:

    2020-06-26

  • Startdate:

    ASAP

  • Consultant:

    #

PYTHON / SCALA / AWS / ETL / SPARK

Data Engineer

This is a key role for the development and operations of the Finance Data Solution to drive maximum value from data for Finance.

You will have the opportunity to work on complex problems, implementing high performance solutions that will run on top of the cloud based big data platform.

Accountabilities

*Work as part of the Data Engineering team to uphold and develop common standards and best practices, collaborate to ensure that the data solutions are complimentary and not duplicative.
*Build and maintain robust automated pipelines to support data solutions across BI and analytics use cases.
*Interface with other technology teams to ingest, transform and load data from multiple data sources, structured and un-structured data.
*Work closely with data scientists and data analysts to implement and productionize data models.
*Create patterns, common ways of working, and standardised data pipelines to ensure consistency.

Experience

*Experience in engineering, optimising and debugging high performance data pipelines in terabyte to petabyte scale data platforms, and in contributing to the creation of data platforms.
*Experience with object oriented and functional design, coding, and testing patterns as well as experience in engineering software platforms and large-scale data infrastructures.
*Experience writing high quality and maintainable SQL on row oriented and column oriented datastores.
*Experience with NoSQL technologies, ideally DynamoDB.
*Experience developing in Python.
*Experience in a second language, ideally Scala
*Experience of delivering in Linux/Unix environments on-premises and in the cloud.
*Experience implementing reliable and robust data pipelines using Spark (batch and streaming).
*Demonstrated experience of working on various data integration tools like Talend, Pentaho, Informatica for both ETL as well as ELT processing.
*Solid understanding of Enterprise patterns and applying best practices when integrating various inputs and outputs together at scale.
*Knowledge of software best practices, like Test-Driven Development (TDD) and Continuous Integration (CI)
*Knowledge of data modelling and understanding of different data structures and their benefits and limitations under particular use cases, with good knowledge of Big Data.
*Understanding and experience of AWS cloud storage and computing platform (especially S3, Athena, Redshift, Glacier, EMR, EC2, RDS).
*Understanding of DevOps principles, tools, and the intersection with cloud architecture.
*Experience on an Agile Environment, familiarity with Jira, Confluence and Git.
*Experience on any one of BI tools such as Tableau, Qlik, Looker.
*Experience of Docker/Kubernetes would be beneficial.
*Knowledge of streaming data technologies such as Kafka, AWS Kinesis and AWS Lambda. Basic AWS platform experience