Data Engineer (Python / GCP / PySpark / Terraform)

  • Location

    London, England

  • Sector:

    Data

  • Job type:

    Contract

  • Salary:

    £570 - £620 per day + negotiable

  • Contact:

    Martin Kettlewell

  • Contact email:

    martin.kettlewell@xcede.co.uk

  • Job ref:

    HQ00023379_1567501879

  • Published:

    3 months ago

  • Duration:

    6 months

  • Expiry date:

    2019-11-02

  • Startdate:

    ASAP

  • Consultant:

    #

Data Engineer (Python / GCP / Spark / Terraform / Kubernetes) required by my client, an eCommerce brand based in central London

This is a 6 month contract paying up to £620/day and is highly likely to extend

As this global eCommerce brand look to move from AWS over to GCP they need a Data Engineer with strong python coding skills and experience in both AWS and GCP

Day to day jobs and experience expected
- Develop the backend data systems on AWS. Designed to support seamless integration, deployed across 20 countries, integrating thousands of partner brands using spark and python for processing and parquet with Athena/Presto as a query engine
- Developed the architecture for a distributed crawler operating using server-less AWS lambda functions along with SQS and SNS that execute concurrently across all countries and retailers.
- Work extensively with the search infrastructure based on Elastic Search to improve its indexing and query performance, scalability and relevance of results
- Assisted with the migration from on-premise infrastructure to GCP using Big Query, Dataflow and Dataproc
- Introduce a Data Warehouse for a product database to lay the data in dimension and fact tables that allowed the development of traditional BI analysis on the pricing data

Tech stack
Python
GCP cloud technologies and AWS
Must have Terraform experience
Hadoop
Spark

This is an opportunity to join one o the fastest growing brands in the country, work with some seriously talented people on vastly interesting projects.

If you would to understand more please get in touch