Job title: Data Engineer

Location: UK

Duration: 6-month contract Inside IR35

Working Mode: Fully Remote

Vetting: Active SC Clearance
:

Data Engineer Skills and Technologies Data Engineering:

• Strong understanding of data concepts
• data types, data structures, schemas (both JSON and Spark), schema management etc
• Strong understanding of complex JSON manipulation - Experience working with Data Pipelines using a custom Python/PySpark frameworks
• Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
• Strong understanding of Data Security principles
• data owners, access controls
• row and column level, GDPR etc including experience of handling sensitive datasets
• Strong problem solving and analytical skills, particularly able to demonstrate these intuitively

Requirements

Required Skills:

Languages / Frameworks

• JSON
• YAML
• Python (as a programming language, not just able to write basic scripts. Pydantic experience would be a bonus.)
• SQL
• PySpark
• Delta Lake
• Bash (both CLI usage and scripting)
• Git
• Markdown
• Scala (bonus, not compulsory)
• Azure SQL Server as a HIVE Metastore (bonus) Technologies
• Azure Databricks
• Apache Spark
• Delta Tables
• Data processing with Python
• PowerBI (Integration / Data Ingestion)
• JIRA