Senior Data Engineer

Job Profile

Job Description

What we are looking for:

• You’ll be able to help understand business requirements, and translate those into technical requirements, design patterns and stories for development.
• You’ll be ready to lead Engineering teams if required, to guarantee successful outcomes for our clients. Providing experience and guidance to less knowledgeable Engineers, passing on your knowledge and experience.
• You will have the opportunity to explore new features and functionality across the various vendor products that we partner with, as we have a deep relationship with most of these vendors and work on their product pipelines.
• You will also have the opportunity as we grow to potentially manage larger teams, and work on R&D projects for “product” based work, as we look to develop SAAS products and also our own enabling technologies.
• You will work with various stakeholders from teams across clients we work at, including engineering, architecture, Senior Manager and CxO level personnel.
• You will work with External parties that work with our client, such as our partners, or other consultancies that may be working with the client.
• Above all, you are a technical expert who will be contribute to excellent quality and value for our clients. Our Senior Data Engineers are a trusted source of information and advice. You will use your experience in different situations and with different products, to assess the needs of the business that are appropriate for them.
• Delivering informed opinions and approaches should, at all times be backed-up by your provable known knowledge and researched where at all possible to ensure that the advice can be acted on.

Candidate Specification

• Collaborate with our clients to design and build modern data platforms using a variety of technologies
• Help lead the design and implementation of complex, cloud-based data ingestion and transformation pipelines
• Implement scalable and secure Data Lakes / Warehouses
• Mentor and upskill other engineers, both client and internal
• Help drive effective development patterns and delivery practices
• Help maintain and improve our internal tools and design patterns
• Continually improve with our internal development program, including mentoring and paid training / certifications

Abilities:

The successful candidate must be a team player and effective communicator, confident in their capability, whilst able to recognise areas for development. They should have a natural ability in developing others, nurturing best practice and consciousness in everyone. They must also be able to think creatively and should always be seeking new and more efficient ways to improve the solution and associated development processes.

Skills & Experience

Skills and experience required:

• Highly proficient with Python
• Extensive experience developing on AWS, including a broad understanding of the service offerings for that provider
• Excellent knowledge of software development best practices
• Experience building CICD pipelines using Jenkins/li>
• Experience using Infrastructure-as-code tools (e.g. Terraform, Cloudformation)
• Advanced SQL skills, including query optimisation
• Hands-on experience working with Snowflake, including the creation of SnowSQL scripts
• Extensive experience with structured, unstructured, and semi-structured data
• Experience working within an Agile environment (SAFe preferred), as well as a strong understanding of Agile principles and delivery practices
• Working knowledge of data regulations (e.g. GDPR)
• Very strong communication skills, including stakeholder management up to CXO level

Preferably, experience working with the following:

• Containers (Docker preferred)
• Real-time / event-based data
• Big data products
• Data quality frameworks
• Orchestration tools (e.g. Airflow)
• Data modelling (e.g. Kimball, Inmon, Data Vault)