Skip links

Head of Data Engineering

Head of Data Engineering

Job Category: Architecture AWS Azure Cloud Engineering Data Engineering Engineering
Job Type: Full Time
Job Location: Remote

You want to be part of something bigger than yourself, and you are entrepreneurial by nature.

You will play the lead solution architect and technical lead for our data engineering aspects of our core products and platforms. You will partner with our sales and delivery team to bring confidence to customers in enabling adoption of our offerings. You enjoy spending most of your time designing, developing, and problem solving distributed cloud data systems that process 1+ billion events a day.

You have a growth mindset and always want to learn and improve yourself, products and the solutions for customers.

Does this sound like you? Apply!

Mandate

  • Design and build a modern data platforms on the cloud
  • Enhance our core product offering with more advanced features to solve customer needs
  • Gather functional and non-functional requirements
  • Define a high-level component architecture aligned with prioritized use cases
  • Provide guidance and consultation (pros, cons, and recommendation) for technology stack selection with options and alternatives to meet the use case and requirements
  • Build a prototype to demonstrate the capability of the system as it meets the clients needs
  • Document a detailed operational and system architectures in the form of a specific and precise blueprint and specification document that can be implemented by engineering
  • Lead engineering sprints to implement the solution

You have a solid understanding and experience:

  • Modern data architectures, including Lakehouse and Meta-data driven architectures.
  • With Azure: Synapse Analytics, Spark/Databricks, SQL Warehouse (dedicated and serverless SQL pools), Event Hub, Azure Data Lake, Azure Data Factory, Azure Synapse
  • With Data: Lakes, Warehouses, Lakehouse architecture models
  • Understand the difference between leading data products like Azure services, Snowflake, Databricks / Spark, Cloudera / Hadoop, Kafka and Storm
  • Experience with GCP, AWS, and Terraform is a plus
  • With Master Data Management (MDM), designing data models and data dictionary
  • Designing for hot and cold data paths for streaming or batch data
  • Designing ETL (and ELT) processes, configuration and rules that perform at scale
  • Designing for data quality, data integrity, and data optimization
  • Applying Apache Parquet and Delta Lake technology
  • Database development and operations with MySQL, PostgreSQL or MSSQL
  • Designing data systems that scale from 1 to 10 billion messages per day and factoring in optimized storage models, schema design, sharding and partitioning configurations
  • Programming in Python, SQL, Java and/or Javascript
  • Integrating with BI tools
  • Migrating data from legacy or current systems to the new solution
  • Cloud automation using infrastructure as code such as ARM templates and Terraform
  • Experience developing software through DevOps pipelines for release engineering and management using Azure DevOps and/or Github.

Responsibilities

  • Provide technical expertise in solution architecture and technical design decisions.
  • Design, build and engineer scalable, reliable, and secure systems using cloud and open-source technologies.
  • Identify and assess customer business and technical requirements, define scope and provide estimates for solutions.
  • Validate proposed project scoping with internal stakeholders including project managers, technical leads, designers, and other project team stakeholders.
  • Ability to communicate architectural decisions to a varied audience including business and technology stakeholders from the conceptual to the physical perspectives
  • Ability to recognize and balance functional and non-functional requirements to ensure the optimal solution for the end customer
  • Produce detailed architecture documentation, blueprints, specifications, configuration guides, build / cook books, operations manuals / runbooks, and user manuals.
  • Apply TOGAF (or similar architecture methods) to design, develop, and support recommended solution architectures that align with customer use cases and/or user requirements.
  • Engage and leverage technology partners into proposed solutions to demonstrate capabilities and strengthen relationships.
  • Build prototypes and demos aligned to the project/customer use cases to build credibility and confidence in the proposed solution. Lead sprint demos during the project.
  • Build cloud-based solutions through infrastructure-as-code (IaC) and design for scale, performance, availability, portability, resiliency and other required non-functional requirements
  • Transition project requirements and solution artifacts to engineering for smooth implementation.
  • Support peer teammates with delivery of the proposed solutions.

How you work

  • Be professional and courteous
  • Act with honesty and integrity
  • Do not embellish and be pragmatic
  • Work in partnership with other members of the team, present a unified approach in making recommendations and executing the project for the client
  • Approach all projects with ruthless focus on the end-user and customer and you are outcome-focused in a manner that creates value for our clients.
  • You’re constantly learning and absorbing the rapid changes in industry
  • You’re a team player and demonstrate this through setting your team members up for success.
  • You’re pragmatic and understand that well-engineered solutions result from balancing competing concerns.

Qualifications

  • 8+ years of proven hands-on experience with the architecture, design, development, deployment and support of software projects.
  • Post-secondary education in Computer Science, Math or Engineering.
  • Strong understanding of object-oriented and functional paradigms and architectural design patterns.
  • Clear presentation and communication skills in customer interactions in a sales setting and externally at events.
  • Team-first mentality
  • Adaptability to changing project and technology requirements while being hands-on with project teams.
  • Demonstrated experience and applied knowledge of both Waterfall and Agile project methodologies.

Apply for this position

Allowed Type(s): .pdf, .doc, .docx