Principal Databricks Data Engineer at Astra-North Infoteck Inc. — NeverHard
Principal Databricks Data Engineer at Astra-North Infoteck Inc. in Toronto, Ontario. Skills: Analytics, Apache Spark, BI, Cloud Object Storage, Data Engineering. Apply on NeverHard.
Company
Astra-North Infoteck Inc.
Location
Toronto, Ontario
Type
full_time
Required skills:
Analytics
Apache Spark
BI
Cloud Object Storage
Data Engineering
Data Lineage
Data Pipelines
Data Quality Frameworks
Data Security
Databricks
12–18 years of overall Data Engineering experience
8+ years of experience with enterprise Data Warehouse and Data Lake platforms
5+ years of hands‑on experience with Databricks and Apache Spark at scale
Strong experience modernizing legacy Cloudera platforms including:
Modernize Cloudera platforms to Databricks Lakehouse architecture
Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
Ensure data reconciliation, audit integrity, and consistency during migration
Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
Curated / Conformed Layer
Semantic / Consumption Layer
Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
Strong experience with finance and risk data models including:
Financial Hierarchies
Liquidity Risk
Enable reporting capabilities including:
Build and manage semantic and consumption layers for BI, reporting, and analytics
Define business metrics, dimensions, hierarchies, and KPIs
Experience with:
Develop and optimize large‑scale data pipelines using:
Optimize workloads using:
Data Quality Frameworks
Establish data lineage and metadata management
Ensure data security, access control, and compliance standards
Experience with AWS or Azure cloud platforms
Experience with CI/CD pipelines using:
Familiarity with:
Experience with dbt is an advantage
Act as a technical authority and lead enterprise architecture decisions
Mentor senior engineers and establish engineering standards
Collaborate with finance, risk, analytics, and governance stakeholders
Translate complex data structures into business‑ready insights
Nice‑to‑Have Skills
Experience in BFSI, Capital Markets, or Regulatory Reporting
Oracle Financials
Experience supporting AI/ML workloads
Databricks and Cloud Certifications
Key Responsibilities
Lead Cloudera to Databricks transformation initiatives
Shape enterprise finance and risk data platforms
Support regulatory, management, and analytical reporting systems
#J-18808-Ljbffr