Position Overview

We seek a VP Data Engineer to own critical data pipelines and establish architectural patterns for our Databricks-on-Azure data platform. This is an opportunity to architect scalable ETL/ELT patterns, establish best practices for Databricks/Spark/Delta Lake development, and design systems that handle both structured and unstructured data at scale.

You will work closely with the Principal Head of Data Engineering and VP Staff Data Engineer to build pipelines that power AI-ready infrastructure. Your code and patterns become the template for how the team builds data systems.

Key Responsibilities

Databricks-on-Azure Architecture & Optimization

• Design and build complex Spark SQL and Python-based ETL/ELT pipelines in Databricks that handle large-scale data transformations

• Master Delta Lake architecture: table design, partitioning strategies, file organization, Z-ordering for query optimization

• Optimize Databricks cluster configurations: choose between interactive, job, and serverless compute based on workload; tune executor memory, shuffle partitions, and parallelism

• Implement cost-efficient patterns: predictive pushdown, broadcast joins, caching strategies; right-size clusters and use spot instances for non-critical jobs

• Design data quality frameworks within Databricks: schema validation, null handling, duplicate detection, completeness checks

Azure Integration & Data Movement

• Design ADLS (Azure Data Lake Storage) layouts: bronze/silver/gold medallion architecture, folder structures, retention policies

• Optimize Azure data movement: leverage Delta Live Tables (DLT), Databricks SQL, and managed ingest patterns

• Design Databricks workspace integration: configure Azure AD authentication, scoped API tokens, cluster policies

• Implement data governance via Databricks Unity Catalog: manage catalogs, schemas, table ACLs, lineage tracking

Document & Unstructured Data Processing

• Design pipelines for document processing: ingest PDFs, Word docs, and other formats from blob storage into Databricks

• Implement text extraction pipelines: use Databricks-native libraries, Azure AI Services (Form Recognizer, Document Intelligence)

• Build structured extraction from unstructured data: extract financial tables, key metrics, and entities from deal documents and financial statements

• Build preprocessing pipelines for NLP and LLM consumption: tokenization, chunking, metadata extraction, quality scoring

• Manage document versioning and lineage: track source documents, extraction versions, and quality metrics

Core Pipeline Development

• Own end-to-end design and implementation of critical pipelines supporting investment teams, ops/finance, and client / IR teams

• Establish patterns for error handling, logging, and monitoring within Databricks jobs

• Implement idempotent pipeline design: support re-runs, backfills, and late-arriving data

• Design incremental data loading: leverage Delta Lake's merge operations, CDC patterns, and change tracking

• Partner on schema design and dimensional modeling of enterprise data sets

Mentorship & Standards

• Mentor junior and mid-level engineers through code review, pair programming, and design guidance

• Establish Databricks/Spark best practices: naming conventions, notebook organization, cluster policies, testing patterns

• Create reusable libraries and utilities: custom Spark functions, data quality frameworks, common transformations

• Own code quality; your code is the reference for how the team builds

• Document patterns and best practices; maintain internal confluences/knowledge base

Troubleshooting & Optimization

• Debug complex Spark issues: shuffle spills, out-of-memory errors, performance bottlenecks

• Optimize Databricks query performance: analyze execution plans, identify skew, apply optimization techniques

• Manage cluster costs and performance: monitor job execution, identify inefficiencies, recommend cluster right-sizing

• Lead postmortems and troubleshooting sessions for production issues

Required Qualifications

• 6-9 years of data engineering experience with 2+ years at senior level or equivalent complexity

• Expert-level proficiency in Databricks/Delta Lake: notebook development, SQL, Spark, performance tuning

• Advanced SQL expertise: complex joins, window functions, CTEs, query optimization

• Strong Python proficiency: PySpark, pandas, data validation libraries

• Proven experience building ETL/ELT pipelines at scale (100GB+ datasets, multi-source ingestion)

• Deep understanding of Delta Lake: transactions, ACID properties, schema evolution, merge operations

• Experience with Azure cloud services: ADLS, Azure SQL, Event Hubs, blob storage, Azure Key Vault

• Demonstrated experience with document and unstructured data processing

• Experience with data orchestration tools (Prefect, Airflow, Databricks Workflows) and building robust error handling

• Ability to mentor other engineers and lead by example

• Comfort with greenfield projects and establishing best practices from scratch

Strongly Preferred Qualifications

• Production experience with Databricks Unity Catalog and governance features

• Experience with Databricks SQL and serverless compute

• Hands-on experience with document extraction: PDFs, forms, OCR, Table extraction

• Familiarity with Azure AI Services: Form Recognizer, Document Intelligence, Cognitive Search

• Experience with NLP libraries (spaCy, NLTK) and text preprocessing at scale

• Experience in financial services or PE environments

• Familiarity with dbt for transformation orchestration

• Databricks certifications or demonstrated expertise

Key Competencies