Security represents the most critical priorities for our customers in a world awash in digital threats, regulatory scrutiny, and estate complexity. Microsoft Security aspires to make the world a safer place for all. We want to reshape security and empower every user, customer, and developer with a security cloud that protects them with end to end, simplified solutions. The Microsoft Security organization accelerates Microsoft’s mission and bold ambitions to ensure that our company and industry is securing digital technology platforms, devices, and clouds in our customers’ heterogeneous environments, as well as ensuring the security of our own internal estate. Our culture is centered on embracing a growth mindset, a theme of inspiring excellence, and encouraging teams and leaders to bring their best each day. In doing so, we create life-changing innovations that impact billions of lives around the world.

We are looking for passionate engineers experienced in building reliable, high‑throughput, cloud‑scale data systems. Our team owns the Microsoft Next Gen Privacy Unified Catalog, an evolving platform that unifies and enriches metadata from diverse systems to power privacy, governance, and responsible data intelligence across Microsoft. You will design core components of a modern distributed data platform that relies heavily on AI‑assisted development and AI‑powered early error detection to improve engineer velocity, data quality, and operational reliability.

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

We are looking for passionate engineers who excel at building reliable, high‑scale, cloud‑native data systems. Our team is responsible for the Microsoft Next Gen Privacy Unified Catalog, an evolving platform that unifies metadata from diverse systems to power privacy, governance, and responsible data intelligence across Microsoft. We build high‑throughput data pipelines, scalable distributed transformations, and enrichment workflows that combine signals from multiple services. This is a unique opportunity to contribute to foundational data infrastructure with deep engineering challenges in distributed systems, metadata processing, cloud architecture, and service integration.

Responsibilities include:

1) Design and implement large‑scale data pipelines that ingest, transform, and unify metadata across multiple systems, enabling advanced governance and analytics capabilities.

2) Build modular transformation and enrichment components, including integrations with internal microservices, ensuring reliability, resiliency, and performance at cloud scale.

3) Leverage AI‑powered tools for code generation, testing, and early‑error detection, improving reliability, accelerating development cycles, and reducing operational overhead.

4) Improve platform scalability, observability, and reliability through robust monitoring, automated recovery patterns, and engineering excellence practices.

5) Collaborate closely with partner engineering teams to shape data models, processing patterns, and APIs that support long‑term platform evolution and new privacy capabilities.

6) Drive engineering quality end‑to‑end, including testing, performance validation, CI/CD automation, and operational readiness for high‑scale production workloads.

7) Participate in on‑call rotations and incident response, contributing to root‑cause analysis and long‑term improvements in system resilience and availability.

8) Contribute to architectural discussions and technical strategy, helping define the next generation of Microsoft’s privacy and governance data platform.

1) Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 10–12 years’ experience in data engineering, distributed systems, software development, or data platform work

2) Strong experience with data or software engineering, with solid proficiency in Python, Java, Scala, or C# - 3+ years in SQL-based query languages (e.g., SQL, KQL)

3) 8+ years of experience with ETL and data cloud computing technologies, including Azure Data Lake, Azure Data Factory, Azure Synapse, Azure Logic Apps, Azure Functions, Azure Data Explorer, and Power BI or equivalent platforms​

4) Hands‑on experience with cloud‑scale ETL and data processing technologies, such as: Distributed compute engines (e.g., Spark / PySpark) Cloud data platforms and orchestration tools Metadata indexing, search systems, or governance frameworks Data ingestion, transformation, and operational analytics pipelines.

Join us to build the Microsoft Next Gen Privacy Unified Catalog, a data governance platform that operates at true planet scale.

You’ll work on distributed data systems that process massive, constantly evolving datasets across Microsoft, designing pipelines and enrichment frameworks that power privacy, compliance, and responsible AI. If you want to solve deep technical challenges, work with world‑class engineers, and help shape the future of data governance for one of the largest data ecosystems on the planet, this is the role for you.

Define and establish the core architecture for the Microsoft Next Gen Privacy Unified Catalog data pipeline, setting foundational patterns for -transformation, enrichment, AI‑assisted automation, and large‑scale metadata processing. - Provide technical leadership and mentor a growing data engineering team, guiding design decisions, elevating engineering standards, and helping early‑career engineers develop strong technical judgment. - Drive AI‑driven improvements across the pipeline, applying AI for code generation, intelligent validation, and early error detection to improve reliability, development velocity, and data quality.

This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.