• Own the Model Feedback Lifecycle: Design and manage processes for collecting, triaging, and analyzing feedback from production usage. Build dashboards and reporting systems to surface insights and drive data-informed improvements to Copilot, addressing customer needs with high quality and efficiency. Identify and mitigate project risks to ensure successful outcomes.
• Cross-Team Coordination: Partner with Research, Engineering, Data, and Product teams to align on goals, identify opportunities for enhancing model performance, share learnings, and drive execution at scale.
• Vendor Management: Oversee external vendors for data labeling and human evaluation programs.
• Quality Frameworks: Build and run systems to assess and enhance model responses, such as annotation platforms and evaluation pipelines. Review performance metrics to find areas for improvement.
• Data-Driven Decision Making: Collaborate with analytics teams to define success metrics and correlate improvements to user impact.
• Operational Excellence: Manage all components of project planning, execution, and delivery by developing consistent processes that accommodate product complexity and organizational growth, ensuring coordination with overall product strategy and deadlines.
• Strategic Program Architecture: Partner with other TPgMs and leadership to design scalable processes and frameworks that extend beyond model response quality, supporting evolving priorities across Copilot. Maintain a deep understanding of the latest advancements in AI and Machine Learning.

• Bachelor's Degree AND 4+ years of experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience.
• 2+ year(s) of experience managing cross-functional and/or cross-team projects.
• 2+ year(s) of experience leading AI-focused products or programs with direct involvement in model evaluation strategies (Evals), optimization through loss function analysis (Losses), or overseeing data collection pipelines to support scalable training and validation workflows (Data Collection).
• 2+ year(s) of experience reading and/or writing code (e.g., sample documentation, product demos).

• Bachelor's Degree AND 8+ years of experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience.
• 6+ years of experience managing cross-functional and/or cross-team projects.
• Prior exposure to Large Language Models (LLMs), model evaluation, or human-in-the-loop systems.
• Experience working in a startup-like environment.

Technical Program Management IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $158,400 - $258,000 per year.

Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay

This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.