What AI systems need from product managers

Product managers have been handed responsibility for AI features at a rate that has outpaced the development of frameworks for thinking about them. A PM who has spent years shipping traditional software features has well-developed instincts for writing requirements, defining success metrics, and managing trade-offs. Most of those instincts transfer to AI features imperfectly or not at all.

This is not a skill gap in the traditional sense. The PM is not missing knowledge they could acquire through a training course. The underlying nature of AI systems is different from traditional software in ways that require different mental models, not just new information.

Why traditional requirements do not work for AI features

A traditional software requirement specifies behavior: given this input, produce this output. The system either produces the output or it does not. The requirement is falsifiable and the implementation either satisfies it or it fails to.

AI feature requirements written in this style produce requirements that are technically satisfied but practically useless. “The AI assistant should provide helpful responses to user questions” is a requirement that is satisfied by any response, because whether a response is helpful is not deterministic. “The AI summarizer should produce accurate summaries” is satisfied by every output the summarizer produces, because accuracy is not defined. The requirement exists on paper and provides no engineering guidance.

Effective AI feature requirements have a different structure. They specify the quality criteria that define acceptable output, the failure modes that are unacceptable, the evaluation method that distinguishes acceptable from unacceptable, and the acceptable rate of failure for each failure mode. This is harder to write than a traditional requirement and requires the PM to think carefully about questions that traditional product management does not force.

What does “good” output look like, specifically? What does “bad” output look like, and what are the consequences when it occurs? How will we know, after launch, whether the feature is producing good output? What failure rate is acceptable for each type of failure, and why?

These questions do not have answers that engineering can supply. They require product judgment about what the feature is for, who it affects, and what happens when it fails. PMs who do not engage with these questions leave the engineering team to answer them through implementation choices that are unlikely to reflect product intent.

The success metric problem

The most common PM mistake on AI features is measuring success with engagement metrics. Users are using the feature: success. Users are not using the feature: failure. This metric works for traditional software features where engagement and value are correlated. It does not work for AI features where users may engage with a feature that is producing bad output, or may disengage from a feature that is producing good output for reasons unrelated to quality.

An AI feature that generates confident but incorrect responses will see high initial engagement as users try it out, followed by declining engagement and trust as they discover the outputs are unreliable. Measuring engagement in the early period shows success. Measuring retention shows the problem later. Without quality metrics, the PM cannot distinguish between a feature that users adopted and then found valuable versus one they adopted and then found unreliable.

AI feature success metrics require a quality layer that traditional metrics do not. The PM needs to define what quality means for this specific feature, how it will be measured (automated evaluation, human sampling, user signals), and at what frequency. The quality metric should be in the launch criteria, not a follow-up investment.

The output quality metrics sit underneath the business metrics. Business metrics (did this feature increase retention? did it reduce support volume? did it improve conversion?) are the ultimate arbiter of success. But business metrics move slowly, are influenced by many factors, and do not tell you why performance is what it is. Quality metrics provide faster feedback and more direct attribution.

The scope problem specific to AI

Traditional software features can be specified completely: the set of inputs the feature accepts, the set of outputs it produces, and the behavior for each input-output pair. This is not possible for AI features because the input space is open-ended. An AI assistant can receive any natural language input. An AI classification system can receive documents with arbitrary content.

This creates a scope problem that PMs often underestimate. When you cannot enumerate the input space, you cannot specify behavior completely. Some inputs will produce outputs you did not anticipate. Some of those outputs will be bad in ways you did not plan for. The question is not whether this will happen, but whether you have thought carefully enough about the input distribution to anticipate the most important failure modes.

The PM’s job here is not to enumerate every possible input. It is to characterize the expected input distribution and identify the subsets of inputs where failure would be most harmful. Users asking about sensitive topics. Inputs in languages the system was not optimized for. Requests that fall outside the intended scope of the feature. Adversarial inputs designed to produce bad outputs. Each of these categories needs a defined response, which requires a product decision, not an engineering decision.

What should the feature do when a user asks something outside its intended scope? What should it do when the input is ambiguous? What should it do when it does not know the answer? These are product questions. The answers belong in the requirements, not in the engineering team’s judgment at implementation time.

Communicating with engineering about AI features

The communication patterns that work for traditional features do not transfer well to AI features. “Make it smarter” is not actionable. “Make it more accurate” is not actionable. “Make it less likely to hallucinate” is not actionable. These requirements provide no guidance on what to measure, what to optimize for, or what trade-offs to accept.

Actionable requirements for AI features are specific about quality dimensions and acceptable ranges. “Reduce the rate at which the feature produces factually incorrect statements in the [specific domain] from the current baseline to below X%” is actionable. “Ensure the feature declines to answer questions outside [defined scope] rather than attempting to answer them” is actionable. “Improve structured output compliance from Y% to Z% as measured by [defined schema validation]” is actionable.

This level of specificity requires the PM to have invested in understanding the current system’s performance and what “better” means in measurable terms. It is more work than writing abstract requirements, and it is the work that makes AI feature development go well rather than producing a series of iterations that improve subjectively without measurable progress.

PMs who develop this skill are significantly more effective at AI feature development than those who do not, because they give engineering teams the information needed to build the right thing and measure whether it is working.

The PM’s role in evaluation

Evaluation infrastructure for AI features is often treated as an engineering concern. The PM specifies the feature; engineering builds it and tests it. This division of responsibility breaks down for AI features because evaluation requires product judgment about what good means.

The PM should own the definition of the evaluation criteria: what the feature is supposed to do, what success looks like for specific input categories, and what failure modes are unacceptable. Engineering owns the implementation of the evaluation infrastructure: the test sets, the automated scoring, the monitoring dashboards.

Without PM ownership of evaluation criteria, evaluation infrastructure tends to measure what is easy to measure rather than what is important. Structural output compliance is easy to measure automatically. Factual accuracy is harder. User value is harder still. If no one with product judgment is driving the evaluation design, the evaluation optimizes for what can be automated, which is often not what determines whether users find the feature valuable.

PMs who are actively involved in defining evaluation criteria produce AI features that improve in the right direction. PMs who leave evaluation to engineering produce AI features that get better on metrics that do not correlate with user value.

The honest version

AI features are harder to product-manage than traditional software features. The requirements are harder to write, the success metrics are harder to define, the scope is harder to bound, and the evaluation is harder to design. These difficulties are not going to decrease as AI capabilities improve; if anything, they will increase as AI features take on more complex tasks.

The PMs who navigate this well are not the ones who have studied AI the most. They are the ones who have internalized that AI features require more specificity about quality, more investment in evaluation, and more ongoing attention to whether the feature is continuing to work than traditional software features. The frameworks they already have are necessary but not sufficient. The additional investment is in defining what good means, measuring whether they have it, and communicating that clearly enough that engineering can build toward it.

That is not a different job. It is the same job, done more carefully, in a domain where imprecision has more consequences than usual.