How to structure an AI team

The question of how to structure an AI team comes up at a predictable point in most organizations: after a few successful pilots, when leadership decides to move from exploration to execution. At that point, someone has to decide whether AI capability lives in a central team, is distributed across product teams, or some combination. The decision has compounding consequences that are hard to reverse.

There is no universally correct answer, but there are structures that create predictable failure modes and structures that tend to work. The right choice depends on your organization’s size, how mature your AI capability is, and what you are trying to build.

The three structural models

Centralized AI team. A single team owns all AI capability: model selection, infrastructure, APIs, tooling, and often the AI components of specific products. Product teams consume AI capabilities as services. The central team is the bottleneck for all AI work.

This model works when AI capability is scarce and expensive to build independently, when there are significant shared infrastructure costs that benefit from consolidation, and when the organization is small enough that the central team can serve all internal customers without excessive coordination overhead.

It fails when the central team becomes a bottleneck that slows product development. Product teams waiting for AI features from a shared service lose the ability to iterate quickly. The central team, serving multiple stakeholders with conflicting priorities, develops a ticket-queue relationship with product teams rather than a collaborative one. Ownership becomes unclear: the product team owns the user experience, but the central team owns the model that drives it, and neither side is fully accountable for outcomes.

Embedded AI engineers. AI engineers sit within product teams rather than a central function. Each team owns its AI capability end-to-end. There is no central AI team, or the central function is minimal and focused on shared infrastructure only.

This model works when your AI use cases are diverse enough that centralization creates more coordination overhead than it saves, when your product teams are sophisticated enough to make good model and architecture decisions independently, and when you want AI to be a first-class consideration in product development rather than a service consumed after the product is designed.

It fails when teams duplicate work, make inconsistent architecture decisions, or lack the depth to handle hard problems independently. An embedded AI engineer working alone on a product team does not have peers to review their approach, no escalation path for hard problems, and no visibility into what other teams are learning. The organization accumulates AI debt in parallel across teams that could have been avoided with shared infrastructure and shared learnings.

Platform plus embedded model. A central platform team owns shared infrastructure: model access, vector stores, evaluation tooling, cost tracking, common abstractions. Product teams own their use cases and embed AI engineers who build on the platform. The platform team is accountable to internal developer experience; product teams are accountable to user outcomes.

This is the model that tends to work best at medium to large scale. The platform team eliminates the worst forms of duplication (every team managing their own model access and cost controls) without creating the bottleneck of a centralized team that owns product-level AI decisions. Embedded engineers have peers in their product teams and access to platform expertise when they need it.

The failure mode of this model is when the platform team and embedded engineers are not well-coordinated. Embedded engineers bypass the platform to move faster. The platform team builds infrastructure that does not match what product teams actually need. The split responsibility for AI outcomes becomes a blame-shifting dynamic when something goes wrong.

What team structure cannot solve

Team structure is not the primary determinant of AI team success. Two factors matter more.

The quality of the evaluation infrastructure. AI teams that cannot measure whether their systems are working cannot improve them systematically. Evaluation infrastructure (golden datasets, automated testing pipelines, production monitoring) is what separates teams that improve through data from teams that improve through intuition. This infrastructure is expensive to build and easy to defer. The team structure that works is the one that treats evaluation as a first-class investment, regardless of where it sits organizationally.

Clarity on who is accountable for outcomes. AI systems span a supply chain: the model provider, the infrastructure, the prompt, the retrieval system, the product integration, the user interface. When something goes wrong, the failure can be anywhere in that chain. Teams that have unclear ownership across this supply chain spend their incident time on attribution rather than resolution. The team structure question is mostly a question of how to assign clear accountability across the supply chain in a way that minimizes gaps and overlaps.

Both of these factors can be solved within any of the three structural models. Neither is solved by the structure alone.

Sizing the team

The most common mistake in sizing an AI team is treating it like a software engineering team with AI skills added. AI engineering has different capacity characteristics.

A production AI system requires ongoing maintenance that traditional software does not. The model’s behavior changes as the provider updates it. The input distribution shifts as users explore edge cases. Evaluation datasets need to be expanded as failure modes are discovered. Prompt engineering requires iteration that is qualitatively different from code changes. An AI system that is “done” is not done in the same sense that a traditional software feature is done.

A rule of thumb that holds up in practice: budget two engineers for ongoing maintenance for every three engineers who built the system. If you ship a production AI feature built by three engineers, plan for two engineers to maintain and improve it. This does not mean the team needs to be five people forever, but it means the maintenance load is not negligible and should be explicitly staffed.

Under-resourcing maintenance is what produces AI systems that decay silently. The team ships the feature, moves to the next project, and no one is watching when the model updates introduce a regression, when the evaluation coverage turns out to be insufficient, or when the feature’s performance erodes as usage scales.

Who to hire first

The first hire for an AI team shapes the team’s capability for a long time, because that person defines what “good” looks like internally and sets the pattern for subsequent hires.

The temptation is to hire for research depth: someone with a strong background in machine learning who can reason carefully about model behavior. This is the right hire if your team’s primary challenge is model selection and training. It is not the right hire if your primary challenge is shipping systems that work reliably in production.

For most organizations at the early stages of building AI capability, the highest-value first hire is someone with production AI engineering experience: building systems, not training models. They should have dealt with evaluation infrastructure, model version management, cost controls, and incident response. Research depth is valuable, but the missing skill in most organizations is engineering discipline applied to AI systems, not more research capability.

A checklist for the structural decision

Before committing to a structure, answer these questions:

How many distinct AI use cases are you building for? One or two use cases suit a centralized model. Five or more suit distributed or platform-plus-embedded.

How fast do you need to iterate on each use case? If product teams need to move at product velocity, embedding is almost always better than central service.

Do you have the depth to make good AI architecture decisions at the team level? If the answer is no, a central team provides guardrails. If the answer is yes, distribution enables speed.

What is your shared infrastructure cost? If every team managing its own model access and observability is significantly more expensive than centralizing it, that is an argument for a platform team.

Who is accountable for AI system quality in production? If you cannot answer this question clearly, any team structure will produce accountability gaps. Resolve this first.

Team structure is a consequence of these decisions, not the starting point. The organizations that get this right tend to start from the accountability question and work backward to the structure, rather than picking a structure and hoping the accountability follows.