How to structure an AI center of excellence

When an organization decides to build serious AI capability, the question of how to organize for it arrives quickly. Who owns AI? Who sets standards? Who helps business units that want to use AI but do not know how? Who maintains the platforms and tooling everyone shares?

The AI center of excellence is the most common answer to these questions. It is also frequently the wrong answer, not because the concept is bad but because the structure gets built in ways that create more problems than they solve.

The organizations that have built effective AI centers of excellence have learned what works through iteration. The patterns are consistent enough to be instructive.

What an AI center of excellence is actually for

Before the structure question, the purpose question: what should an AI center of excellence do?

The answer that produces effective centers of excellence is different from the answer that produces bottlenecks.

A center of excellence that exists to control AI creates a bottleneck. Every AI initiative requires approval. Standards are enforced through review gates. The center of excellence becomes the approving body that teams must get through to use AI. The center gets slower as the number of teams grows. Teams learn to work around it or wait for it. The organization’s AI capability does not grow at the rate it could.

A center of excellence that exists to accelerate AI creates leverage. It builds shared infrastructure that reduces the cost of any team’s first AI integration. It develops expertise that teams can access without having to build from scratch. It establishes standards that teams adopt because the standards make things easier, not because they are required. The center of excellence is a resource, not a gate.

The difference is not just philosophical. It produces different organizational structures, different hiring profiles, different success metrics, and different relationships with the business units the center serves.

The four functions that matter

Effective AI centers of excellence organize around four functions. Organizations that try to do more than this tend to spread thin; organizations that do less tend to leave gaps that limit adoption.

Shared platforms and tooling. The most consistent value centers of excellence provide is infrastructure that would be too expensive for any single business unit to build on its own but that every unit benefits from: model API access with appropriate rate limiting and cost tracking, prompt management infrastructure, evaluation tooling, monitoring dashboards. Building this once centrally is dramatically cheaper than having every team build its own version. The center maintains it; the teams use it.

Standards and patterns. Security, privacy, and compliance requirements for AI use do not vary by business unit. Neither do the engineering patterns that work well for production AI: how to handle prompt injection, how to implement appropriate content filtering, how to structure evaluations, how to handle inference failures gracefully. Codifying these into standards and reference implementations that teams can adopt is a multiplier on every team’s engineering effort. The center establishes the patterns; the teams implement them.

Expertise and enablement. Most business units that want to use AI have domain knowledge and problem clarity but limited AI engineering expertise. Centers of excellence that provide embedded support, office hours, or consulting help teams move from “we think AI could help with X” to “we have a working prototype we can evaluate” faster than they could alone. The center provides the expertise; the teams apply it to their domain.

Evaluation and measurement. AI quality does not evaluate itself. Centers of excellence that develop shared evaluation methodology, help teams define quality criteria, and provide tooling for measuring output quality against those criteria raise the floor on AI quality across the organization. Without this function, quality evaluation is inconsistently applied or skipped. The center builds the methodology; the teams apply it to their use cases.

What effective centers do not do

Several things that centers of excellence commonly try to own tend to make them less effective rather than more.

Owning all AI projects. Centers of excellence that try to own AI implementation for the whole organization rather than enabling business units to own their own implementations become development bottlenecks. The center does not have the domain knowledge that business units have. Projects run by the center on behalf of business units tend to be slower and less well-suited to actual use than projects run by business units with center support.

Approving all AI use. Centers of excellence that position themselves as approval bodies for AI initiatives create friction that reduces adoption and pushes teams toward workarounds. Approval gates work when the risk of unapproved use is high enough to justify the friction. For most AI use cases in enterprise settings, guidance and standards serve better than gates.

Setting AI strategy for the business. Centers of excellence are an operational function, not a strategic one. Business units know where AI creates value in their domain. The center’s role is to make capturing that value easier, not to tell business units where to apply AI. Centers that try to set AI priorities for the business tend to misallocate effort toward what the center finds technically interesting rather than what the business finds valuable.

Staffing the center

The staffing profile for an effective AI center of excellence is different from what many organizations hire for.

The instinct is to hire researchers and data scientists. These profiles are valuable, but they are not what centers of excellence need most. Centers of excellence that are primarily staffed with researchers tend to focus on sophisticated technical problems rather than on making AI easier to use across the organization. The output is research and prototypes rather than platforms and patterns.

The profiles that produce effective centers of excellence are platform engineers who can build shared infrastructure that is reliable, maintainable, and easy for other teams to use; AI engineers who understand production AI systems well enough to define the standards and patterns other engineers can implement; and technical program managers who can run the enablement function: running office hours, supporting teams through their first implementations, and synthesizing what is working and what is not across the organization.

Research and data science capability is valuable and belongs in the center, but it should be in service of the platform and standards functions rather than separate from them.

Measuring whether it is working

Centers of excellence are easy to run in ways that feel productive without actually accelerating organizational AI capability. The right metrics focus on what the center enables, not what it does.

Teams using shared infrastructure. If the center has built shared platforms and tooling, adoption of that infrastructure by business-unit teams is the metric that matters. A center whose infrastructure is not being used is either building the wrong things or failing at the enablement function.

Time from idea to prototype. For teams going through the center’s enablement support, how long does it take to get from “we have an AI use case” to “we have something we can evaluate”? This measures whether the center is actually accelerating teams.

Standards adoption without enforcement. If teams are voluntarily adopting the center’s standards and patterns because doing so makes their work easier, the standards are working. If adoption requires mandates, the standards are probably too burdensome or not providing enough value.

Reuse of shared components. When teams build AI features, what fraction of what they build is using shared components from the center versus building from scratch? Higher reuse means the center is building the right things.

The organizational positioning question

Where the center of excellence sits in the organization affects what it can do and how it operates.

Centers that sit inside IT tend to focus on infrastructure and governance. This is appropriate if the primary need is platform and tooling, but it limits the center’s ability to do the enablement work that requires business proximity.

Centers that sit inside a business unit tend to focus on that unit’s problems and be less available to the rest of the organization.

Centers that sit at the enterprise level, reporting to a CTO or CPTO function, are best positioned to serve multiple business units without organizational ownership biases. This is the most common structure for centers that operate effectively across large organizations.

The reporting line matters less than the mandate. A center with a clear mandate to enable business-unit AI adoption will usually find a way to operate effectively regardless of where it sits. A center with an ambiguous mandate or a mandate that conflicts with enabling adoption will struggle regardless of where it sits.

The stage where a center of excellence is worth building

A center of excellence is not the right structure for every stage of an organization’s AI journey.

In the earliest stages, when an organization is exploring whether AI is valuable and where, a center of excellence is premature. Small, cross-functional teams doing exploration are more effective than a center at this stage.

A center of excellence makes sense when multiple business units are trying to use AI simultaneously, when there is enough shared infrastructure need to justify building and maintaining it centrally, and when the enablement problem is real: teams that want to use AI but cannot figure out how without help.

Organizations that build centers of excellence too early before the demand exists tend to build bureaucracy that slows things down. Organizations that build them too late when multiple teams are all building the same infrastructure in parallel waste the leverage that centralization provides. The timing is a judgment call, but the indicators are usually visible: consistent requests from multiple teams for the same kind of help is the clearest signal that the time is right.