Evaluating an AI gateway comes down to three questions: does it technically do what you need? Will your security team sign off on it? And will the vendor be there when something breaks in production? 

Key takeaways:

  • 45% of software companies surveyed already have MCP in production, and security concerns are the top obstacle for the rest, cited by 64% of respondents
  • A technical capability matrix, not a vendor’s own pitch deck, should drive the shortlist
  • Vendor responsiveness during the evaluation itself is a leading indicator of support quality after signing, not a separate concern
  • Pricing negotiations go better when they’re separated from the technical decision, not folded into the same conversation
  • The vendors that consistently win these evaluations tend to share three traits: fast, transparent answers during diligence; MCP credential handling that goes beyond static API keys; and commercial terms built around the customer’s growth rather than against it

What’s the first step in evaluating an AI Gateway?

Bbuild your own requirements list before taking a single vendor demo, not after. A demo shaped entirely by what the vendor chose to show you will always look impressive and will always skip the gaps.

The strongest evaluations start with a technical capability matrix: a spreadsheet of specific, testable requirements (identity integration, MCP tool-call governance, audit logging format, data residency, redaction policy, and so on) scored against each vendor under consideration. One real evaluation we were part of ran two full working sessions through more than 50 requirements each, deliberately skipping line items that were already clearly met and spending the time on the gaps instead. 

How do MCP-specific requirements differ from general AI requirements?

MCP introduces a credential and tool-governance surface that a generic “AI policy” checklist usually misses entirely. If your evaluation only asks about model access and content filtering, it’s evaluating last year’s problem.

This gap is bigger than most buyers assume. An audit of more than 5,200 public MCP servers found 88% require credentials of some kind, and only 8.5% support OAuth, meaning the large majority rely on static API keys that are harder to rotate, audit, and revoke safely (Affinco, 2026 MCP Enterprise Adoption Statistics). A vendor evaluation that doesn’t specifically ask “how do you handle MCP servers that only support static API keys” is missing the exact failure mode most MCP deployments actually run into.

Security concerns are already the dominant blocker industry-wide: the same research found 64% of software company respondents cited security concerns as their top obstacle to broader MCP adoption, ahead of any other factor, even as 45% already have some form of MCP in production. 

FAQs

How many vendors should you evaluate in parallel?

Two to three, in most cases. Evaluating one vendor in isolation removes your negotiating leverage and your ability to sanity-check claims. Evaluating five or more spreads your technical team too thin to run a real capability matrix against each one, which usually means the evaluation degrades into a pricing comparison instead of a technical one.

What should the security and compliance review cover?

The direct answer: a completed security questionnaire, reviewed alongside the actual contract paperwork (data services agreement, order form) rather than in isolation, because the two need to agree with each other before signature.

A common and costly mistake is running the security review and the legal/commercial review as two disconnected tracks that only meet at the finish line. If the data services agreement says one thing about data retention and the security questionnaire says another, that gap surfaces during a compliance audit, not during the evaluation, which is a much worse time to find it. Reviewing them together, even if it means a slightly slower evaluation, catches inconsistencies while they’re still cheap to fix.

What should happen before you sign?

A clear plan for production rollout, not just a signed contract. The best evaluations end with an explicit answer to “what does week one in production look like,” not just “what does the contract say.”

Ask specifically about onboarding support for the first 30 to 60 days, what a non-production or sandbox environment looks like before your team touches live traffic, and how support requests get triaged and prioritized once you’re a paying customer versus during the sales cycle. Vendors who can answer these concretely, with a real process rather than a general assurance, are signaling they’ve done this rollout before. 

How long should an AI gateway evaluation take?

Most thorough evaluations run four to eight weeks from first technical call to signed contract, including a technical capability review, a security and compliance review, and a pricing negotiation. Rushing past the technical or security review to close faster is the most common source of regret six months into a rollout.

Should pricing be discussed during the technical evaluation?

Keep them as separate conversations where possible. Folding budget pressure into a technical evaluation tends to bias the comparison toward the cheapest option rather than the best fit, and makes it harder to negotiate price later since you’ve already signaled budget constraints before establishing which vendor actually meets your requirements.

What questions should be on every AI gateway RFP?

At minimum: how MCP servers with only static API key support are handled, what the deployment model requires from your team operationally, how identity and SSO integration works, what audit log format and retention are provided, and what a production incident response actually looks like in practice, not just in the SLA document.

Is a lower price always a red flag in an AI gateway evaluation?

Not automatically, but it’s worth understanding why. A lower price sometimes reflects a genuinely leaner product with less infrastructure to fund, sometimes reflects a vendor still building an early customer base and pricing aggressively to win logos, and sometimes reflects a product that requires more of your own team’s operational effort, which shows up as costing you later.

What does a vendor that passes this checklist look like?

One that treats the evaluation itself as the first proof point, not a formality before the real relationship starts. Everything in this checklist is really testing the same underlying question: will this vendor still be this good once the contract is signed and the pressure to perform eases off.

Barndoor is built to be evaluated this way on purpose. On the technical side, MCP tool calls are governed and credentialed individually rather than through the static API keys that 91.5% of public MCP servers still rely on, which is the specific gap most generic “AI policy” checklists miss. On responsiveness, evaluation questions get answered the same day over Slack rather than routed through a support queue, which is deliberate: how a vendor treats you during diligence, when they have the most incentive to perform well, is the honest floor for how they’ll treat you afterward, not the ceiling. And on commercial terms, pricing is built to scale with a customer’s actual usage and growth instead of being structured to extract the most from a signed contract.