AI vendor selection: the questions that separate a partner from a demo
AI vendor demos are unusually good at looking impressive, because the demo is exactly the case the product handles best. The questions below are the ones that reveal what happens outside it.
By Quality AboveAll · · 8 min read
- Ask what happens to your data, in writing: where it is processed, whether it trains anything, how long it is retained.
- Ask which models they depend on and what happens when those are deprecated, because that risk becomes yours.
- Ask how you would leave, before you join. Vague answers here are the most reliable warning sign available.
Data handling, in writing
Where is our data processed and stored, geographically. Is it used to train or improve models, for us or for anyone else. How long is it retained, and what is the deletion process. Who at the vendor can access it, and is that access logged.
Verbal assurances are worth nothing here; these belong in the contract. If a vendor cannot answer precisely, they either do not know their own architecture or the answer is one you would not like. See data privacy in AI systems for the framework.
Model dependency and continuity
Which underlying models does the product use, and what happens when one is deprecated or changes behaviour. A vendor whose quality shifts because their upstream provider shipped an update has passed that risk to you without disclosing it.
Ask how they detect and communicate quality changes. Vendors with their own evaluation suites and a change-notification process are operating professionally; those who say quality is stable because they have not noticed problems are not measuring.
Every vendor has an upstream provider. Ask what happens on the day that provider changes something, because it will happen.
Accuracy claims and evaluation
Ask how accuracy was measured, on what data, and whether you can run an evaluation on your own. A vendor confident in their product will support a paid pilot against your data with your own scoring.
Beware benchmark figures with no methodology attached, and be sceptical of any single accuracy number for a task with multiple failure modes. Our approach to evaluation is a reasonable template for what to ask them to support.
Pricing at your real volume
Model the cost at three times current volume before signing, since per-seat and per-request pricing scale in ways that can outpace the value. Ask what happens at overage and how much notice you get on price changes.
Ask what is excluded. Integration, support tiers, environments and data export are common line items that appear after the headline price is agreed.
Exit, security and the deal-breakers
How do we export our data, in what format, and does that include derived artefacts like embeddings and configuration. What notice period applies. What happens to our data after termination. A vendor who has not thought about exit has not thought about being a long-term partner.
Deal-breakers worth holding: no clarity on data usage, no security certification or willingness to complete a security review, no ability to run your own evaluation, and no named contact when something breaks. Broader vendor logic is in build versus buy for AI.
Frequently asked questions
Should we run a paid pilot before committing?
Almost always. A short paid pilot on your own data with your own scoring reveals more than any demo, and vendors who resist it are telling you something.
What certifications should we require?
Match the requirement to your risk: recognised security certification for anything touching customer data, plus sector-specific requirements where they apply.
How do we compare vendors fairly?
Score them against the same evaluation set on your own data, and weight the operational answers, data handling, continuity, exit, alongside accuracy.
Evaluating AI vendors and want a technical second opinion? A free 30-minute consultation will tell you which claims deserve scrutiny.
