The Problem With How Most SMBs Evaluate AI Tools
Most companies buying AI tools today do the same thing: they check the vendor's security page, maybe skim a SOC 2 report summary, and move forward. That's not a risk assessment — that's a procurement checkbox. When an auditor or regulator asks how you evaluated the tool before you let it touch customer data or internal systems, "we looked at their website" is not going to hold up.
on Unsplash
Running a defensible AI tool risk assessment doesn't require a GRC platform or a dedicated compliance team. It requires a structured set of questions, a documented record of the answers, and evidence that you actually made a decision based on what you found. Here's how to do that.
Start With the Data Flow, Not the Vendor's Marketing Page
The first question in any AI tool risk assessment isn't "Is this vendor secure?" It's "What data does this tool see, and where does that data go?"
Before you send a single question to a vendor, map the data flow internally. What data types will the tool access or process? That includes input data (what users type or upload), output data (what the model generates and stores), and metadata (usage logs, prompt history, API call records). If the answer is "employee names and internal documents," that's a different risk profile than "anonymized log files."
Once you have that picture, you can ask the vendor questions that actually mean something. The key categories are:
Data handling and retention. Does the vendor store your inputs and outputs? For how long? Where? Under what access controls? Many AI vendors default to storing prompts for model improvement — some allow you to opt out, some don't. You need to know which situation you're in, not assume.
Subprocessors. The vendor you're buying from is rarely the only entity that touches your data. Ask for a current subprocessor list and find out what each subprocessor does. A vendor that routes data through multiple third-party inference or storage providers without disclosing it is a real risk — and it's surprisingly common among newer AI platforms.
Model training on your data. This is the one that trips up most SMBs. Ask directly: does the vendor use customer inputs to train or fine-tune models, either now or under future terms? Ask whether that's opt-in or opt-out, and whether opting out requires a separate contract addendum or is available to all tiers. If you're on a free or entry-level tier, there's a reasonable chance your data is being used for training by default.
Breach notification. Your vendor agreement should specify how quickly the vendor will notify you if there's a security incident involving your data. "Promptly" is not a useful standard — ask for a specific timeframe. Under the GDPR's breach notification requirements, controllers have 72 hours to notify supervisory authorities after becoming aware of a breach, which means you need vendor notification faster than that to meet your own obligations. If you have any EU exposure at all, this term matters.
Contractual protections. Check whether the vendor offers a Data Processing Agreement (DPA). If your organization is subject to GDPR or processes data on behalf of EU-based customers, a DPA isn't optional — it's a legal requirement for engaging any processor. Many AI vendors have standard DPAs, but the terms vary significantly. Read the data subject rights section and the subprocessor change notification clause.
Building a Consistent Scoring Record
A risk assessment isn't just a list of answers — it's a documented judgment. Auditors want to see that you gathered information, applied a consistent methodology, and made a reasoned decision. That means every vendor evaluation needs to produce a record with the same basic structure.
At minimum, document: the tool name and version evaluated, the date of the assessment, who conducted it, what data the tool accesses, the questions asked and answers received, the risk rating you assigned, and any compensating controls or contractual terms you put in place as a result.
Risk rating doesn't have to be complicated. A simple three-level scale — low, medium, high — applied consistently across all your AI tool assessments gives auditors what they need to see: that you treat tools differently based on their actual risk profile, not just their vendor size or popularity. A publicly available AI writing assistant that sees no internal data is a different risk tier than a vendor API that ingests customer-facing support tickets.
The consistency is the point. If an auditor sees five AI tool assessments that all use different formats, different criteria, and different scales, it looks like you're improvising. A standard template applied to all five vendors looks like a program.
What Auditors Are Actually Looking For
If you're going through a SOC 2 audit, preparing for a GDPR inquiry, or working through a customer due-diligence questionnaire, the evidence you'll be asked to produce centers on a few consistent themes.
First, they want to see that you identified your AI tool inventory. You can't govern what you haven't catalogued. A list of AI tools in use, who owns each one, and what data it processes is the foundation everything else sits on.
Second, they want evidence that you assessed risk before or shortly after deployment — not three weeks before the audit. Dated assessment records matter. A risk assessment completed today for a tool you deployed eighteen months ago raises questions. If you're in that position, document it honestly: note when the tool was deployed, acknowledge the gap, and treat the current assessment as a remediation action.
Third, they want to see that risk findings connected to decisions. If your assessment flagged a vendor's subprocessor list as incomplete and you followed up to get the full list — document that. If you decided a tool's data retention terms were acceptable because it only sees non-sensitive data — document that reasoning. Evidence of a decision process is more valuable than a perfect score.
Finally, for any vendor handling personal data, they'll want the DPA in hand. Not a reference to it, not a screenshot of the vendor's privacy page — the executed agreement.
The Part Most Assessments Skip: Ongoing Monitoring
AI vendor risk assessment isn't a one-time event. Vendors update their terms of service, change their subprocessors, and modify their data retention practices — sometimes quietly, sometimes buried in a changelog that almost no one reads. Your initial assessment is only defensible if you have a process for catching material changes.
At a practical level, that means two things: first, subscribe to vendor terms-of-service update notifications where available, and flag AI vendors specifically for closer review; second, set a calendar reminder to re-evaluate high-risk AI vendors at least annually, or whenever you become aware of a significant change (an acquisition, a major product update, a publicized security incident).
This doesn't require a lot of labor. It requires a process that's documented, assigned to someone, and actually followed.
Do This Week
Pick the AI tool your team uses most frequently and that has the broadest data access — likely a productivity AI, a customer-facing chatbot, or an internal document assistant. Pull up your current vendor agreement and answer these five questions:
- Does the vendor store your inputs and outputs? For how long?
- Is there a published subprocessor list, and is it current?
- Does the vendor use your data to train models by default?
- What is the contractually specified breach notification timeframe?
- Is there a signed DPA in place?
Document your answers in whatever format you have available — even a shared document is a starting point. Note any gaps. That single exercise produces the first record in what should become your AI vendor risk register, and it gives you a baseline to work from on every subsequent evaluation.
If you want a structured template that produces consistent, audit-ready evidence across all your AI tool evaluations, the AI Risk Assessor (PROFESSIONAL tier) is built specifically for that workflow — it walks through vendor questionnaires, stores responses, and exports evidence in a format designed for auditors, not spreadsheet archaeology.
Closing Thought
The goal of an AI tool risk assessment isn't to find perfect vendors — it's to document that you asked the right questions, understood what you were agreeing to, and made informed decisions. Most AI vendors won't fail a well-run assessment. But an assessment that doesn't exist, or that can't produce evidence when an auditor asks for it, fails automatically.
The bar for "defensible" is lower than most IT managers expect. Structure, consistency, and documentation will get you there. The hard part is usually just starting.
