What AI Governance Actually Requires at the SMB Level
AI governance sounds like something large enterprises do in conference rooms with legal teams and dedicated compliance staff. In practice, most SMBs are already doing some version of it — they’re just doing it inconsistently, without documentation, and without a clear owner. If you’re an IT manager at a 100- or 200-person company and you’ve been handed responsibility for AI governance, the problem isn’t that you don’t know what the goal is. The problem is that nobody has handed you a practical starting point.
This post gives you one. The checklist below is organized around four operational areas: inventory, risk, policy, and evidence. Work through them in order and you’ll have a defensible AI governance baseline — not a perfect enterprise GRC posture, but something real that you can build on.
Before we get into the checklist itself, a quick framing note: AI governance isn’t a one-time audit. It’s an ongoing operational practice, which means the checklist items below aren’t checkbox-and-forget tasks. A few of them require monthly or quarterly attention. We’ll flag which ones.
—
Step 1: Build Your AI Tool Inventory (This Is Where Governance Starts)
You can’t govern what you don’t know about. The first step in any AI governance program is producing an accurate AI tool register — a centralized list of every AI or machine learning tool in active use across the organization, including tools employees are using on their own initiative without IT approval.
That second category is the one that gets companies in trouble. Shadow AI — unapproved AI tools employees adopt through personal accounts, browser extensions, or free-tier SaaS products — is common at companies of every size. At a 150-person company, it’s not unusual to find a dozen or more AI tools in active use that IT has no visibility into. Some of them are processing business data.
Building your initial inventory doesn’t require specialized tooling. Start with three sources: your SaaS spend data (check your expense reports and credit card statements for AI-adjacent vendor names), your SSO or identity provider logs, and a short employee survey asking people what AI tools they use for work. Cross-reference those three sources and you’ll surface most of what’s out there.
For each tool, capture at minimum: the tool name, the business function using it, the data types it touches, whether it was IT-approved, and who owns the vendor relationship. That’s your AI tool register. If you want a structured template to start from rather than a blank spreadsheet, download the free AI tool register template — it includes the core fields and a basic risk-tier column you’ll use in the next step.
Inventory review cadence: quarterly at minimum. AI tool adoption moves fast, and your register will go stale if you only update it annually.
—
Step 2: Run a Basic AI Risk Assessment on Each Tool
Once you have your inventory, triage it. Not every AI tool carries the same risk, and you don’t have time to do a deep vendor risk assessment on a grammar checker. Build a simple risk tier for each tool based on two variables: the sensitivity of the data it processes and the criticality of the business function it supports.
A tool that processes only publicly available information and supports a non-critical workflow is low tier. A tool that processes customer PII, employee records, or confidential business data and sits in a critical workflow is high tier. Everything else falls in the middle. This triage doesn’t require a formal methodology — it requires honest answers from the business owners of each tool.
For your high-tier tools, go deeper. Pull the vendor’s privacy policy and terms of service and answer these questions: Does the vendor use your data to train its models? Where is your data processed and stored? Does the vendor have a published AI use policy or responsible AI framework? What are the data retention and deletion terms? If the vendor can’t answer those questions in writing, that’s a finding.
For mid-market companies with EU customers or EU-based employees, you also need to consider whether any of your high-tier AI tools fall under the EU AI Act‘s prohibited or high-risk categories. The Act’s prohibited practices provisions apply regardless of where the vendor is based if the system’s output affects people in the EU. That’s not a hypothetical for most SMBs selling into Europe — it’s a current compliance question.
Risk assessment review cadence: annually for low-tier tools, semi-annually for high-tier tools, and immediately when a vendor announces a material change to its data practices.
—
Step 3: Get Your AI Policy in Writing
An AI tool inventory without a policy is just a list. The policy is what tells your employees and your auditors what the rules are — what tools are approved for what data types, what the review and approval process looks like for new tools, and what happens when an employee violates the policy.
For most SMBs, an AI use policy doesn’t need to be a 40-page document. A focused, three-to-five-page policy that covers acceptable use, data classification requirements, approval workflow for new AI tools, and incident reporting obligations is enough to start. The goal is to close the gap between “we have a list of tools” and “we have documented, communicated rules about how AI can be used here.”
Three things the policy must address that teams often omit:
First, data handling rules by tier. Employees need to know what categories of data they’re allowed to put into an AI tool. If your risk assessment categorizes tools by data sensitivity, the policy should map to that same taxonomy. Don’t make employees guess.
Second, personal account prohibition (or conditions). Free-tier and personal-account AI tools are where most shadow AI problems originate. The policy should either prohibit them outright or define exactly what conditions make them acceptable (e.g., public information only, no business email, no client data).
Third, a clear reporting mechanism for AI incidents. If an employee suspects data was sent to an AI tool it shouldn’t have been — or if an AI output caused a business problem — they need to know where to report it. An AI governance policy that doesn’t include an incident reporting path is incomplete.
If you need a starting point, download the free AI use policy template. It’s structured for SMBs and covers the core elements above without requiring a legal team to make it usable.
Policy review cadence: annually, and whenever a material regulatory change occurs (the EU AI Act’s prohibited practices provisions, for example, are already in effect).
—
Step 4: Assign Owners and Document Your Controls
AI governance for IT teams fails most often not because the policy is wrong but because nobody owns the individual controls. An AI tool register with no assigned reviewer goes stale. A risk assessment with no owner doesn’t get updated when a vendor changes its terms. A policy with no enforcement owner doesn’t get enforced.
For each tool in your register, assign a named business owner and a named IT reviewer. The business owner is accountable for how the tool is used and for flagging changes in use cases. The IT reviewer is accountable for the periodic risk assessment and for keeping the vendor record current.
Beyond tool-level ownership, you need someone accountable for the AI governance program itself. At a 150-person company, that’s usually the IT manager, but it should be documented, not just assumed. Write it down. If your company ever faces an AI-related audit or incident response, “we had informal governance” is not a defensible position. “Our IT manager owned the program with documented quarterly reviews” is.
—
Step 5: Build Your Evidence Trail Before You Need It
The final area of the checklist is the one most IT managers skip until they’re under pressure: evidence collection. Governance posture means very little if you can’t demonstrate it. Auditors, enterprise customers, and regulators aren’t asking whether you have a policy — they’re asking whether you can prove you’re operating it.
At minimum, your evidence trail should include: dated exports of your AI tool register, completed risk assessment records for high-tier tools (with reviewer name and date), policy acknowledgment records showing employees have reviewed and accepted the AI use policy, and a log of AI-related incidents (even if the log is currently empty — an empty log with a documented review date is better than no log).
Keep these records somewhere retrievable. A shared drive folder with consistent naming works. A dedicated AI governance platform works better, because it provides timestamped records and audit-ready exports without manual assembly — but the shared drive approach is a legitimate starting point for teams that don’t yet have tooling in place.
Evidence review cadence: quarterly spot-checks; full evidence export before any customer security review or audit.
—
Do This Week: Run a 30-Minute AI Tool Inventory Sprint
If you’ve read this far and you’re not sure where to start, start here: block 30 minutes this week and pull your last three months of company expense reports and SaaS invoices. Flag every line item that looks like an AI tool — any product name you don’t immediately recognize, any AI-adjacent vendor category. Add them to a spreadsheet with three columns: tool name, who’s using it, and what data it touches. That’s your version-one AI tool register.
It won’t be complete. You’ll find more tools through your SSO logs and through employee conversations. But a version-one register you actually have beats a comprehensive register you haven’t started.
Once you have that list, you’ll know which tools need a risk assessment and which can wait. That sequencing is most of the work. The free AI tool register template gives you a structured format for that first sprint, including the risk-tier column that feeds directly into your assessment work.
If you want to see where your program stands against a full AI governance framework — not just the inventory step — the free RAGP maturity assessment maps your current posture across all four governance phases and shows you where the gaps are. It takes about ten minutes and produces a result you can actually act on.
For teams that want to operationalize the full checklist above without building everything manually, InfoDefenders’ AI Governance Manager tier handles the register, risk assessments, policy library, and evidence exports in one place. See AI governance pricing to find the tier that fits your team size and maturity level.
