What Flock Safety’s AI System Reveals About AI Governance Failures
A WIRED investigation reconstructed the code behind Flock Safety’s next-generation AI surveillance platform and found it goes well beyond license plate tracking — analyzing behavioral patterns, flagging individuals based on inferred characteristics, and feeding those inferences into law enforcement workflows. According to WIRED, the system is already in use by some police departments, deployed well ahead of any public accountability framework.
That’s not an AI security story. It’s an AI governance story — and the failure mode it illustrates isn’t limited to law enforcement tech vendors.
The specific problem here: an AI tool expanded its capabilities significantly, it was already operational in a sensitive context, and the organizations deploying it had no formal mechanism to know what it was actually doing. That gap — between what a vendor says their AI does and what it actually does — is exactly what AI governance frameworks exist to close.
The AI Vendor Risk Problem Most IT Teams Ignore
If you’re an IT manager at a 100-person company, you might look at a story about police surveillance AI and think it doesn’t apply to you. But strip away the law enforcement context and you’re left with a pattern that’s common in mid-market AI deployment: a vendor’s AI tool does more than the sales deck described, and the buyer has no process in place to find out.
You probably have AI tools touching HR (resume screening, performance summarization), customer service (chatbots, sentiment analysis), finance (expense categorization, forecasting), and internal ops (meeting summarization, email drafting). Do you know what data each of those tools processes? Do you know whether the vendor updated their model in the last six months? Do you know what inferences those systems are drawing and whether any of those outputs feed into decisions about real people?
Most mid-market IT teams don’t, because AI vendor risk assessment hasn’t been a standard part of the procurement or renewal process. That’s changing — the EU AI Act is accelerating it for any company with EU exposure — but most SMBs are still operating on trust and a SaaS subscription agreement.
What an AI Tool Register Would Have Caught
The Flock situation exposes a specific control gap: no visibility into AI system behavior post-deployment. That’s different from the more familiar vendor risk concerns about data handling or SOC 2 status. It’s about whether the AI is doing what you think it’s doing, and whether you’d know if that changed.
An AI tool register — a living inventory of every AI system your organization uses, including what it does, what data it touches, who owns it, and how it’s been risk-assessed — is the most basic version of that visibility. It doesn’t catch everything, but it creates accountability. When you have a register, you have a named owner for each tool. That owner is responsible for staying current on what the vendor is actually shipping.
For mid-market AI compliance, this isn’t a compliance team problem. It’s an IT function, and it’s tractable with the right structure.
Do This Week: Start an AI Risk Registry With Vendor Review Criteria
You don’t need a GRC platform to do this. You need a structured template and about two hours of your own time, plus cooperation from department heads to surface the tools their teams are actually using.
Here’s the minimum viable version. For every AI tool in your environment, capture: the tool name and vendor, the business function it supports, the data categories it processes (especially personal data or sensitive business data), who the internal owner is, the last date someone reviewed the vendor’s current capabilities, and a simple risk tier (high/medium/low based on data sensitivity and decision impact).
Once you have that list, do a gut-check on the high-tier tools: read the vendor’s latest documentation, check their changelog if they publish one, and ask yourself whether you’d know if they added a new model capability without telling you. If the answer is no, that’s your first follow-up.
The Flock story is an extreme case, but the underlying question it raises is the same one you should be asking about your HR AI vendor or your customer service chatbot: what is this system actually doing today, and how would I know if that changed?
If you want a head start on structuring that inventory, download the free AI tool register template — it’s built for IT teams that don’t have a dedicated compliance function and need something they can actually maintain.
AI Governance for IT Teams Isn’t About Perfection
No mid-market IT team is going to audit every AI vendor’s model weights. That’s not the goal. The goal is having enough visibility that you’re not the last person to find out when an AI tool you deployed is doing something your leadership, your legal team, or your customers would object to.
The organizations using Flock’s system weren’t necessarily negligent in any conventional sense. They deployed a tool from an established vendor in their space. What they lacked was a governance structure that would have prompted someone to ask what the next version of that tool was going to do before it was live.
That’s a solvable problem. It starts with a register, it matures into formal risk assessment and policy controls, and it scales as your AI footprint grows. But you have to start somewhere, and a tool inventory you actually maintain beats a governance framework you don’t.
