Council Post: What Should AI Actually Be Trusted To Do In The Enterprise?
Rujul Zaparde, cofounder and CEO, Zip.
getty
On the morning of October 20, 2015, a young research team invited a legally blind man into a car on a residential street in Austin, where he rode for 10 minutes with no one at the wheel.
This was the first fully driverless ride on a public road in America, courtesy of the team that would become Waymo.
Now, eleven years later, I can sit in a driverless Waymo and cruise the streets in San Francisco while answering Slack messages. The sight of driverless Waymos has faded into the background in the city; other than a few tourists on Market Street, almost no one bats an eye anymore at the sight of a fully autonomous car with an empty driver’s seat.
When business leaders ask me how to bring reliable AI into the enterprise, my answer starts with the “trust layer,” which is exactly what Waymo built between those two rides. Here is how, at our company, we build trust through governance.
How Waymo Earned The Trust To Scale
Waymo logged tens of millions of supervised miles, and billions more in simulation, under the tightest oversight on the hardest decisions a machine can face. These logged miles produced a record that regulators, insurers and cities could freely audit, culminating in a record that shows 90% fewer serious-injury crashes than with human drivers in the same cities.
Waymo earned the right to the empty driver’s seat by building proof that outsiders could trust and verify. After demonstrating repeatable governance to the required guardrails, autonomy was free to follow.
Trust is the emergent property of transparent governance. This holds for a car at a four-way stop, and it holds for an AI agent about to approve a seven-figure business software renewal.
Executives Ready For Autonomy They Can Trust
In my work, I speak to executives looking for new, faster and more autonomous ways to run procurement and sourcing. They’re eager for sourcing events that run themselves and renewals negotiated without a human in every thread. Finance leaders I speak to want a close that doesn’t put half the team on accruals for the last week of the month.
According to our recent survey (login required) of 1,050 procurement, finance, IT and operations leaders, 93% expect procurement technology to change materially over the next three to five years. We expect them to be right about that.
But almost no one can predict the exact scope of this future. Only 13% say they can plan with confidence three to five years out. Only 2% expect a fully autonomous procurement function.
At the same time, almost everyone is using AI tools in their work that both accelerates and automates a tremendous amount of day-to-day work, and it’s clear many of us are becoming reliant on it, perhaps overly so.
While most business leaders say they’re using AI “multiple times per day”—and I am one of them—many say they are also using AI tools their employers haven’t actually sanctioned or provided. These are seasoned executives, not summer interns.
Unsanctioned, ungoverned AI running rampant within the enterprise is a risk that compounds. It’s not what any auditor wants to hear about.
The leaders expecting change are right about where this ends in theory, but uncertain about how to get there in practice. Autonomy isn’t a result of the frontier labs building increasingly better models leading to shinier new toys; it arrives mile by governed mile, just as it happened on the roads of Austin.
The Constraint: Permission, Not Capability
AI is already capable of far more than most companies realize or are taking advantage of. What’s missing for scalability is permission, and permission is a question of governance. In this era of agents “going rogue,” the enterprise is more in need of guardrails than ever.
Embedded governance ensures AI adheres to your company’s policies, acts under the same threshold an employee would and leaves a record an auditor can reconstruct. As that record accumulates, the set of decisions it’s trusted with by nature is able to widen.
The alternative is the approach we’re seeing too many companies run today, which is a policy outline that systems aren’t built to enforce. Policy instructions without a strict enforcement mechanism isn’t governance.
Those building these mechanisms within new foundations in mind are already seeing returns. Few of the organizations we surveyed reported clear, measurable ROI from AI investments in their procurement technology. But most have put procurement in a central or collaborative role in AI governance.
We’re building a tool to properly instill an AI trust layer for procurement and finance—the most difficult functions at any company to properly govern.
An agent trusted to approve spend against policy, negotiate within critical guardrails and touch the financial close has cleared a bar higher than any other function demands. If AI can earn trust where the money moves, it can earn it anywhere, and everything after that gets built on top.
Every company will inevitably have to end up with governed AI policies.
The choice for business leaders today is whether you instill this trust layer yourself, or inherit it from a regulator, an auditor or an incident.
Those that ask only how AI can accelerate their business will get an impressive sounding answer. They will run their pilots, publish a corporate blog about transformation and may soon discover they can’t rely on what they built for business-critical spend.
Those who begin by asking what AI should be trusted to do will hand their teams governed Superagents working within guardrails, that are then permitted to work autonomously, build value sustainably and grow business exponentially.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?