Interview with Al Calabrese from Netrio

1. A lot of messaging in the market confuses AI adoption with plug-and-play infrastructure, and when it’s time to evaluate realized ROI, companies question the tool- never their methods. With mid-market companies struggling to move beyond failed AI pilots, where do you think the real hitch in their AI roadmap is?

The hitch is a governance gap, not a tooling gap.

We recently commissioned a research project, conducted by the independent firm Censuswide, of 401 U.S. mid-market IT leaders, and 96.3% said they’re confident of measurable ROI from AI within 24 months and 81.8% already have AI in production or widespread use, but only 26.2% say that use is actually scaled and governed enterprise wide.

Everyone else sits somewhere on a sliding scale of ungoverned, siloed adoption, and that’s exactly where pilots stall. Companies rush a tool out to get ahead of demand, skip the methodology, then question the model when ROI doesn’t appear.

The fix is to treat AI like any other portfolio decision a CIO already makes: do a readiness assessment up front and have governance built into the conversation from day one, not bolted on after something breaks.

2. You’ve spent over two decades deploying highly rigid, deterministic enterprise systems. How do you build executive confidence around the probabilistic nature of LLMs, given their tendency to hallucinate?

You don’t build it by pretending these systems behave like the deterministic platforms I’ve deployed for twenty years – you build it with governance and human checkpoints.

Our recent Netrio-Censuswide survey flags the danger: 55.9% of mid-market leaders already act on AI outputs, in most cases, without any additional human review. That’s confidence outrunning control, and it’s how unchecked, probabilistic output turns into a problem.

The reassurance for executives is in the design – we build approval steps into the workflow, at the end and sometimes in the middle, so a human is confirming the model isn’t drifting and is producing what’s needed. That comes straight from my security advisory background and is baked into how we deliver these services, which turns an LLM from a leap of faith into a technology with a defined, manageable risk profile.

3. Shadow AI has often been perceived as a security threat, but many employees- including decision-makers- quietly rely on it for practical value. With the correct guardrails in place, do you believe Shadow AI can be legitimized as a growth engine?

Yes, it can.

The first move is to stop treating shadow AI purely as a threat and start treating it as signal. In nearly every engagement, leaders already know it’s happening, and our data backs that up: 81.8% of organizations report having AI already in production or in widespread use, but only about half (53.1%) have full visibility into what’s actually being used.

A shadow AI discovery closes that gap – it captures genuine demand and turns it into a report and a risk register, so an invisible liability becomes an actual input to your roadmap. From there it’s about offering an approved toolkit and a clear policy, because employees largely want to play by the rules once they know what the rules are.

The value that drove the shadow usage gets legitimized and scaled, while the risk gets managed instead of ignored.

4. There’s a grey area with shadow AI: most employees use it to hit their goals without realizing the data exfiltration they may be contributing to. How can you create a culture of education on this security gap without hindering productivity?

Education works best as clear policy backed by data classification, not a blanket “no.”

The cautionary tale we share is real: one organization found its own proprietary IP coming back out of a public model, because early on people had dropped documents into a free tool without turning training off. The exposure is widespread – only 63.1% of respondents in our survey have assessed whether sensitive data is being entered into AI tools and put controls in place, which leaves more than a third flying blind.

You close that gap with a DLP regime and a classification system: label your most sensitive documents and define what each label permits, so “internal use” material might be allowed in an AI tool while “confidential” is not.

A fast first step is simply moving people off personal accounts onto their work email and approved tools. And none of this hurts productivity, because you’ve drawn the line around the data, not the work.

5. Over 42% of enterprise AI tools are replaced or discontinued- either to keep pace with the foundational models or due to a lack of ROI. To what extent does this transient nature of most AI initiatives harm the enterprise’s internal momentum compared to the vendor’s financial hit?

The internal-momentum cost is the one I’d worry about more, and it’s the one that organizations underestimate. If you anchor your practice to a single tool, every replacement resets your users, your training and your trust – and that churn is expensive when the mean expected AI investment over the next 12–24 months is roughly $446,000. 

It’s also structural: our survey found no single vendor dominates the market and 96.6% of leaders are satisfied with a mix of tools, so a multi-platform reality is here to stay. A lot of the “failure” that drives replacement is really a failure to optimize. We see customers using AI daily but defaulting to the heaviest model for every task, when Sonnet is a workhorse and Haiku is right for plenty of jobs.

The way to insulate yourself is to anchor on methodology rather than product: keep an approved tool list, match the model to the use case, and stay disciplined about usage. We do this internally – we monitor our own seats and shift them to active users, so cost and value stay aligned even as the tools underneath keep changing.

6. The marketing hype keeps pushing fully autonomous agents like a cure-all, but enterprise reality requires humans in the loop. Have you laid down criteria to gauge precisely when and where human judgment must take the wheel?

The criteria come down to where the output is going and what it would cost you to get it wrong. The encouraging sign is that users are already starting to think this way: in our survey, 25.9% of respondents said they act on AI outputs without review only for low-stakes decisions. That instinct is exactly what you formalize.

A concrete example: with email and similar integrations, read access is a powerful, low-risk leg up, but you slow down on write access until there are skills or agents built to frame and constrain it. So, “research the email and just respond” is never a good idea without a practice behind it.

More broadly, we sequence the journey: capture simple productivity gains first, then move to agentic workflows with approval steps, and only then extend real autonomy where the risk allows.

Compliance raises the bar sharply – our finance and healthcare customers operate under strict regimes, so the heavier the stakes and the compliance exposure, the earlier and more often a human takes the wheel.

7. To scale Netrio’s AI practice, you’re also internally transforming your workforce- traditional IT engineers and managed service professionals who built their careers on legacy systems. What steps must leaders take to build a holistic culture of unlearning for such professionals?

We’ve lived this internally, so I’d start by saying the goal isn’t to discard what these professionals know…instead, let’s take what they know and repurpose it.

My own path is the template: a security advisory and risk-assessment background didn’t become obsolete when we built the AI practice, it became the governance backbone of it. Netrio became its own first customer, baking AI workflows into our ServiceNow instance and our customer-support operations, so our engineers felt the gains in their own day-to-day rather than in the abstract.

The market is moving the same way – 42.1% of respondents in our survey have already launched proactive reskilling and upskilling programs, and AI training and upskilling for employees was the single most common area of planned AI investment.

The practical levers are training and change management, anchored by communication: be clear-eyed about the chosen toolkit and stand-up AI committees at the team level, so people feed real use cases into a living matrix.

When you connect AI to instincts, the old way loosens its grip, because they recognize their hard-won judgment still matters.