Numbers Lie too, a real life example of AI screwing small business

This is an actual story of a client of mine getting screwed by an AI program which doesn’t have thoughtful governance and human oversight.

Here is the history you need to know: 

My client is a fleet service company servicing fleets for both medium and large companies. Their biggest client is Amazon. iNow amazon doesn’t manage their fleet directly nor delivery process directly. They hire Delivery Service partners (DSP’s) who use a company like Element to track everything to do with service and maintenance. It is a big ecosystem with a lot of moving parts. Now you know the players, here is the game which my client got screwed.

Element uses a proprietary AI to track and flag Fleet service companies like my clients. The AI looks at 3 categories: Turn around time, Pricing, and behavior. Here comes the meat of the story. The AI tries to find inconsistencies and patterns inside these metrics to flag them. While this is a good use for AI in principal, as I continue the story you will realize that the lack of governance and human oversight is going to cost my client ⅓ of his business revenue, The DSP’s, who are very happy with my client, to have slower turnaround times and not as quality of work. First let's talk about governance. Element, is the largest publicly traded automotive fleet manager in the world. It is smart and wise for them to build AI. It is not smart to build AI that does not have appropriate governance around it. First, AI generative flags with proper governance should generate strikes or the like for the service provider to learn from. Next, when enough flags have arose to warrant termination, there should be protocol to appeal and look to see if these infractions are legitimate.

The way this AI works it just generates a strike. It essentially reports that strike back to Amazon and just like that Amazon cancels the companies ability to work with them because they have too many strikes. Next let’s talk about human oversight. In my professional opinion at this point in AI agent adoption you need AI managers to have human oversight on decisions, “throught” processes, and metrics as to how well the program is doing its job. In cases where the AI flags a company enough to draw business away from them, the human with oversight should be able to flush out the knowledge AI brings with the wisdom of a human. Now you may say, Josh, Perhaps your clients infractions are more important than you realize and the AI program has worked perfectly. Ill let you be the judge: My client was given 3 flags that it received in 2025. 0 in turnaround time. 0 in pricing. All in “Behavior” broad category. Here are the AI flags and responses that aren’t able to be heard because of little to no human oversight. 1. Door roller concern

We were told there was a truck where we replaced 4 door rollers, then about a month later submitted for 3 more.

Our response: At the first visit, only 4 rollers were bad, so we only recommended and replaced those 4. A month later, 3 additional rollers failed, so we submitted for those. To us, this shows integrity — we did not recommend replacing every roller just because some were bad. We only recommended what was actually needed at the time.

2. Hino oil change 

A Hino was brought in for an oil change in July, completed, and then another oil change request was submitted about two weeks later.

Our response: The second submission was requested to check approval and determine whether service was due. Nothing was serviced at that time. The ticket remained open until September, when the vehicle came back and was actually due. At that point, the approved service was completed and paid.3. Auto Integrate 

Maintenance items were submitted, but some were denied due to mileage/service interval concerns.

Our response: We do not recommend maintenance unless it is requested or appears due through the system/customer request. In this case, there appears to have been a mileage inconsistency. Maintenance was denied, except for the oil change. The approved repairs were completed and paid. Again, this seems to come from us checking approvals in advance, not from intentionally submitting improper work.Again, 0 flags on turn around times, and 0 flags on pricing. The DSP’s, the people that actually do the work are over the moon with my clients performance and also trying to get past the AI gate keeper. So because of low wisdom in AI governance of their program and low human oversight my client looses, the DSP’s lose, and amazon loses. I am not advocating against the use of AI tools by reporting my clients case. I am advocating to use “wise” likely 3rd party companies that can institute common governance practices for your AI programmes. Preferable companies that will put humans first. Also, I am advocating for companies to upskill employees to be AI managers. Keep people in the loop, it will save a lot of money and frustration in the long term. 

What will happen to my client? IDK but I am sure they are not alone in being the victim of shotty ai governance and no human in the loop procedures. If you have a story or know of a story like this, send it my way. It is a part of my mission to make sure we do AI right…In a humans first way.