The internal AI tools that get used six months in look very different from the demos that won the budget. Here is what the durable ones share.
Operator copilots are the AI category we get asked about more than any other. The premise is simple: put a model next to a team doing knowledge work, and let it help them work faster. The demos are good. The retention curves, six months after launch, mostly are not.
We have spent the last year looking carefully at the copilots that survive and the ones that don't, and the pattern is clear enough that we are comfortable saying it out loud.
What the durable ones share
The copilots that stick are embedded where the work already happens. They live in the ticket queue, the CRM, the support inbox, the IDE. They are never a separate tab the operator has to remember to open. The work brings the team to the tool, not the other way around.
The durable ones are also faster to dismiss than to engage with. The default is to ignore the suggestion; engagement is opt-in. This feels counter-intuitive, especially to the team that built the model and wants people to use it. It is also the reason the team learns to trust the tool. They engage when a suggestion is good and tune it out when it isn't. That gives the team running the system an honest signal, which an engagement-by-default tool will never have.
The third thing the durable copilots share is that they learn from the operator's actual decisions, not from a parallel labeling project. The people who use the tool are also the people who train it, and they don't have to do extra work for that to be true. That last bit matters more than it sounds. The tools that require a separate labeling effort to improve mostly don't improve.
What the failures share
The copilots we have watched fade out usually launch with a leaderboard. Adoption metrics dominate the first review meeting. Six months in, the leaderboard has been quietly retired and the tool is used by three people, none of whom are the ones who originally advocated for it. The leaderboard told the wrong story for too long.
They also tend to produce text that nobody reads. The summaries are technically accurate and operationally useless. The operator scrolls past them and goes to find the underlying record. The tool is now a tax on the workflow, and it will not be there in a year.
A test that works
Six months after launch, ask the operators what they would miss if the tool went away tomorrow. If the answer is the time-savings number on the dashboard, the tool is not going to make it to month nine. If the answer is a specific decision that would be harder to make without it, you have a copilot that sticks. That answer is the only metric we trust.