Notes

I read Dario Amodei’s essay We Must Pace the Frontier, and one part stayed with me: AI capability is moving faster than our ability to evaluate and control it properly.

This is not only a frontier model problem. We are already seeing it while building production AI systems.

In one of our workflows, the error rate was around 18%. Initially, it looked like the model was making poor decisions. After reviewing the traces, we found that the reasoning was mostly correct. The model was simply receiving the wrong context from the retrieval layer. Once we fixed that and improved how the evidence was selected and checked, the error rate dropped to 4%.

That experience changed how I think about AI reliability. Looking only at the final answer is not enough. We need to know what information the system used, why it selected that information, which actions it took, and when it decided not to involve a person.

This becomes even more important when agents are allowed to update records, communicate with customers, process financial information, or trigger business workflows.

I agree with the idea of pacing, but I think it also applies at the application level. Teams should not increase an agent’s autonomy just because a newer model performs better in a benchmark or demo. Autonomy should increase only when the system can consistently explain and support its decisions, detect failures, and escalate uncertain cases.

A confident answer and a well-supported answer are not the same thing. Production systems need to know the difference. The evidence I ask a team for before an agent acts without approval is the subject of Refuse Over Guess.

First posted on LinkedIn.

Agno’s recent work on Learning Machines caught my attention because we are building a controlled self-learning capability for a production financial document intelligence system.

The system already achieves strong field-level accuracy. But high accuracy does not necessarily mean operational autonomy. A single incorrect field can still require someone to review and correct the entire result.

Those human corrections contain valuable knowledge. The difficult part is deciding what the system should learn from them. Does a correction represent a recurring pattern? Is it a one-time exception, an operational preference, or simply noise?

If the system learns the wrong lesson, it can make the original problem worse at scale.

This is why I see production learning as a system-level capability, not a model feature. It needs evaluation, traceability, human judgment, and a safe way to remove incorrect learning.

Agno’s experiment also raises an interesting possibility: learn with a capable model, then transfer that knowledge to smaller, more economical models.

The line I keep coming back to for teams running agents in production: what can the system learn automatically, and what should always require human approval? The dual-model extraction pipeline is where we are working that out.

First posted on LinkedIn.

When matching a bank transaction to an open invoice, no single signal is reliable on its own. Vector similarity on the description catches paraphrases but is fooled by boilerplate. Semantic matching handles meaning but not identifiers. Counterparty identity is precise when present and useless when the payer name is truncated or misspelled.

The fix that held up in production was to run all three as separate ranked lists and fuse them with Reciprocal Rank Fusion: each candidate scores the sum of one over its rank in each list, plus a smoothing constant. A candidate that ranks well everywhere wins. A candidate that ranks first in one method and nowhere in the others does not, which is exactly the behavior you want when the methods fail differently.

Two practical notes. Normalize currencies before you rank, or cross-currency payments will never fuse. And treat disagreement between the lists as a confidence signal in its own right: when the fused top result is not clearly ahead, route the match to a human instead of guessing.