Why observability and evals matter in lending AI — and how we built for over 99% accuracy

TL;DR: Ocrolus, an AI workflow and analytics platform for lenders, uses a four-layer evaluation system combining human baselines, AI-native evaluation, deterministic validation and targeted workflows to maintain greater than 99% accuracy across predictive models. Strong observability isn’t just an operational necessity. It’s what makes AI viable in regulated financial services environments.

In lending, there’s a threshold that separates AI that works in theory from AI that works in production. It starts with accuracy, but it doesn’t end there. Latency, cost and consistency are equally non-negotiable. Meeting that threshold once isn’t enough. The real challenge is maintaining it as you scale across diverse data, customer populations and an ever-expanding range of use cases.

That’s where observability and evaluation become foundational: not nice-to-haves, but the scaffolding that makes a regulated AI system viable.

The challenge: predictive systems don’t come with guarantees

Legacy systems in financial services tend to rely on deterministic logic: decision trees, if/then rule chains and switch statements. Those systems are predictable when inputs stay consistent, but then also outputs do, too.

Predictive models are different. Ocrolus is an AI workflow and analytics platform for lenders with over 10 years of AI-native experience (not a wrapper built on a foundational model) and runs many models across different workflows, each architected to solve unique problems with specific behaviors. There’s no single rule to point to when something goes wrong; no single line of code to adjust – rather, a dynamic system involving experiments and testing, which derives the desired behavior. A model might encounter a use case it thinks it can handle and fall short. That’s the nature of working at the frontier of AI-native systems, and it’s exactly why a robust eval stack has to be embedded throughout the factory, measuring and testing every event and process all the time.

Without strong observability and evals, you cannot achieve or sustain the scale, accuracy and reliability that regulated lending environments require.

The strategy: data-centric, multi-model, customer-transparent

Ocrolus doesn’t pick one model and optimize around it. As an AI-native platform, the approach is to run many best-in-class models across workflows simultaneously. That architectural decision requires an equally deliberate operational one: being data-centric, in addition to infrastructure-centric.

Traditional observability tools answer the question is the system running? We need to answer a different one: is the system delivering? Is data being processed consistently? Are accuracy thresholds holding across data inputs, customer workload populations and edge-case failure modes? Is the turnaround time lagging beyond norms? Is traceability within the data stacks?

This matters for two reasons: 

1. First, this enables proactive issue detection before customers feel any impact. Our systems dynamically adjust workload routing, inference runtimes and serving infrastructure according to the moving customer data populations every minute of every day. But, this also enables early warning for machine-learning (ML) and data science teams to identify growing system faults and back-pressure before they surface in a lender’s workloads. This goes beyond just site reliability engineering (SRE) infrastructure logs; it extends into the data processing layer itself, looking for anomalies created by live data population shifts, model drift, servicing request changes and overall service fluctuation.
2. Second, it creates customer transparency. When processing metrics and confidence scores are surfaced, customers can see what’s happening with their data. When something looks off, it becomes possible to quickly determine whether the issue is on the processing side or the customer’s side: a data shift, a format change or a population skew.

That transparency is the foundation of trust in any regulated environment.

The stack: 4 layers working together

The robustness of the eval system comes from combining four distinct approaches, none of which is sufficient on its own.

1. Human-defined baselines
   Before AI can evaluate quality, humans define what “good” looks like. Golden datasets, carefully curated ground truth, establish the baseline for accurate processing. This human input is what makes everything downstream meaningful. These measures are graphed across the stack:
   • Data
   • Schema
   • Validations
   • Workflows
   • E2E Business Workload
   These human groundtruth-set expectations for accuracy, precision, operating patterns, turnaround time and are updated regularly to capture real-world moving expectations and coverage expansion.
2. AI-native evaluation
   Classic confusion matrices and F1 AUROC scores are combined with LLMs as judges, with competing models evaluating the same output and reaching consensus. This scales evaluation beyond what human review alone can cover, catching issues that deterministic rules would miss. This layer is powered in part by Universal Calibration, an internal system built to consistently benchmark model performance across data types and workflows. Automated feedback for model adjustment is collected in our Automation Gateway, enabling for the dynamic data populations and workflow edge cases to be surfaced and readdressed in our ML/AI deployments.
3. Deterministic validation rules
   Some things are simply true and well-defined in the business logic. Transactions should add up. Extracted fields should fall within expected ranges. These rules provide a consistent, non-negotiable floor for quality that doesn’t drift as models evolve.
4. Targeted evaluation workflows
   Newer models are evaluated against older ones and benchmarked against closed models alongside internal builds. Post-facto audits confirm whether performance holds over time. Every layer of the workflow has explicit checkpoints and thresholds for what success looks like, where tradeoffs are made, and how the larger system orchestration can bend to the workflow customizations needed for each of our customers.

The results: explainability, traceability and consistency

When all four layers work together, the output is something more valuable than accuracy alone: explainability.

Because every step is logged, every checkpoint is explicit and confidence scores are generated throughout the pipeline, shared externally with customers and traced back to exactly where in a data workflow an issue occurred. That audit trail turns a black-box model into an auditable, compliant system.

The other metric that matters is consistency. Not just accuracy on average, but accuracy that holds across different data types, customer applications and data populations. Think of it as bias monitoring at the processing level. The eval stack surfaces where performance varies, enabling proactive identification and setting clear actions to meet lender expectations before issues arrive at their desk.

This is how Ocrolus has reached and maintained greater than 99% accuracy: not through any single technique, but through the combination of human judgment, AI-native evaluation, deterministic validation and continuous workflow-level monitoring, all orchestrated together.

Takeaways

• Observability and evaluation are what separate viable production AI from AI that only works in theory. In lending, greater than 99% accuracy is the threshold, and meeting it requires active infrastructure, not passive monitoring
• Running multiple best-in-class models instead of a single model is a strategic advantage, but it demands a data-centric approach to maintain quality and consistency across all of them
• The four-layer eval stack, combining human baselines, AI-native evaluation via Universal Calibration and LLM-as-a-judge, deterministic validation rules and targeted workflows, is what makes greater than 99% accuracy achievable and sustainable
• Humans remain essential for baselining the process: defining what “good” looks like is still a human job, even as AI takes on more of the evaluation work
• Explainability and traceability aren’t just compliance benefits. They’re operational ones, enabling faster performance tuning, clear customer communication and a system that can identify exactly where and why something needs to be readjusted in a dynamic system
• Consistency across data types and customer populations is as important as average accuracy. The eval stack functions as bias monitoring at the workflow level, surfacing variance before it becomes a customer issue
• This infrastructure isn’t just about maintaining current performance. It’s the foundation for automating continuous improvement at scale, which we’ll cover in depth in part two

FAQ

What is observability in AI systems and why does it matter for lending?

Observability in AI is the ability to monitor and understand what a predictive system is doing at every stage of a workflow, tracking accuracy, consistency, latency and confidence across all models in use. In lending, it matters because predictive models can behave differently and hallucinate results depending on input data, data type or use case — unlike deterministic rule-based systems, which typically have clear failures rather than hidden false positives. Without robust, scalable observability/explainability, there’s no reliable way to know whether accuracy thresholds are being met, where variances originate, or whether performance is consistent across customer populations.

What is the difference between observability and evaluation (evals) in AI?

Observability monitors system behavior in real time. Evals assess whether model outputs meet defined quality thresholds. Together, they create a closed loop: observability surfaces how the system is performing and evals determine whether that performance is good enough. In a multi-model production environment, both are necessary. Observability alone doesn’t tell you if outputs are correct, and evals alone don’t give you real-time visibility into system health.

How does Ocrolus maintain greater than 99% accuracy in production?

Ocrolus maintains greater than 99% accuracy through a four-layer eval system: human-defined ground truth  datasets that establish baseline quality, AI-native evaluation via Universal Calibration and LLM-as-a-judge, deterministic validation rules that provide a hard quality floor and targeted workflows that benchmark models continuously. No single layer achieves that threshold on its own. It’s the consistent combination of all four across every workflow that sustains it at scale.

Why are humans still necessary in an AI-native evaluation system?

Humans are essential for defining what “good” looks like before any model can evaluate quality. At Ocrolus, this takes the form of golden datasets: curated ground-truth examples that set the standard for all model outputs. Without that human-defined baseline, AI-native methods like LLM-as-a-judge have no grounded reference point. As AI takes on more evaluation work, the human role in establishing ground truth as we expand to new use cases remains critical to sustaining greater than 99% accuracy.

What is Universal Calibration and how does it work?

Universal Calibration is an internally built system at Ocrolus that consistently benchmarks model performance across data types and workflows. It enables cross-model comparison, making it possible to identify where performance varies, where newer models outperform older ones and where additional tuning or human review is needed. It is one of several proprietary tools Ocrolus has built in-house as part of its broader eval infrastructure.

What does “LLM as a judge” mean in practice?

LLM as a judge means using large language models to evaluate the outputs of other AI models. Multiple LLMs independently assess the same output and the system looks for consensus. An agreement signals high-confidence convergence; a disagreement flags the output for further review. At Ocrolus, LLM-as-a-judge works alongside human baselines, Universal Calibration and deterministic validation rules as one layer in a four-part eval stack.

How does Ocrolus provide explainability and traceability in its AI system?

Ocrolus achieves explainability through three mechanisms: logging every action throughout the factory workflow to create a complete audit trail, applying explicit business-logic checkpoints and thresholds at each stage so issues can be traced to a specific operating point in the process and generating confidence scores that are shared with customers at the output. Together, these mean that even when a model’s internal reasoning isn’t fully interpretable, the system can pinpoint where something went wrong and provide customers with a defensible record of how their data was handled in a compliance environment.