Validating Clinical Documents at Scale

Every prior authorization comes with a hidden answer key. Miss a clinician's signature or a single diagnosis code on a document, and the insurance company denies the request, leaving the patient waiting longer for care. These requests are built from clinical documents: notes, treatment plans, and reports that justify the care. Providers submit this clinical information through web portals and over phone calls. In this post, I'll share how we at Silna validate those documents at scale and catch problems before the insurance company ever sees them.

Denials, Denials, Denials

The unfortunate truth: oftentimes, healthcare delayed is healthcare denied.

To justify a prior authorization, insurance companies (or "payors") require supporting documentation, and the requirements for these documents can get oddly specific:

1. A patient's name and insurance policy ID must be in the header of every single page
2. A treatment plan must have a specifically labeled x-axis on its graph
3. A school schedule must have to-the-hour granularity for when a patient is in the classroom

Miss one, and the prior authorization can be denied.

Here's what a few of those requirements look like on a single document. Click a pin to see why each one matters:

These checks are tedious for a human but easy for an LLM. Today's multimodal LLMs reason over long reports full of images and charts, so - given a clear checklist - they validate these checks well.

The catch is scale. Every payor has a different checklist, and - even within one payor - it varies by specialty, state, and plan. To make matters more complicated, those checklists keep changing as payors grow stricter.

Our product Document Validation solves this need by integrating a comprehensive database of payor requirements with the ability to validate the requirements against the clinical documents we process using LLMs. We uncover the answer key for each payor and use AI to automatically check these documents before submission, preventing denials and other delays in the approval process. That means patients get their care quicker, and providers can ensure the payor properly compensates them.

In this post, we'll dive into how we built our rules engine from the ground up, how anyone can configure their rules in natural language, and where we see the space going.

Rules and How We Configure Them

Our Rules Engine

Formally, we define a rule R as a boolean predicate that evaluates to True or False. For example, a rule can be that the word "Silna" starts with the letter "S", that today is a Wednesday, or that a particular document includes a clinician's signature. As we can see, each of these rules evaluates to either True or False. And, with Document Validation, we can use LLMs to evaluate these rules against documents.

To power Document Validation, we built a rules engine which has two primary modes - rule configuration and rule evaluation. We first configure a certain rule; then, at a later point, we evaluate this rule in the context of other data (e.g., documents).

For instance, let's imagine we configure the rule R₁ from above as "the input document has a clinician's signature". If the input document D₁ is indeed signed by a clinician, then R₁ is True upon evaluation.

Here is that single rule live. Toggle whether the document is signed and see what R₁ evaluates to:

Default and Custom Rules

Rules come in two flavors, and we maintain an ever-growing bank of both.

Default rules ship with the product and are implemented directly in code. We reach for these when a check has clear, deterministic logic that we would rather not hand to an LLM, like comparing today's date against a document's expiration date. A few we support today:

• Document expiration check
• ICD-10 diagnosis code check

On the other hand, custom rules are written in our app in plain natural language. The idea is that adding a new kind of check shouldn't require any engineering, so anyone can write one. We don't have a default rule for whether a document lists the patient's age, for instance, so you would write a custom rule as a short prompt: "does the document contain the patient's age?" The LLM answers Yes or No, which we map to True or False.

Combining Rules into Configurations

Individual rules are useful on their own - but the real power comes from composing them with ANDs and ORs. Our rules engine supports this out of the box! We call a set of rules joined this way a Document Validation Configuration, and it evaluates to True only when every rule passes.

Below is one such configuration live. It checks that a document has a clinician's signature, is not expired, and carries an approved ICD-10 code. Toggle the document's attributes or flip a connector to see the result change:

Gating Logic: Configuration Processes

We want our rules to be flexible. To scale our configuration abilities for the N+1th payor, and to avoid reconfiguring the same rules over and over again, we treat Document Validation Configurations as first-class objects. For example, we can encode a set of rules once as configuration C₁ (like above) and then reuse it for different payors.

We can also encode gating logic based on document type, a common use case we see. For example, C₁ composed of a set of 5 rules might apply for referral documents, but we can also configure C₂ composed of a set of 4 additional, but different, rules for diagnosis report documents.

We refer to this higher-order gating logic as a Document Validation Configuration Process. Formally, a Document Validation Configuration Process is a mathematical graph G = (V, E), where the vertices are Document Validation Configurations and the edges are defined by conditions that gate whether or not those configurations apply. For instance, an edge may be a particular document type. G has a single source, its root, and an execution of a process is defined as a single path in the graph.

For a particular document, we have enough information to traverse the graph - every node (or configuration) in the path contains rules relevant to that document. We then collect all of the configurations, AND their rules together, and use our rule engine to evaluate the rules.

Pick a document type and trace its path:

Self-Serve Rules for Providers

Finally, we empower our providers to create their own rules. This gives them the ability to write their rules as natural language prompts, combine rules together, and see how they perform on their own documents. Providers can convert their intake SOPs into checks that run automatically on every one of their prior authorizations. They can specify which types of documents to run the rules on and choose the payors and specialties to which they are relevant.

Try writing a rule yourself and see how it works on these three documents:

Payor Intelligence

Custom rules, our rules engine, and multimodal LLMs together let us validate documents against arbitrarily complex requirements, across both text and images. We can ask questions of long documents full of dense charts and graphs and get reliable answers.

Behind the rule bank sits an always-current database of payor requirements, maintained by our Payor Intelligence team. They track payor bulletins and mine the thousands of prior authorizations we process each week for signal. In Autism Therapy, for instance, we handle more prior authorization volume than any single provider, which gives us an unparalleled view of what payors actually check.

For providers, this means the rules stay current on their behalf. As payors grow stricter and we encode new requirements, every provider's checks update automatically.

Efficient and Durable Inference

So far, we've covered the what: what rules exist and how they're configured. Now, let's look at the how: the inference pipeline that evaluates those rules against real documents.

Extracting Structured Data with Pydantic

To reason about a document, we extract its fields into a Pydantic schema. A clinician schema, for example, might look like this:

class Clinician(BaseModel):
    first_name: str      # the clinician's first name
    last_name: str       # the clinician's last name
    credentials: list[str]  # e.g., MD, PhD, PsyD, etc.
    has_signature: bool  # handwritten or electronic signature

Both default rules and custom rules evaluate against a schema like this; they just build it differently. For a default rule, we define the schema and its evaluation logic in code: the clinician signature rule simply checks that has_signature is True. For a custom rule, we generate the schema on the fly from the natural-language prompt and let the LLM populate it.

Either way, type checking enforces that every rule resolves to a boolean, so it cleanly passes or fails. For custom rules, that strict schema and forced type casting also guard against prompt injection.

Chunking Long Documents

Long documents pose a challenge for this extraction. We know that long documents degrade the context of LLMs since more input tokens worsen the performance of the models. To remedy this, we separate long documents into smaller chunks, run inference on them independently, and then use another LLM to synthesize the chunks into a single schema before evaluating the rules.

Bringing It All Together - The Prior Authorization Product Experience

We have our robust bank of rules as well as a detailed understanding of payor requirements. With this, and with Document Validation, we can now check our clinical documents as part of the prior authorization submission process.

In Silna, we create prior authorizations by populating some basic information about the patient and uploading supporting clinical documentation. From there, we use our rules engine above to determine which rules apply to which documents. We run inference on these documents, evaluate the rules, and return the results in the UI. At this point, providers can review their documents and decide whether to update our documents or proceed anyway.

Towards Longer Running Document Agents

As mentioned above, Document Validation is grounded in LLM inference. However, at this moment, we leverage one-shot LLM calls to extract the relevant data from the documents via Pydantic schemas. We are starting to encounter scenarios, though, in which a single LLM call is not sufficient.

For example, for documents that are over 100 pages or those that are filled with lots of handwritten notes, our models diminish in accuracy. We can't use heavier models because those are slower, and Document Validation is a real-time flow - meaning we need to ensure the latency is low enough to allow for fast user experiences.

We are exploring ways to remove the latency constraint by offloading Document Validation to run in the background. Instead of checking documents during the prior authorization creation flow, we can validate them in the background and notify the provider via email once results are ready. The provider reviews and submits, all within 15 minutes or less, replacing a tedious manual review that can take an intake specialist much longer.

This will allow us more grace in the latency of our processing; in fact, we can opt to use document agents (multi-turn loop with tool calls), which we have found to be even more accurate. Finally, another benefit of Document Validation using agents is that we can begin to consider even more complicated rules, involving questions of medical necessity or those that piece together information from disparate places in a long document.

One-shot calls and background agents trade off between speed and accuracy. Toggle the two approaches:

The Bigger Picture

Document Validation ties these threads together: a rules engine that anyone can configure in plain language, a continuously updated picture of what each payor demands, and inference that holds up across long, messy clinical documents. The result is that requirements get checked before a prior authorization ever reaches a payor, so denials are caught at the source rather than weeks later.

As we push toward longer-running document agents and more nuanced rules, the goal stays the same: every requirement met the first time, so patients get their care faster.