Applied AI
Retrieval-Augmented Auto-Justification for Compliance Assessments
July 2026 · 5 min read
Compliance assessment is, at its core, an exercise in justified judgment. Someone reads a requirement — a control from ISO 27001/27002, or a clause from whichever framework the organization has committed to — and then works through policies, procedures, and system records to answer a deceptively simple question: have we actually implemented this, and how well? Done honestly, that answer is expensive. The evidence is scattered across documents that were never written with an assessor in mind, and the conclusion has to survive scrutiny from an auditor who was not in the room when it was made.
On the cybersecurity compliance platform I contributed to, we built a feature that drafts that judgment automatically. A user selects a requirement from any framework loaded into the platform, and the system produces a proposed assessment: a written justification, a maturity rating, citations to the organization’s own source documents, and an actionable recommendation. I describe the pattern as retrieval-augmented auto-justification, and building it taught me more about the relationship between AI and auditability than any abstract debate about the two.
The interesting part was never that a language model can write a plausible paragraph about compliance. It can. The interesting part is everything around the model that turns a plausible paragraph into an assessment an organization is willing to defend.
Grounded in the organization’s own evidence
An assessment is only meaningful if it reasons from what the organization has actually done, not from what organizations in general tend to do. So the feature is built on retrieval-augmented generation (RAG) over the organization’s own context: every document uploaded to the platform — including scanned documents brought in through OCR — and the structured data teams enter into the platform itself is chunked, embedded, and indexed for retrieval.
That index is strictly per-tenant. Each organization’s evidence lives in its own isolated index, and nothing retrieved for one tenant can ever inform an assessment for another. It is a relevance decision and a security decision at the same time — fitting, because the platform’s whole premise is that those are rarely different decisions. Background reindexing keeps the index current as company data changes, so an assessment reflects the organization as it is now, not as it was at the last upload.
Retrieval quality is assessment quality
When an assessment runs, the selected requirement drives a vector search across that tenant’s index, and the retrieved passages are re-ranked with MMR (maximal marginal relevance). That re-ranking step matters more than it sounds: without it, retrieval tends to return several near-identical fragments of the strongest-matching policy, and the model ends up assessing one document instead of the organization. MMR trades a little raw similarity for coverage, so the context the model reasons over spans distinct pieces of evidence.
A structured verdict, not free text
The model’s answer is never free-form prose. It is a structured JSON output with defined parts:
- A written justification of the implementation judgment.
- A maturity status on a seven-level scale: Nonexistent, Initial, Limited, Defined, Managed, Optimized — plus Not Applicable.
- Citations of the source documents the judgment rests on.
- An actionable recommendation for closing the remaining gap.
Fail loudly, never silently
That structure is enforced, not requested. Every response is validated strictly against the expected shape, and a response that does not conform surfaces as an explicit error — never a silent fallback that quietly ships a half-formed assessment. In a compliance context this is the only defensible choice: a wrong answer that fails loudly is a bug you can fix; a wrong answer that passes quietly is a governance failure you may not discover until an audit does.
The same discipline applies to quality over time. An eval harness scores retrieval and answer quality, so a change to chunking or ranking is judged against evidence rather than against whether a demo looked convincing that afternoon. Evaluating the system the way an auditor would evaluate a control ended up shaping most of our other decisions.
Human oversight by design
Two guardrails carry most of the governance weight. The first: the Not Applicable status can only be set by a human. The AI is not allowed to conclude that a requirement does not apply to the organization, because declaring something out of scope is exactly where automated judgment would be most dangerous — it removes a control from consideration entirely.
The second is a deliberate separation between narrative and traceability. Justifications are written without referencing file names, so the prose never leaks the internal structure of an organization’s document store — while the source citations are recorded separately alongside every assessment. A reviewer keeps full traceability of what the judgment was based on; the narrative stays clean enough to share.
None of this required exotic technology. It required treating an AI feature the way the rest of the platform treats every control: with defined ownership, explicit failure modes, and evidence you can trace. That is the thesis I keep returning to across my work — a technical decision is also a security decision, and a governance decision. AI-assisted assessment, built under that discipline, does not erode auditability. It hands the assessor a draft judgment with the evidence already attached, and keeps the decisions that must stay human, human.