Architecting a Human-in-the-Loop RAG Platform for Legal AI

The core innovation addressing the trust problem is what we call the SME (Subject Matter Expert) Survey System, which is a three-stage feedback loop that continuously improves AI accuracy through expert verification.

Stage 1: Automated Quality Detection - The system automatically flags potentially problematic answers using two mechanisms:

• Confidence scoring: Vynyl engineered a custom quality scoring system that evaluates each answer across three dimensions: reference sourcing, relevance, and reasoning quality. When the composite score falls below the configured threshold, the question enters an "Unresolved Questions" queue for expert review.
• User feedback: Any answer receiving a thumbs-down rating from users (or anyone else assigned quality control access) gets flagged for review

This creates a quality assurance layer that operates in real-time, catching problems before they get worse.

Stage 2: Expert Review and Verification - Administrators can create surveys from flagged questions and distribute them to subject matter experts with domain expertise. These experts provide verified, correct answers through a secure survey interface. If clarification is needed, answers can be escalated up the expert chain. Administrators review and approve final answers, creating a human verification layer that improves accuracy.

Stage 3: Knowledge Graph Storage - Approved question-answer pairs are stored in AWS Neptune, a graph database that enables sophisticated knowledge management. Unlike a simple relational database storing Q&As in rows, Neptune structures knowledge as nodes (concepts) and relationships (connections between concepts).

This architecture provides two critical capabilities:

• Vector-based semantic search: Neptune Analytics embeds questions as vectors, enabling the AI to find similar questions even when phrasing differs. "Can I take ibuprofen with blood thinners?" and "Is it safe to use Advil while on anticoagulants?" are recognized as semantically equivalent, even though the wording and terminology differ.
• Extensibility for complex relationships: As the knowledge base grows, the graph structure can capture nuanced relationships. A drug might connect to multiple side effects, which connect to related conditions, which connect to treatment protocols. These are all queryable in ways that a traditional database structure would struggle to represent.

The result: the AI doesn't have to focus on exact matches. It reasons about relationships between concepts, drawing on institutional knowledge that gets smarter over time, making the tool more valuable the more it is used.

JAINE operates with two distinct knowledge sources, and the system intelligently routes queries based on context:

Retrieval Augmented Generation (RAG): The AI searches uploaded case documents (settlement agreements, medical studies, court filings) to find relevant passages before generating answers. This grounds responses in actual case materials and provides citation transparency. Users can verify answers by reviewing the source documents.

Expert-Verified Knowledge Graph: For questions that have been previously answered and verified by subject matter experts, the system pulls from Neptune rather than re-processing documents. This provides deterministic, reliable answers for common questions while maintaining the flexibility to handle novel queries through RAG.

The architectural sophistication lies in the decision layer. The system evaluates whether to use document retrieval or expert knowledge, and flags low-confidence responses for future expert review. This creates a continuously improving feedback loop where the most common and critical questions become locked-in, verified answers.

Vynyl also developed JAINE's intake system using a novel approach in legal tech. Rather than presenting claimants with a static 10-page form, the intake system uses conversational AI to guide potential claimants through eligibility screening.

The system dynamically adapts questions based on responses. If a potential claimant indicates they didn't take the medication in question, the conversation short-circuits immediately rather than wasting time on irrelevant questions. Users can ask clarifying questions in natural language if they don't understand medical or legal terminology. At the end of the process, qualified claimants can digitally sign representation agreements.

This creates value on a couple fronts: it functions as both a marketing tool (reducing friction in the lead capture process) and an automated screener (filtering out unqualified claimants before they consume lawyer time). If the law firm deploys this to 10,000 previously uncontactable leads, potential claimants who didn't respond to phone outreach but might engage with a lower-friction digital process.

Vynyl delivered the MVP (including multi-tenant infrastructure, AI chatbot functionality with AWS Bedrock integration, and knowledge base management) in four months from kickoff. The platform launched to initial clients in June 2025 and has been in continuous feature development since, with new capabilities funded by specific client needs as the platform scales.