Titan Architecture

System Architecture

Titan is cloud-agnostic, built on a microservices architecture that leverages containerization with Docker and orchestrated through managed container platforms and Kubernetes. The system is designed to be secure, scalable, resilient, and maintainable.

Infrastructure Components

API & Compute

Databases

Dgraph AKS Deployment

Frontend & CDN

Compute Resources

Container Apps Services

All Container Apps run inside the virtual network via Container Apps VNet integration and pull images from a private Azure Container Registry repository.

AKS Cluster (Dgraph)

The AKS cluster uses Workload Identity, Calico network policies, Managed Prometheus, and Trusted Launch nodes. Maintenance window is Sundays 2–10 AM ET.

AI Inference Cluster (AKS)

All AI model serving runs on a dedicated AKS cluster with autoscaling GPU node pools, exposed through an Application Gateway with WAF (Prevention mode) — TLS and authentication enforced at the gateway.

The cluster runs 5 NVIDIA H100 GPUs always-on (scaling to 8 under load): full GPUs serve primary LLM inference, while MIG-partitioned GPUs host embedding, reranking, NER, and vision services on dedicated slices.

Services hosted on the inference cluster:

Defense in Depth — Security Layers

Titan implements security at every layer of the stack — from the network edge through the AI response. No single layer is trusted in isolation; each reinforces the others.

Layer 1: Edge & Network

Layer 2: Cloud Infrastructure

Layer 3: Application Security

Layer 4: Code & Supply Chain

Layer 5: Data Protection

Layer 6: AI & LLM Security

This layer addresses threats unique to AI-powered applications — prompt injection, hallucination, data leakage, and model misuse.

Layer 7: Monitoring & Incident Response

Ingestion Architecture & Data Flow

High-Level Ingestion Overview

The ingestion process consists of several key components:

Ingestion Data Flow Process

1. Discovery: The Orchestrator triggers the Discovery Worker for each configured connector
   1. The worker crawls the source system (e.g., HelpJuice API, Blob Storage container)
   2. Discovered documents receive a stable ID derived from client name, connector type, and source URI
   3. Metadata is extracted (e.g., source URL, fetch timestamp, content type)
   4. Permissions are read from the source (if applicable) or from a predefined configuration
   5. A fetch task is published for each discovered document
2. Fetch: The Fetch Worker retrieves the document content
   1. Content is downloaded from the source connector
   2. The document is uploaded to Blob Storage with an auto-generated path
   3. A content hash (SHA-256) is calculated for deduplication
   4. An ingestion task is published with the Blob Storage reference and metadata
3. Deduplication: The Ingestion Worker checks for existing copies before processing
   1. Content hash is compared against existing documents in the graph (by source ID and connector)
   2. Four outcomes: Skip (unchanged), Update (hash differs, replaces existing), Similar (content similarity detected), or Create (new document)
   3. Duplicate documents are skipped to avoid unnecessary reprocessing
4. Malware Scanning: The document is scanned for threats before parsing
   1. Cloud-native malware scanning (Microsoft Defender for Storage) validates the uploaded file
   2. The worker waits with exponential backoff retries until the scan completes
   3. Infected files are quarantined and rejected from the pipeline
5. Document Parsing: The Document Parser (Docling) processes the document
   1. Content is parsed based on content type (e.g., PDF, HTML, DOCX, XLSX)
   2. Text extraction is performed with semantic chunking and page/section offsets
   3. Additional metadata is extracted (e.g., title, author, keywords)
   4. Tables are extracted and handled separately for structured data preservation
6. PII Detection: Extracted text is scanned for sensitive information
   1. If Tier 0/1 PII is detected, the document is rejected per compliance policies
   2. Tier 2 PII is flagged but allowed through by default
7. Embedding & Storage: Processed chunks are prepared for the knowledge graph
   1. Embeddings are generated for each chunk using a local LLM for semantic search
   2. Chunks are batch-inserted into the dgraph graph database with metadata and embeddings
   3. The document Record node is created with source metadata and access control information
8. Enrichment (parallel, post-ingestion): The Enrichment Worker extracts structured knowledge
   1. Entities (people, organizations, regulations, etc.) are extracted from chunks via a local LLM
   2. Entity names are normalized and deduplicated using canonical keys and fuzzy matching
   3. Relationships between entities are identified and stored with confidence scores
   4. Entities and relationships are linked into the knowledge graph
9. Summarization (parallel, post-ingestion): The Summarization Worker generates a document summary
   1. A concise summary is generated from the document's text chunks using a local LLM
   2. The summary is stored on the document's Record node in the graph
   3. The document is marked as ready for agent queries

PII Detection & Rejection Policies

• No PII is sent to external LLMs.
• PII Rejection rules are configurable.
  • By default, Tier 0 PII is rejected (Credit Card Numbers)
  • Tier 1 PII (SSN, IDs, etc) are rejected in the chat experience and masked in the agents.
  • Tier 2 PII (Full Names, Email Addresses, Phone Numbers, etc) masked.
• Local SLMs and BERT models are used in the ingestion pipeline to avoid sending sensitive data to external services.
• PII Detection uses a fine-tuned GliNER model to identify sensitive information.
  • GliNER Paper

Default PII Tier Classifications

Knowledge Graph Structure

Record Nodes

• Each ingested document is represented as a Record node in the graph.
• Record nodes contain metadata such as:
  • source: URL or file path of the document
  • timestamp: Ingestion timestamp
  • summary: Generated summary of the document
  • permissions: Access control information

Chunks

• Documents are broken into smaller Chunk nodes for efficient retrieval.
• Each Chunk node contains:
  • text: The text content of the chunk
  • embedding: Vector representation for semantic search
  • metadata containing context such as page number, section title, etc
  • keywords: Extracted keywords for indexing

Entities and Relationships

• Named entities (e.g., people, organizations, locations) are extracted and represented as Entity nodes.
• Relationships between entities and documents are captured using edges, enabling complex queries and insights

Query Architecture & Data Flow

High-Level Query Overview

1. User submits a query via the Titan UI
2. PII Detection scans the query for sensitive information
3. Query is tested for possible prompt injection or model hacking attacks
4. The Titan Agent processes the query...
   1. Attempts to understand the prompt.
   2. What is the user asking for?
   3. What data sources are relevant?
5. If a search of the Knowledge Graph is required:
   1. Unmask any PII in the query
   2. Search the Knowledge Graph
   3. Check user permissions for each result
   4. Rank results based on relevance
   5. Mask any PII in the results
6. The Titan Agent generates a response using the retrieved documents and LLM
7. Response is unmasked for PII and returned to the user
   1. All interactions are logged for auditing
   2. Chain-of-Thought reasoning is stored and used to provide citations and explainability
   3. Answer and citations are displayed in the UI

Explainability & Auditing

Titan is built on the principle that every AI-generated answer must be traceable, verifiable, and auditable. Users should never have to take an answer on faith — they should be able to see why the system answered the way it did, what sources it used, and how confident it is.

Citations

Every answer includes citations linking claims back to specific source documents. Citations are not simply appended — they go through a multi-stage verification process to ensure they are genuine and relevant.

Citation Verification Process

1. Evidence Extraction: After the agent generates a response, all evidence retrieved from the knowledge graph during the query is collected (search results, table queries, etc.)
2. Score-Based Filtering: High-relevance evidence (score >= 0.85) is automatically included. Lower-scoring evidence proceeds to LLM verification.
3. LLM Verification: A local LLM evaluates each piece of lower-scoring evidence against the generated answer to determine if the evidence was actually used. For each piece of evidence, the verifier produces:
   • Whether facts from the evidence appear in the answer
   • Which specific facts match (up to 3)
   • A reason for inclusion or exclusion
4. Citation Grouping: Verified evidence is grouped by source document, producing a deduplicated set of citations with metadata (document title, page number, section heading).
5. Citation Propagation: When the agent answers from prior conversation context without making a new knowledge graph search, a fallback propagation step finds relevant citations from previously-retrieved chunks using vector similarity and keyword overlap filtering.
6. Source Resolution: Each citation is resolved to a viewable document — generating secure, time-limited URLs for stored documents and navigation hints (page number, highlight text, section) so users can jump directly to the cited passage.

Confidence Scoring

Each response includes a confidence indicator based on citation strength:

Chain-of-Thought Reasoning

The agent's decision-making process is captured at each step — what tools were called, what the agent was thinking, and how it arrived at the final answer. This chain-of-thought reasoning is stored alongside the response and will be available to users, providing full transparency into the agent's reasoning process.

Audit Logging

• All ingestion and query activities are logged with detailed metadata
• Logs include timestamps, users, documents accessed, and tools invoked
• Correlation IDs trace each request end-to-end across the pipeline (API, worker, agent, knowledge graph)
• Citation verification decisions are logged — including which evidence was included, excluded, and why

Advanced Explainability

Beyond citations and chain-of-thought, Titan is actively investigating techniques from the field of Explainable AI (XAI) to give users deeper insight into why the system produced a particular answer and how much each piece of evidence contributed.

Shapley Values for Feature Attribution

Shapley values, originally from cooperative game theory, provide a mathematically principled way to attribute an outcome to individual contributors. Applied to Titan's agent pipeline, Shapley values would quantify each retrieved document's contribution to the final answer — answering the question: "If this document hadn't been retrieved, how would the answer change?"

This goes beyond binary citation (cited vs. not cited) to provide a continuous importance score for each source, helping users understand which documents were most influential and which were merely corroborative.

LIME (Local Interpretable Model-Agnostic Explanations)

LIME generates human-readable explanations by approximating model behavior locally around a specific prediction. In Titan's context, LIME would explain individual agent responses by perturbing inputs (e.g., removing paragraphs from retrieved context) and observing how the answer changes.

This would surface explanations such as: "The answer focused on Q3 revenue primarily because of paragraphs 2 and 5 from the quarterly report — removing them shifts the answer toward Q2 data."

Attention Visualization

Attention visualization surfaces which parts of the input the model focused on when generating each part of its response. Titan is exploring exposing attention patterns from the local small language model (SLM) using libraries such as BertViz and TransformerLens, as well as token-level log probabilities from cloud LLM providers.

The goal is an interactive view where users can see which words and passages in the source documents the model attended to most heavily, providing an intuitive visual explanation of the reasoning process.

Counterfactual Explanations

Counterfactual explanations answer the question: "What would need to be different for the answer to change?" For example: "If the policy document had not mentioned the 30-day exception, the agent would have answered that no exceptions exist."

This technique helps users build trust by making the boundaries of the system's reasoning visible — showing not just what the answer is, but what conditions would produce a different one.

Correctness

Titan uses RAGAS (Retrieval Augmented Generation Assessment), an industry-standard framework for measuring the quality and accuracy of AI generated answers. Evaluation metrics ensure that responses are not only relevant but also grounded real data.
RAGAS evaluations are run internally on a regular basis to monitor system performance and identify areas for improvement.

Key Quality Metrics

Faithfulness Measures whether the answer is supported by the retrieved documents. A high faithfulness score means Titan isn't "making things up" — every claim in the response can be traced back to your source data. This is critical for compliance and trust.

Example: If Titan says "Q3 revenue was $4.2M," that figure must appear in one of the retrieved documents.

Answer Relevancy Evaluates how well the response actually addresses the user's question. A high score means Titan understood what was being asked and provided a focused, on-topic answer rather than tangential information.

Example: If a user asks "What are our data retention policies?", the answer should directly address retention — not general security policies.

Context Precision Measures whether the documents retrieved to answer the question were actually relevant. High precision means Titan is pulling the right documents, not cluttering its reasoning with unrelated content that could dilute answer quality.

Context Recall Evaluates whether Titan found all the relevant documents needed to fully answer the question. High recall means important information isn't being missed, reducing the risk of incomplete answers.

How We Use These Metrics

• Continuous Monitoring: Quality scores are tracked over time to identify degradation or improvement.
• Threshold Alerts: Teams are notified if quality metrics fall below acceptable thresholds
• Feedback Loop: Low-scoring responses are reviewed to improve retrieval and ranking algorithms
• Reporting: Monthly quality reports provide visibility into system performance