AI Safety & Guardrails Architecture

Prompt injection is the most critical vulnerability in LLM applications: an attacker embeds instructions in user input that hijack the model's behavior. Direct injections ('ignore previous instructions and...') are easy to detect, but indirect injections are far more dangerous — malicious instructions hidden in retrieved documents, emails, or web pages that the LLM processes. Defense strategies operate at multiple levels. Input classifiers (fine-tuned BERT/DeBERTa models) detect known injection patterns with 95%+ accuracy but struggle with novel attacks. Instruction hierarchy (Anthropic's approach) trains the model to prioritize system prompts over user inputs. Prompt sandboxing isolates user-provided content in clearly delimited sections with explicit instructions to treat it as data, not commands. Canary tokens — unique strings injected into the system prompt — detect when the model has been manipulated into revealing its instructions. No single defense is sufficient; production systems layer all of these together.

Constitutional AI (CAI), developed by Anthropic, replaces the expensive human labeling step in RLHF with a set of written principles (a 'constitution'). The process has two phases. In the supervised phase, the model generates responses, then critiques and revises its own outputs based on the constitution ('Is this response harmful? If so, rewrite it to be helpful while avoiding harm'). In the RL phase, the revised responses train a reward model that is then used for reinforcement learning, replacing human preference labels. The constitution typically includes 15–20 principles covering helpfulness, harmlessness, and honesty. CAI's key advantage is scalability: writing principles is far cheaper than labeling thousands of output pairs. The approach also makes alignment decisions transparent and auditable — you can inspect and modify the constitution. However, CAI can be overly conservative, refusing legitimate queries that superficially resemble harmful ones, requiring careful principle tuning.

Output safety classifiers evaluate generated text before it reaches the user. Toxicity classifiers (e.g., Perspective API, OpenAI Moderation endpoint, Meta's Llama Guard) detect harmful content across categories: hate speech, violence, sexual content, self-harm, and dangerous instructions. Hallucination detectors compare generated claims against retrieved source documents, flagging unsupported statements — approaches include NLI (Natural Language Inference) models that classify each claim as 'supported', 'contradicted', or 'neutral' relative to sources. Fact-checking pipelines decompose outputs into atomic claims and verify each against a knowledge base. The critical design decision is the false positive rate: an aggressive classifier blocks harmful content but also blocks legitimate queries about sensitive topics (medical, legal, security research). Production systems typically use tiered thresholds — strict for consumer-facing applications, relaxed for enterprise/research contexts — with human review queues for borderline cases.

Red-teaming proactively discovers vulnerabilities before attackers do. Manual red-teaming uses domain experts who craft adversarial prompts across categories: jailbreaks (bypassing safety training), prompt injection (hijacking behavior), information extraction (leaking training data or system prompts), and bias exploitation. Automated red-teaming uses LLMs to generate adversarial attacks at scale — one model attacks while another evaluates success. Frameworks like Microsoft's PyRIT and NVIDIA's NeMo Guardrails provide structured attack libraries covering 200+ known attack vectors. Gradient-based attacks (GCG — Greedy Coordinate Gradient) automatically find adversarial suffixes that bypass safety training, though these are computationally expensive. The red-team cycle is continuous: new attacks are discovered, defenses are updated, and the red team tests again. Organizations typically run red-team exercises before every major model or prompt change.

LLM applications process user inputs that frequently contain personally identifiable information: names, emails, phone numbers, SSNs, medical records, and financial data. PII detection operates at both input and output stages. Input-side PII detection identifies sensitive data before it reaches the model, either redacting it (replacing with tokens like [EMAIL]) or encrypting it for later reconstruction. NER (Named Entity Recognition) models fine-tuned for PII achieve 97%+ recall on structured PII formats (emails, phone numbers) but struggle with contextual PII ('my neighbor John told me...'). Output-side DLP (Data Loss Prevention) prevents the model from regurgitating training data containing PII — a known risk with large language models that memorize rare sequences. Regex-based detection catches structured formats while transformer-based classifiers handle contextual PII. Differential privacy during training provides mathematical guarantees against memorization but reduces model quality. Production systems combine all approaches with audit logging for compliance with GDPR, HIPAA, and CCPA.