Threat Modelling an AI Assistant

Scenario

You are reviewing an AI assistant that can search an internal knowledge base and call a ticketing tool. Your job is to identify assets, trust boundaries, attack paths, and controls before it goes live.

Learning outcomes

• Map the model, prompt, retrieval, identity, data, and tool boundaries in an AI application.
• Identify direct and indirect prompt injection, data poisoning, excessive agency, and sensitive-data exposure.
• Propose layered controls: least privilege, input handling, output validation, logging, testing, and human approval.

Watch: attack and defence perspectives

• — a practitioner’s perspective on testing AI attack surfaces.
• — a focused primer on untrusted content influencing an agent.

Read: practical threat-modelling references

• OWASP Top 10 for LLM Applications — use this as a risk checklist during the exercise.
• Microsoft Learn: Security planning for LLM-based applications — concrete examples of prompt injection, least privilege, and adversarial testing.
• MITRE ATLAS — map potential behaviours to a shared adversarial-ML vocabulary.

Book shelf

• Threat Modeling, Adam Shostack — apply its structured questions to models, data stores, and tools.
• AI Engineering, Chip Huyen — useful background on designing production foundation-model applications.