Capstone: Secure an AI-Powered Helpdesk

Capstone brief

Design a secure AI-powered helpdesk that can answer internal policy questions and create tickets. Produce a short threat model, a control plan, and an evaluation plan that show how the system remains useful without gaining unsafe authority.

Capstone success criteria

• Every data source, tool, and model action has a named owner and trust boundary.
• The design applies least privilege, retrieval access controls, and approval gates for consequential actions.
• The evaluation plan includes ordinary use, edge cases, direct/indirect injection attempts, and a human review process.
• Logs support incident investigation without retaining unnecessary sensitive content.

Watch: testing secure AI systems

• — a concise discussion starter for the capstone risk register.
• — revisit with an eye for governance, assurance, and measurable outcomes.

Read: governance and assurance

• NIST AI RMF — use the Govern, Map, Measure, and Manage functions to structure the design review.
• NIST Generative AI Profile — a companion profile for generative-AI risks.
• Microsoft Learn: Operationalise security and safety evaluations — examples of adversarial test cases and continuous evaluation.

Book shelf

• Threat Modeling, Adam Shostack — a strong structure for explaining your design choices.
• AI Engineering, Chip Huyen — helpful for moving from prototype to accountable production operation.