What is an AI safety model?

A second set of eyes

An AI safety model is a specialized model that helps detect or evaluate risks in another AI system. It might inspect a user’s request, classify a generated answer, or review a proposed action. The main model does the work; the safety model contributes another judgment about that work.

Safety in a model, and a model for safety

These are related ideas. Safety in a mainstream assistant includes its training, behavior rules, and product controls. A specialized safety model has a narrower job, such as flagging a risky response. One does not replace the other, and neither guarantees that an application is safe.

What happens in practice?

Imagine an assistant that reads an invoice and proposes paying it. Input checks can look for suspicious instructions in the invoice. A policy check can examine the proposed payment. A permission system can require your approval. An activity log records what happened. Only some of these steps need a model.

A safety model can make mistakes too

A detector may flag harmless content or miss a risky request. Its usefulness depends on the task, language, policy, deployment, and cost of errors. Evaluate the entire workflow with realistic examples, including ambiguous and adversarial cases.

Different checkers answer different questions

A content moderator asks whether a prompt or answer falls into a policy category. An attack detector asks whether text is trying to redirect instructions. A groundedness judge asks whether an answer follows supplied evidence. These are different tasks: passing one check does not imply passing the others. Meta’s Prompt Guard documentation is a concrete example of a detector whose malicious label concerns instruction attacks rather than all harmful content.

Who decides what a flag means?

A model can flag an item; a product still needs a decision rule. A community forum might send an ambiguous post to review, while a file-sharing tool might stop an external transfer. The same predicted risk can justify different responses because the action, people affected and options for correction differ. Our view is that this decision belongs in an explicit product policy, with someone responsible for revising it.

Evaluate usefulness as well as blocking

XSTest investigates exaggerated safety behavior by pairing safe requests with unsafe contrasts. That is a useful reminder to test legitimate use alongside misuse. A checker that blocks everything may avoid one kind of failure while making the product unusable. The paper’s historical results should not be read as a ranking of today’s assistants.

Questions to take with you

• Name the risk the checker is meant to detect.
• Find out what happens after a flag: block, review, clarification or logging.
• Ask for examples of both missed risks and legitimate requests wrongly stopped.

Keep exploring