AI Safety

AI safety is the field of research and engineering that aims to keep artificial intelligence (AI) systems from causing harm, whether through accidents, deliberate misuse, or systems pursuing goals their developers did not intend. It covers deployed machine learning systems, such as the perception stack of an autonomous vehicle or a customer-service chatbot, as well as the more capable frontier AI models and agents that labs are building. The work spans technical research, testing practices, and the frameworks companies and governments use to decide when a system is safe enough to release.

Core Problems in AI Safety#

The 2016 paper Concrete Problems in AI Safety, by researchers from Google Brain, OpenAI, Stanford, and UC Berkeley, framed accidents in machine learning as five research problems, illustrated with a cleaning robot, that still organize much of the field:

• Avoiding negative side effects: A robot rewarded only for a clean floor has no reason to avoid knocking over a vase on the way.
• Reward hacking: Satisfying the letter of an objective while defeating its purpose, such as covering messes with material the robot's camera cannot see through.
• Scalable oversight: Training good behavior when checking every action is too expensive, such as asking a human how clean the room is only occasionally.
• Safe exploration: Letting a learning agent try new actions without trying catastrophic ones, such as putting a wet mop into an electrical outlet.
• Robustness to distributional shift: Recognizing inputs unlike the training environment and behaving cautiously on them, such as a robot trained in an office moving to a factory floor.

Each problem became a research area of its own. Scalable oversight became a core line of superalignment research, the study of supervising systems more capable than their supervisors, safe exploration is a constraint on reinforcement learning, and distributional shift appears in production systems as data drift.

Where AI Risks Come From#

Google DeepMind's 2025 paper An Approach to Technical AGI Safety and Security sorts serious risks into four areas, distinguished by who or what is responsible for the harm:

Each area calls for different defenses. Misuse is countered by refusal training, AI guardrails, and restricted access to dangerous capabilities. Misalignment is difficult to test for, because a misaligned model may behave well while it is observed: studies have found alignment faking during training, sandbagging on evaluations, backdoored sleeper agents that survive safety training, and emergent misalignment after narrow fine-tuning. Mistakes are reduced through robustness, testing, and monitoring, and structural risks, such as harms that emerge from competition among many developers or from interactions among many AI agents, are mainly a matter of governance.

Methods in Practice#

Safety work runs through the whole lifecycle of a model. During training, reinforcement learning from human feedback (RLHF) and constitutional AI shape what a model will and will not do. Before release, AI red teaming and dangerous-capability evaluations probe for failures, and mechanistic interpretability inspects internal computation for goals that outputs alone would not reveal. Frontier safety frameworks, such as Anthropic's Responsible Scaling Policy, define capability thresholds at which a model requires stronger security and deployment safeguards.

After deployment, model monitoring catches the inputs a model was never tested on. Ultralytics Platform deployment monitoring charts the confidence distribution and the mean confidence over time for a deployed YOLO26 endpoint, so a drop after a camera is moved or the lighting changes flags inputs drifting away from the training data. Systems that act in the physical world also need a fallback, such as the minimal risk condition defined for automated vehicles, a safe stop the vehicle reaches when its automated driving system can no longer continue.

AI Safety vs. Related Fields#

AI safety overlaps with several fields that are often used interchangeably with it:

• AI ethics: Asks what AI systems should do and for whom, covering fairness, privacy, and accountability. Harms such as biased decisions fall under both fields.
• AI security: Protects AI systems from attackers, through defenses against adversarial examples, data poisoning, prompt injection, and model theft.
• AI alignment: The subfield of AI safety concerned with making a system pursue the goals its developers intended, which addresses the misalignment area above.
• AI governance: The laws, standards, and internal policies that decide how AI systems are built and released, such as the risk tiers of the EU AI Act.

The boundaries move with policy: in February 2025 the UK renamed its AI Safety Institute, a government body that tests frontier models before release, the AI Security Institute, to focus on risks with security implications such as cyberattacks, fraud, and the development of chemical and biological weapons.