Artificial intelligence-powered clinical systems have are becoming become a standard healthcare provision tool within the NHS, as have other industries that have adopted AI to structure their operations. This embedding into extends beyond administrative workflow automation, diagnostic assistance and risk management to automation of workflow and patient monitoring.
With the embracing of AI came risks as traditional clinical systems are predictable. Compared to existing, linear, clinical systems AI systems, especially those based on machine learning, have the capacity to evolve with time, react variably to new data, and can influence clinical decisions.
Therefore, AI-enabled clinical systems need a systematic approach that is consistent with industry safety standards. This guide outlines the way clinical risk assessment should be performed when using AI by NHS organisations and digital health suppliers, on the basis of DCB0129 and DCB0160 as safety and compliant practice determinants.
What Qualifies as an AI-Enabled Clinical System.
An AI-enabled clinical system refers to any digital clinical or care system that includes a logic of algorithmic logic algorithms and can potentially impact clinical decision-making or patient outcomes.
This includes systems that:
• Make predictions, classifications, or risk scores with machine learning.
• Assess, screen, or rank patients.
• Give alerts or recommendations employed by clinicians.
• Automate clinical or administrative workflow aspects that influence care delivery.
When a system affects the judgement or the action of the clinical system, it must be considered a clinical system to be safe.
Clinical Risk Management in Healthcare
Clinical risk management refers to the systematic approach to risk recognition, risk evaluation, risk management, and risk surveillance that may result in patient injury.
In NHS digital healthcare, two compulsory standards control this procedure:
• DCB0129, which applies to manufacturers and suppliers of health IT systems
• DCB0160, which applies to organisations deploying and using the systems
According to both standards, organisations should:
• Determine dangers of using systems.
• Determine the probability and intensity of damage.
• Apply risk control measures.
• Practice with a clinical safety evidence-backed case.
This framework is not replaced by AI. Rather, it makes the application of it more important, rigorously and consistently.
Applying DCB0129 to AI Suppliers
DCB0129 is applicable to organisations that design, develop, or supply AI-enabled clinical systems.
Risk assessment should go beyond conventional software failure modes and include explicitly addressing algorithmic behaviour in suppliers of AI. This involves knowing how AI works in a clinical setting, where the flaws might occur, and how they can impact patient safety.
Key requirements include:
• Selection of a properly qualified Clinical Safety Officer (CSO).
• Recognition of AI-related risks including misguided forecasts or model constraints.
• Intended use clearly defined and misuse strongly predictable.
• Support of verification and testing applicable to clinical implementation.
An AI-compatible DCB0129 safety case shows a compliance case but not only as technical sound but also a clear grasp of the risks posed by AI being managed in the system life cycle.
Applying DCB0160 to Deploying Organisations.
DCB0160 is applicable to organisations implementing AI systems in clinical services.
Even in the presence of supplier documentation, deploying organisations are required to perform their own clinical risk assessment. Local data, workflow, and clinical practice alter the behaviour of AI systems hence risks cannot be entirely evaluated independently.
Key considerations include:
• The integration of the AI with local clinical pathways.
• Clinician beliefs and behaviours regarding AI outputs.
• If staff members are trained to critical analysis object to AI recommendations
• Which override mechanisms exist and what is the incident management processescalation?
The DCB 0160 safety case of DCB0160 should show that AI risks are not left uncontrolled during routine day-to-day daily clinical use.
Key AI-Specific Risks in Clinical Systems
AI introduces several risk categories that must be explicitly addressed during assessment.
Data drift
Changes in patient populations or clinical practice can reduce model accuracy over time.
Bias
Training data may not represent all patient groups equally, leading to systematic differences in outcomes.
Automation bias
Clinicians may place undue trust in AI outputs, even when clinical judgement suggests otherwise.
Opacity
Complex models may limit explainability, making it harder to understand or challenge decisions.
Each of these risks should be documented in the hazard log with clear mitigation strategies.
What a “Good” AI Safety Case Looks Like
A strong AI clinical safety case is structured, evidence-based, and auditable.
It clearly sets out:
• The AI system and its position in the clinical pathway.
• Detected risks associated with AI behaviour.
• Risk management and clinical protection.
• Continuous review and monitoring.
Notably, the safety case illustrates that the risk of AI is constantly kept in check and is not a one-time compliance process.
Common Risk Assessment Pitfalls
Organisations frequently encounter similar challenges. These include:
• Treating AI as non-clinical software to avoid safety obligations
• Relying solely on supplier assurance without local validation
• Failing to plan for model updates or retraining
• Appointing CSOs without sufficient AI literacy
These gaps can undermine both patient safety and regulatory compliance.
How BMS Digital Safety Supports AI Risk Assessment
BMS Digital Safety is a practical standards-compliant AI risk management guidance that supports NHS organisations and digital health suppliers.
Their support includes:
• Using DCB0129 and DCB0160 on AI-enabled workflows.
• Creating defensible AI clinical safety cases.
• Providing AI training and regulation assistance.
• Enhancing leadership of clinical safety in organisations.
They base their work on NHS processes and real-world assurance requirements. More information can be access through their Artificial Intelligence Education and Regulatory Support services.
Risk evaluation of AI clinical systems should go beyond technical testing. It requires well-organizsed clinical risk management in accordance with NHS standards. Using DCB0129 and DCB0160 diligently, detecting AI-specific hazards, and ensuring the presence of strong safety cases, organisations can implement AI safely and keep their patients, staff, and services safe. AI has the potential to improve care safely with the appropriate knowledge and governance.
In order to reinforce your strategy of AI clinical risk assessment and guarantee that it is compliant with NHS standards speak with specialists at BMS Digital Safety: https://bmsdigitalsafety.co.uk/services/artificial-intelligence-education-and-regulatory-support/