7 Steps in Managing AI Risk in the Public Sector

Artificial intelligence can help public-sector organisations improve service delivery, reduce administrative burden, analyse information, and support staff decisions. But its use also creates new risks. Sensitive data may be exposed, inaccurate outputs may influence decisions, suppliers may change how systems operate, and staff may use unapproved tools without understanding the consequences.

Firebrand’s 2026 survey of senior UK leaders across energy, financial services, retail, telecoms, professional services, and IT reveals that almost half half of organisations openly acknowledge this high tech skills and knowledge gaps in cyber security.

With this skills gap, the main question that leaders and managers working in the public sector should be asking is how they can use AI in a way that is secure, lawful, accountable, and worthy of public trust.

Before we go through the steps of managing risks in any public sector group, let’s first understand why leaders in these organisations are critical in making their workplace more secure, benefitting the public they’re serving.

Why is AI risk a leadership issue in the public sector?

AI is not simply another IT tool, especially in public service bodies. It can shape communications, staff workflows, case management, service eligibility, fraud detection, resource allocation, and citizen interactions.

When an AI-supported process fails, the impact may extend beyond operational disruption. It may also affect people’s rights, safety, access to public services, trust in institutions, just to name a few.

Leaders do not need to become model engineers. They do, however, need to establish clear accountability, decide which uses are acceptable, ensure appropriate controls are funded, and make sure staff know how to use AI safely.

Weak oversight can lead to:

• Personal or confidential data being entered into an unapproved generative AI tool
• Incorrect or fabricated AI outputs being used in public-facing content or staff advice
• Biased recommendations affecting people or communities unfairly
• AI-enabled phishing, impersonation, and fraud targeting staff or citizens
• Unclear responsibility when an AI supplier, model, or connected system fails
• Reputational damage that undermines confidence in public services

A useful principle is simple: if a service owner needs to explain a decision to a citizen, auditor, regulator, or elected representative, they must understand how AI contributed to it and who remains accountable.

What are the main AI risks that the public sector must manage?  

It goes without saying that AI makes several of these risks more immediate.

For example, staff can paste large amounts of unstructured information into a prompt. There is also a possibility of models generating plausible but inaccurate content. AI assistants can be connected to email, document stores, websites, or other systems, which can increase the potential impact of weak access control or poor configuration.

Here are other examples and how it may look like: 

Risk area	What it may look like	Management priority
Cybersecurity	Prompt injection, compromised AI tools, insecure integrations, AI-assisted fraud	Treat AI as part of the cyber-risk programme
Data Protection	Personal, case, health, financial, or operational data entered into public tools	Define approved tools and strict data-handling rules
Reliability	A chatbot produces a confident but false answer or summary	Require human verification before important use
Fairness	An AI-assisted process disadvantages a group of people	Test outcomes and provide human review routes
Governance	No inventory, owner, approval process, or evidence trail	Establish leadership accountability and oversight
Supplier Risk	A provider changes its model, data terms, or sub-processors	Strengthen procurement and contract assurance
Workforce Risk	Staff use convenient AI tools without guidance	Provide role-based, cybersecure AI training provided by a trusted provider such as Firebrand

So how can leaders manage these risks? Here are seven steps based on best practices of regulatory and public governance bodies.

Step 1: Establish Governance Before Scaling AI

Good AI governance should help the organisation make better, faster decisions—not create unnecessary bureaucracy. The aim is to ensure that every AI system has a clear purpose, an accountable owner, a known risk level, and proportionate safeguards.

Start with five foundations:

1. 1. Assign an accountable senior owner: Name a leader responsible for organisational AI risk, supported by clear reporting lines and decision-making authority.
2. 2. Create a cross-functional governance group: Include service delivery, cybersecurity, data protection, legal, procurement, records management, risk, and equality or ethics expertise. AI risks rarely sit in one department.
3. 3. Maintain an AI inventory: Record every AI tool, pilot, embedded supplier feature, automation, and chatbot in use. For each one, document its purpose, owner, data inputs, supplier, connected systems, user groups, and risk level.
4. 4. Define acceptable use: Set out which AI uses are permitted, restricted, or prohibited. Staff need clear guidance on what information may be used, which tools are approved, and when specialist review is required.
5. 5. Use approval gates: Require formal review before a high-impact AI system is procured, piloted, deployed, materially changed, or connected to sensitive data or operational systems.

Tip: The NIST AI RMF provides a useful structure: governance sets the culture, roles, and policies; mapping defines the system and its context; measurement evaluates risks; and management applies controls and responds to incidents.

Step 2: Apply a Risk-Based Approach

Not every use of AI carries the same risk. A tool that helps a team brainstorm generic internal communications is very different from a system that influences safeguarding or access to essential services. You must do an AI audit to look into all the tools and systems you use.

Leaders should therefore require a proportionate risk assessment before deployment, considering the intended public-service outcome. They should also consider potential consequences if the system is inaccurate, biased, unavailable or manipulated.

Another big consideration is whether it could affect people’s rights, safety, opportunities or access to services.

They should also establish where meaningful human judgement sits, who is accountable for the final decision, what evidence demonstrates that the system is secure and fit for purpose, and how it will be monitored after launch.

This helps prevent a common failure of treating AI adoption as a one-off technology purchase rather than an ongoing operational responsibility.

Step 3: Secure Data, Systems, and Integrations

AI security builds on established cybersecurity practice, but AI introduces specific threats that need dedicated attention.

The UK National Cyber Security Centre recommends security throughout the AI lifecycle, including secure design, development, deployment, and ongoing operation and maintenance.

Leaders should ensure their organisation has the following controls in place:

• Data classification and handling rules: Clearly identify what staff must never enter into consumer or unapproved AI services, including personal data, sensitive case information, credentials, security details, and confidential operational material.
• Approved enterprise tools: Provide secure, organisation-approved AI services with appropriate data protections, access controls, retention settings, and supplier commitments.
• Strong identity and access management: Use multi-factor authentication, role-based access, and least-privilege permissions so people and systems access only what they need.
• Secure integrations: Assess APIs, plug-ins, connected knowledge bases, and automated workflows before enabling them. A model connected to internal documents may expose more information than an unconnected chatbot.
• Logging and monitoring: Maintain suitable records of access, configuration changes, high-impact prompts, outputs, and AI-supported actions, particularly where accountability or investigation may be needed.
• Security testing: Test for prompt injection, insecure output handling, unintended data exposure, misuse, and weaknesses created by third-party components.
• Incident readiness: Include AI systems in cyber incident-response plans, tabletop exercises, vulnerability management, and business-continuity arrangements.
  
  

Step 4: Protect Privacy, Fairness, and Human Oversight

A technically secure AI system can still cause harm if it uses data inappropriately or produces unfair outcomes.

Leaders should ensure that AI uses only the data it genuinely needs, with clear arrangements for privacy, retention and deletion. Systems should also be tested for fairness and accessibility, particularly across different groups and contexts, and designed so that language, disability or digital exclusion do not create additional barriers.

Tip: AI-supported decisions should remain subject to meaningful human oversight. People must have the context, authority, time and training to question, validate, override or stop an AI output, particularly where it could materially affect an individual. Clear routes should also exist for people to challenge decisions, correct information and raise concerns.

Step 5: Make Procurement a Core Control

Many public-sector organisations will access AI through vendors, cloud platforms, software updates, or embedded features rather than building models themselves. This makes procurement and supplier management central to AI governance.

Before approving an AI supplier, leaders should seek clear answers to the following questions:

• Where will organisational data be processed and stored?
• Will the provider use customer data to train or improve its models?
• Can that data use be disabled contractually and technically?
• Which sub-processors, cloud providers, models, and plug-ins are involved?
• What evidence exists of security testing and independent assurance?
• How quickly must the provider report a security incident or material service change?
• How are model updates, new capabilities, and changed terms communicated?
• What audit rights, deletion commitments, exit arrangements, and continuity plans are available?
• How does the supplier manage reliability, bias, harmful content, and misuse?
  
  

Step 6: Build a Cybersecure AI Culture

Policies will not prevent risky AI use if staff do not understand them. Employees are likely to turn to AI because it is convenient, fast, and widely available. The leadership task is to make secure behaviour the easiest behaviour.

Training should be role-specific:

• Senior leaders and managers → Understand accountability, risk appetite, procurement decisions, and escalation responsibilities.
• Frontline staff → Need clear guidance on approved tools, restricted data, output verification, and when to seek help.
• Cybersecurity and IT teams → Need AI-specific threat awareness, testing methods, integration controls, and monitoring processes, as well as up-to-date cybersecurity certifications.
• Procurement and legal teams → Need supplier-assurance questions, contractual safeguards, and change-management requirements.
• Data-protection and service teams → Need to assess privacy, fairness, transparency, and the impact on citizens.

A practical message for every employee is: Stop, check, protect. Stop means physically stopping before entering information into an AI tool. The second step is to Check whether the tool and data use are approved.

The last step, Protect, is a reminder to protect people, information, and public trust by verifying outputs and escalating concerns.

Tip: Staff should also be trained to recognise AI-enabled phishing, deepfakes, impersonation, and manipulated content. Those working closely with advanced technology should also be upskilled and certified in cybersecurity.

Step 7: Monitor, Audit, and Prepare for Incidents

AI risk management does not end at launch. Models, data, suppliers and integrations can change, while new vulnerabilities and attack methods can emerge. Just take into account these new incidents of AI-powered cyber attacks.

Leaders should establish an ongoing cycle of review, monitoring systems for unexpected outputs, accuracy issues, misuse, data leakage and anomalous access. Risks should be reassessed after significant changes, with regular security and privacy testing, AI-specific incident-response exercises, and clear records of decisions and remedial actions.

Organisations should also be prepared to pause, restrict or withdraw an AI system when its risks exceed acceptable levels.

This aligns with the NIST approach of continuously measuring and managing AI risk rather than treating it as a one-off exercise. But effective AI governance also depends on having the right skills in place.

Firebrand, a technology certification specialist, helps organisations build practical, role-specific capabilities across AI, cybersecurity and governance, with tailored training designed to equip teams with the knowledge and skills needed to implement, manage and secure emerging technologies responsibly.

Frequently Asked Questions: AI and public sector

What is AI risk management in the public sector?

AI risk management is the process of identifying and reducing security, privacy, legal, fairness, operational, and reputational risks associated with AI used in public services. The NIST AI RMF groups this work into four functions: Govern, Map, Measure, and Manage.

What are the main AI security risks for government organisations?

Key risks include leakage of sensitive data, insecure integrations, prompt injection, unreliable outputs, unauthorised use of public AI tools, AI-enabled phishing, and supplier vulnerabilities.

Who is responsible for AI governance?

Senior leadership should assign a clear accountable owner, supported by a cross-functional group covering service delivery, cybersecurity, data protection, legal, procurement, risk, and records management.

How can staff use generative AI securely?

Staff should use approved tools only, avoid entering restricted or sensitive information, verify AI outputs before relying on them, follow organisational guidance, and report suspected misuse or incidents promptly.

What should leaders ask an AI supplier?

Ask where business data is processed, whether it is used for model training, which third parties are involved, how incidents are reported, what independent security assurance exists, how changes are managed, and what deletion and exit rights apply.

How would you adapt the examples and risk scenarios to reflect the day-to-day services your target public-sector leaders manage?

Responsible AI Requires Visible Leadership

Public-sector organisations earn trust through reliable services, fair treatment, and responsible stewardship of information. Effective AI governance protects those foundations.

When leaders combine strong cybersecurity, proportionate controls, supplier assurance, workforce training, and meaningful human oversight, AI can support better public services without compromising the public interest.

Leaders do not have to approve every prompt of all employees. Rather, they have to make sure their employees are trained enough to be responsible of its use. And that starts with upskilling and making sure they are certified.

Firebrand helps organisations develop the AI, cybersecurity and digital skills needed to navigate emerging technologies with confidence. Explore Firebrand’s tailored solutions to identify your organisation’s skills gaps and build the capabilities needed for responsible AI adoption.

Explore Firebrand’s training solutions