An organisation can have an approved AI risk assessment, a validated model, security controls and a named accountable owner, yet still have gaps in how those controls work together. The model may have been tested on verified data while the deployed workflow accepts unverified inputs. A human approval process may exist, but the reviewer sees only the model’s summary rather than the action being approved. Each control may appear satisfactory on its own while the overall system remains exposed.
These connections matter most at hand-offs. A data owner marking a source as unreliable should be able to identify which approvals depend on it. An engineer widening a tool’s permissions should know which risk decision must be revisited. Governance has to make those dependencies visible.
Risk classification, data and model governance, guardrails and security, monitoring and controls, and human oversight and accountability provide a useful structure. This article examines how to connect them. We argue that effective AI governance depends on more than having the right controls. Those controls must remain connected to the business decisions, evidence and conditions they were designed to govern.
Risk classification should define the conditions of use
Take a supplier-onboarding workflow as an example. AI reads registration documents, extracts company details and prepares a record for review. Giving it permission to activate the supplier changes the assessment, particularly if activation enables purchasing or payment. The model may be unchanged, but the business consequence has changed. This is the distinction behind bounded authority: permission for defined actions under stated conditions and enforceable limits.
Risk classification should establish what the organisation can permit. It should specify permitted uses, unacceptable outcomes, required evidence and who may accept the remaining risk. Labelling a system “medium risk” leaves those questions unresolved. For an advice-only assistant, assess how staff will use its answer, not only whether it can call a tool.
Regulatory classification needs separate attention. The EU AI Act imposes duties according to its scope, uses and operator roles, with phased application dates. NIST’s voluntary AI Risk Management Framework 1.0 organises risk management through Govern, Map, Measure and Manage. These references serve different purposes; neither removes the need to define the actual workflow.
Data and model governance must preserve business meaning
Accurately extracting a supplier’s bank account number does not establish that the supplier authorised it. A document can be read correctly and still be out of date, contradictory or fraudulent. Governance must preserve the distinction between a reported fact and a verified one. The system should retain the source, relevant dates and verification status of material facts. Where finance and procurement use different supplier identifiers, the mapping must be explicit. Otherwise, a correct extraction can be attached to the wrong organisation.
Our ontology article examined shared, machine-readable definitions of business entities and their relationships. An ontology can help where decisions cross systems, although simpler schemas may suffice. It supplies business meaning; separate controls enforce permission.
Model validation should reflect the configuration and operating conditions in which the system will be used. The test record should identify the model version, prompts, retrieval configuration and relevant tools. An accuracy figure without those conditions cannot explain which deployed configuration was assessed. Validation should also cover the people and cases the system will encounter. Where outputs affect individuals, aggregate accuracy can hide uneven error rates. Evaluation should examine relevant groups and difficult cases, with limitations recorded before anyone relies on the result.
Guardrails and security must survive a wrong decision
Suppose the onboarding agent proposes activating a supplier before verification is complete. The target application should reject that operation, even if the model gives a convincing justification. Restricting access to supplier records is insufficient if the agent can still invoke an unrestricted update operation. This is the separation we examined in our control-plane article: governance defines authority, while identities, credentials, policy and applications enforce it.
The National Cyber Security Centre’s advice supports defence in depth, restricted credentials and network access, sandboxing and active oversight. Model-level safeguards are only part of that design. Testing should include malicious instructions embedded in documents, attempts to bypass approvals and failures in the controls themselves. A policy appearing in configuration does not establish that every execution path respects it. Testing should also cover alternative paths that might bypass the controls.
Controls also need testing against ordinary failure. What happens when verification times out or an update succeeds but its acknowledgement is lost? The workflow needs a defined response that neither treats uncertainty as approval nor repeats a consequential action blindly. For this example workflow, pausing activation while continuing document extraction could preserve useful work without relaxing the verification requirement.
Monitoring should test whether approval still holds
An approval made at deployment may no longer be justified when the system or its operating conditions change. Suppose a new connector changes how verification status is supplied. The original model may remain untouched, yet evidence from the old workflow may no longer support automatic activation. That change should trigger targeted revalidation and, where necessary, restriction of the affected action.
The records must connect inputs and source versions to requested operations, policy decisions, approvals and resulting changes. Recording only model responses leaves the business outcome unexplained. We would measure harmful outcomes alongside blocked attempts, missed errors, unnecessary interruptions and review effort. A system that blocks everything can look safe while doing no useful work. Conversely, a high completion rate can conceal expensive correction.
Monitoring needs agreed consequences: who investigates, when processing pauses and what evidence is required before it resumes. NIST’s Manage function explicitly connects post-deployment monitoring with incident response, recovery and change management.
Human oversight has to work in practice
In the onboarding example, the reviewer needs the proposed record change, its supporting evidence and unresolved discrepancies. A fluent explanation from the agent is not an adequate substitute. Approval must apply to the operation that actually executes. The reviewer must also have enough time, information and authority to challenge the proposed action. If deployment doubles the queue without adding review capacity, “human oversight” may describe a process the organisation cannot sustain.
Responsibility should follow the work. The authorised business owner accepts residual business risk. Data owners resolve source-quality failures. Engineering and security teams implement and test controls. Operations needs an accessible route to pause processing and escalate exceptions. NIST’s Govern function treats roles, accountability and staff training as part of risk management. Naming an accountable owner means little if that person cannot intervene.
People affected by an incorrect outcome also need a route to challenge it. Internal approval does not settle whether a decision was fair or correct, and correction must reach the records used downstream.
Proportionality is part of the design
Not every AI deployment needs the same architecture. A staff assistant using public documents for low-consequence drafting should not automatically inherit the controls of an agent authorised to change business records. Nor should every change force a complete reassessment. Reuse common identity, logging and approval mechanisms, while testing the risks specific to the workflow.
Finance might require verified supplier details before activation; manufacturing might require the correct equipment identity and engineering approval before a parameter change. The governance process can be shared, but its evidence and acceptable consequences must be assessed separately for each deployment.
Governance has to be judged by how the controls perform together, particularly when conditions change or something goes wrong. The aim is to prevent unacceptable outcomes without unnecessarily restricting useful work.
Disclaimer: This article is for information only and may change without notice. It is provided “as is,” without warranties (including merchantability or fitness for a particular purpose), and does not create any contractual obligations. Data Nucleus Ltd is not liable for any direct, indirect, incidental, special, consequential, or exemplary damages arising from use of or reliance on this document.
Data protection/UK GDPR: data-controller@datanucleus.co.uk