Human in the loop: keeping people in AI decision processes

A fraud analyst at a mid-sized bank must clear a queue of flagged transactions before lunch. The model has already assigned a score to each transaction. Her official responsibility is to review these scores and make a final decision. In reality, she begins clicking approval in the order the queue appears after the first hour. Making four hundred attentive decisions every day is not possible for a human, and the bank did not redesign the role when the workload tripled. The bank describes this process as human oversight. This is inaccurate. It is a signature placed next to an automated answer.

Human in the loop refers to the practice of involving a person in an automated decision process. This person reviews outputs before they are finalised, manages cases where a model is uncertain, or corrects data used to train future versions. This term only applies if the person has enough time and information to disagree with the machine, rather than just having the technical ability to intervene.

Human in the loop, or HITL, refers to the practice of including a person within an automated decision process. This role involves reviewing outputs before they are finalised, managing cases where a model is uncertain, and labelling data to improve future versions of the system. A person might also intervene when a situation exceeds the system's programmed capabilities. The term is often used loosely to suggest that "a person could technically intervene" rather than confirming that "a person meaningfully does." These are distinct concepts, and the difference between them is frequently the reason that HITL designs fail without being noticed.

We develop data and AI systems at Shipshape Data. Discussions about human oversight typically occur on nearly every project that involves a decision impacting a person's finances, health, job, or legal rights.

The problems we observe are rarely caused by the model.

This guide addresses the handover process. Often, there is no plan for what a person should do at this checkpoint, how much time they should spend, or how they should handle the work once it becomes routine. This text explains where human input provides actual value in an AI process and how to design handovers that ensure reviews are meaningful. It also describes how automation bias can cause oversight to become a perfunctory task. Finally, it covers the governance and regulatory requirements that make human involvement mandatory.

What "human in the loop" actually means

Human-in-the-loop (HITL) operates along a spectrum instead of being a single setting. One extreme of the spectrum is human-in-the-loop proper. In this arrangement, the model proposes a result and a human makes the final decision. No action is taken until the person intervenes. Examples are content moderation queues, loan underwriting for large amounts, and most clinical decision support systems. Human-on-the-loop occupies the middle of the spectrum. The system operates independently while a human watches the output. The human can intervene, but this typically occurs after the action rather than before it. Fraud systems work this way by automatically blocking transactions and permitting a human to reverse the block later. At the opposite extreme is human-out-of-the-loop. This represents full automation with no human involved in the process. Recommendation engines and spam filters usually employ this approach because the cost of an incorrect individual decision is low and easy to correct.

The label hides the design issue

None of these positions are inherently superior to the others. The mistake lies in selecting a position to seem cautious instead of picking one grounded in actual risk. A system may demand a human to approve actions yet operate as though the human were absent. This occurs when the individual overseeing the checkpoint does not have the time or the information needed to oppose the machine.

That is the fraud queue displayed in the first scene. The process name and the way the process actually functioned had diverged before anyone noticed.

Where a person adds real value, and where they don't

You do not have to staff every checkpoint. Considering all checkpoints equally important makes review teams spend excessive time on low-value tasks, leaving them with insufficient time for difficult cases. Four specific areas justify the cost of human review.

Review of significant and unclear decisions

If a decision has a major impact on an individual and the model displays low confidence, a human reviewer ought to examine the entire case instead of only the score. A reviewer can spot details that a numeric threshold overlooks, including information the model failed to process, particular policy exceptions, or pertinent customer history. Such manual review is effective only when applied selectively. Reviewing every decision irrespective of the confidence score spreads attention too widely and captures cases that do not need human intervention.

Escalation for cases that are truly uncommon

Models operate according to the data distribution employed in their training. Escalation acts as a required process for cases that lie outside those parameters. This covers accounts that do not align with established fraud patterns or medical symptoms that differ from the typical cases in a training set. A working escalation system routes these unusual cases to a person who possesses the authority and specific context needed to resolve them. This stops the model from guessing and prevents a generalist reviewer from deciding without the proper expertise.

Labelling and correction with feedback loops

Someone must notify the system of an error and supply that feedback in a format suitable for the next training cycle. This work is frequently outsourced or under-resourced, yet it remains a critical stage for human intervention in the development process. Each correction influences the model's future behaviour rather than only a single decision. If a labelling team cannot affect the retraining process, their work has no impact on the system.

Exception handling that a rule can't cover

Any system built on rules or models will eventually face a situation its designers did not anticipate. A decision-making person, rather than a script-following individual, prevents that situation from being forced through a process intended for a different purpose.

Zillow's iBuying arm learned this the hard way: its home-pricing algorithm continued making offers as the housing market shifted in 2021, volume grew faster than the human checks surrounding it, and the company wrote down a large portion of its housing inventory before shutting the business down.

The algorithm operated as intended. The monitoring systems failed to adapt to the algorithm's current operations.

Designing the handover

Most human-in-the-loop failures are not caused by poor judgement. Instead, they stem from the design of the handover. This includes the specific information given to the person, the format of that data, the time available for a decision, and the existing incentives. If these elements are designed incorrectly, the human participant becomes merely a formality.

Present the evidence instead of the conclusion

When a review screen displays a score alongside approve and reject buttons, it encourages users to base their decisions on that score. If the screen instead presents the underlying evidence and treats the model's assessment as one of several inputs rather than the primary focus, it encourages users to examine the data.

This appears to be a minor decision regarding the interface. However, it affects the entire game.

If you place the model's confidence score in bold at the top of the screen, you signal to the reviewer which answer is expected.

Protect the time to think

If the daily target is four hundred reviews, the system design assumes the review process will be superficial, regardless of the official job description. A genuine review process requires a volume and pace that allows staff to disagree with the machine. When a model's recommendation is never overturned, it typically indicates that the review process is no longer functioning, rather than proving the model is accurate.

Make disagreement safe and visible

Reviewers learn that agreeing is the easiest option when they are penalised for slowing the process or when they receive no feedback on their overrides. You should track and review override rates. If these rates decrease towards zero over time, you should investigate the cause rather than assuming it is a positive development.

Automation bias, and how review turns into rubber-stamping

Automation bias is the tendency to favour a system's output over your own judgement because the information was generated by a computer. This behaviour is documented in fields such as aviation, radiology, and clinical decision support. It does not occur because a person is careless or lacks training. Instead, it is a standard way that people process repetitive decisions when working with a machine that appears confident.

Vigilance decays faster than anyone expects

The clearest example does not come from a corporate process. In 2018, an Uber self-driving test vehicle hit and killed a pedestrian in Tempe, Arizona. The vehicle's sensors detected the pedestrian but the system did not brake in time. The human safety driver was responsible for intervening if the automation failed, but they did not react before the impact. An investigation by the National Transportation Safety Board found that the driver was looking at a phone instead of the road.

The lesson learned was not about the behaviour of a single driver. Instead, the situation demonstrated that humans cannot reliably maintain constant alertness to override a system that functions correctly most of the time. Expecting a person to remain ready to intervene under these conditions is a requirement that exceeds the natural limits of human attention.

Attention to rare errors decreases when it must be maintained over a long shift. Every human-in-the-loop design that expects a bored reviewer to remain constantly ready faces this same issue on a smaller scale.

A reviewer who processes four hundred decisions before lunch fails to perform a review. This person follows a repetitive pattern.

Rubber-stamping is a system property, not a personal failing

People often assume that rubber-stamping happens because a reviewer is lazy. This behaviour is usually a result of the workload, the software interface, and the performance metrics created by management. If you reduce the queue size, improve the interface, and change how reviewers are evaluated, their behaviour will change. If you blame the individual instead, the system design will cause the next person in that role to make the same mistakes.

Why regulators keep asking for a human in the loop

Automation bias is a sufficient justification, but current laws also require a human participant. These regulations treat human oversight as a mandatory control rather than an optional feature.

The right to a person

Article 22 of the GDPR provides individuals the right to avoid decisions based only on automated processing if those decisions result in legal or significant consequences. It also grants them the right to request human intervention in these cases. The EU AI Act includes stricter requirements for systems categorised as high-risk. These systems must include oversight measures that allow a person to understand the outputs and take action, rather than just having the option to deactivate the system. Neither framework considers it enough if "a human was technically in the org chart". Both laws are designed to ensure that human oversight is a functional reality rather than a formality on paper.

Courts have begun to examine this discrepancy

In 2024, a Canadian tribunal ruled that Air Canada was responsible for incorrect information its website chatbot gave to a customer regarding bereavement fares. The airline claimed the chatbot was a separate legal entity and was responsible for its own statements. The tribunal rejected this argument. The company was held liable for the automated system's response because no human reviewed the information before the customer received it. While the financial penalty was small, the case establishes that an organisation is responsible for the output of its systems if those systems operate without human oversight.

This work is connected to AI governance because humans cannot oversee decisions they do not understand. It is part of the compliance framework described in our guide to GDPR compliance. In that context, human oversight is one of several required controls rather than a complete solution on its own.

Measuring loop performance

Most organisations can tell if they include a human-in-the-loop step. Fewer organisations can assess whether that step works effectively. Providing an answer to the second question needs different evidence than providing an answer to the first.

Override rate is the number that matters

Track how often humans disagree with the model. View this metric as an ongoing trend instead of a single data point. If the disagreement rate stays close to zero for several months, it usually points to one of two possibilities: the model has achieved a high accuracy level, or the human review process has lost effectiveness. To find out which is true, conduct an independent audit by sampling a subset of decisions that were previously approved.

Time-on-task is a cheap, useful proxy

When a review takes eleven seconds while the case file contains three pages, the reviewer is not reading the whole document. Identifying this problem does not require complex tools. Simply look at the timestamps. People frequently skip this step because they do not want to acknowledge that the work is not being performed thoroughly.

Feed corrections back, and check that they land

A labelling or correction process that fails to alter the model is ineffective. You need to verify that overridden decisions and corrected labels are delivered to the team that retrains the system. These corrections must lead to modifications in future model versions. This aligns with the same discipline outlined in our guide to model monitoring in production. If a correction never reaches the retraining pipeline, it is not a genuine fix. It will be ignored rather than improving the system over time.

When human-in-the-loop is overkill

This does not argue that human intervention must be required for every automated decision. That approach is a frequent mistake. Organisations often answer governance discussions by imposing review requirements on all processes, even when the actual risks are low. Such a practice creates bottlenecks that are hard to staff and results in teams that dislike the process.

Low stakes, high reversibility, high volume

Product recommendations, spam filters, and spell-check suggestions illustrate tasks where mistakes have low costs and are easy to correct. Mandating a human review for these decisions is both inefficient and unnecessary. The evaluation standard is whether a wrong decision causes harm and how hard it is to reverse. When risks are low and reversal is easy, the suitable design is automation with monitoring instead of manual review.

Where to start

Identify the decisions that your AI systems make or influence. Classify these decisions according to their consequences and how easily they can be reversed, rather than emphasising the technical novelty of the model.

Organising the checkpoints this way reveals which require a skilled person and which are reviewed out of habit rather than necessity.

Look at the checkpoints that presently involve human review. Check the data for override rates and the time taken for each task. If a reviewer has approved ninety-nine per cent of cases during a six-month span, you should probe the reason. It is preferable to resolve this internally rather than waiting for an enquiry from a regulator, a journalist, or a court. Design the handover process based on the real time and information the staff have, instead of building a system that only appears acceptable in a process diagram.

Success in this area typically does not come from buying review software. It demands identifying which particular tasks gain from human intervention and giving staff the time and context needed to carry out that work. If you require an objective assessment of where your human reviewers add value and where they approve outputs without critical thought, talk to us.

If you are moving from research to planning, our AI consulting services can help with strategy and building production-ready systems.

Frequently asked questions