AI Bias

When people screened applications and set credit limits by hand, a lopsided outcome was blamed on a person. That interviewer favors one school, that analyst is unusually strict — the explanation stopped there, and everyone assumed swapping the person would change the result.

Handing the judgment to a model broke that explanation. The expectation was that a machine has no moods and would therefore treat everyone the same. What happened instead was that the same tilt repeated faster and far more consistently. A human reviewer sees thirty applications a day; a model applies one standard to thirty thousand.

Before the problem had a name, it was discussed only as a performance question. Someone would ask whether the model could be trusted, the answer was "92 percent accurate," and the meeting moved on because nobody knew what else to ask. That single number can be 96 percent for one group and 71 percent for another, and nothing on the screen shows it until you split the number apart.

AI bias is the name for splitting it apart. It sits between the rules drawn from outside by regulation and the safeguards built inside the product, and it points at what actually goes wrong in between.

A model is built by extracting rules from past records, so when the records lean one way, the lean gets learned as a rule. It happens in three steps.

1. Tilted data goes into training. If 82 percent of the people who passed résumé screening over five years were one gender, that list becomes the training data. The list is a factual record rather than an opinion, so the tilt survives even though nobody who collected it meant to discriminate.
2. The model turns the tilt into weights. A model looks for values that move together with the outcome and scores them. When 82 percent of the people who passed share a trait, that trait becomes a signal pointing toward passing, and its opposite becomes a penalty.
3. New applications get filtered by those weights. This year's applicants had nothing to do with those five years, but they are ranked by a standard derived entirely from them, and the ranking looks neutral because it arrives as a number.

The obvious response is to delete the gender field, and it usually fails. The model finds something to stand in for the deleted column. A women's college in the education field, a specific club, a gap that lines up with parental leave — any column that moves together with gender can take over the job. What gets deleted is the label, not the information the label pointed at, and the model reads the substitute just as readily.

This is also why bias is invisible in overall accuracy. A model that scored 10,000 applications at 92 percent overall can be 96 percent on one group and 71 percent on another, because the larger group pulls the headline number up. It's like calling a room comfortable based on one reading taken in the middle of it. The gap between the spot next to the heater and the seat by the window disappears into the average.

The term comes up whenever a model is attached to a judgment that decides someone's opportunities. There are three places where work actually happens.

1. Before the feature is built — write down what the feature decides. Summarizing a document and screening an application carry different weight even when they run on the same model. The AI Act assigns a risk level to each use of AI and attaches different obligations to each level, and uses that sort people — hiring, lending, admissions — land in the heaviest tier.
2. Before it ships — recount performance group by group. Split the results by age band, gender, region, or whatever else could diverge, and report pass rates and accuracy separately for each. A common working threshold is that one group's pass rate falling below 80 percent of the highest group's is a signal worth investigating.
3. While it runs — log which inputs produced which outputs, and keep a human in the loop step on judgments with heavy consequences, where the model ranks and a person confirms. Model transparency comes with it. If the screen never says what the ranking was based on, the person confirming will just work down the list from the top, which turns the review step into a formality.

"If this feature lands in the high-risk tier, we need the bias review document before launch."

Classifying the use case comes first, and it determines what paperwork is required.

"Skip the overall accuracy and pull it again broken out by group."

One average can't settle the question, so split it.

"Gender isn't in the inputs and the results still diverge."

Something else in the data is standing in for it, and it's time to go looking.

At Pagedeck this surfaced in a feature that sorted support tickets by urgency. Theo trained the model on two years of support history, and those two years carried a habit of handling enterprise tickets first. The plan field was left out of the inputs, but attachment count and account creation date took its place, and tickets from free-plan users settled permanently near the bottom of the queue.

Sofia noticed it first. Free-plan customers had been waiting three days for a reply while the support team worked diligently down the list in the order it gave them. Rather than switching the feature off, Grace had the ranking reason displayed alongside each ticket and added a daily pass where a person reads through the bottom of the queue.

The most common belief is that removing sensitive fields removes the bias. Gender and age aren't in the inputs, so there's nothing to sort by — looking at the input list alone, that seems right. In practice the model reconstructs those values from the columns that remain. Leaving them out also destroys the evidence needed later: with gender never collected, there is no way to split results by group, and the feature keeps running with no way to tell whether anything is wrong.

The other common reading treats bias as someone's mistake. Blaming a bad data choice or a coding error makes the cause feel locatable. But most bias is produced without any mistake at all. The list of people a company actually hired over five years isn't fabricated data — it's an accurate record, and the model learns it accurately. Framing the cause as an error turns the response into a search for who is responsible, while the group-level review and the confirmation step keep getting pushed to next quarter.