AI Hiring Assessments and Disability Bias: What Maki People’s Answer Reveals
AI hiring assessments are almost never tested for disability bias. Maki People answered our questions about the Mochi assessment on the record and in full, and their answer on adverse impact analysis by disability and veteran status shows exactly how that testing gap persists. Here are the five reasons the answer is not good enough.
Happy Friday, Job Board Doctor friends.
It is somehow August and my brain is already deep in fall conference season. So tell me: where will I see you this fall, Europe or the US?
Talroo brings clarity to improve new-hire retention
You filled the seat, but not with someone who plans to stay. The candidate made it through your entire interview process while quietly interviewing with other companies that pay 25% more.
When expectations aren't clear up front, retention suffers.
Get clarity up front with Talroo's SmartQualify.
A year and a half into Jeff’s chair, something unexpected happened. After 15 years building Disability Solutions, I did not think I would find another mission that pushed me to work the same way. That started to shift with the Monster and CareerBuilder bankruptcies, amplified through the Indeed disposition debacle, and solidified when I first wrote about Maki People and Recruitics.
This industry has plenty of coverage that depends on vendor goodwill. Job board operators, tech builders, and TA professionals deserve more analysts who aren’t.
Which brings us to this week.
After my overview of Maki People’s integration with Recruitics, I heard from CEO Maxime, corrected the record, and went back with more questions. I did not expect a response. I had two follow-up articles written and ready to publish. Instead, Maki People showed up and answered every question in full.
So I am publishing my questions and their responses, verbatim, with their permission, in a companion piece. Go read the full Q&A first. Then come back, because engagement like that earns goodwill, but it does not end the conversation.
This week and next, I am digging into the two questions that deserve a deeper look. This week: question 3, and the disability data topics where I have particular standing to weigh in.
Question 3: Adverse Impact by Disability and Veteran Status
THE QUESTION
Your response says clients can collect voluntary self-identification data to support applicant-flow recordkeeping. Per MakiPeople’s privacy policy and help section, demographic data including disability status is also collected by MakiPeople to improve fairness and unbiased in the handling of assessments. Have any adverse impact analysis, internal or external, ever been run on Mochi outcomes by disability status or protected veteran status, using voluntary self-identification data or any other method? If yes, what did it show? If no, is one planned, and on what timeline? Why have these demographic groups, which are of significant size in the United States, been omitted?
MAKI PEOPLE’S RESPONSE
As of this audit cycle, we do not yet have sufficient voluntary self-identification data on disability status or protected veteran status to support statistically defensible adverse impact analyses on Mochi outcomes for these groups. This is not an omission by design. It reflects the reality of where we are in our US market entry and the voluntary nature of demographic disclosure.
Protected veteran status represents approximately 6% of the US adult civilian population. Given that we have only recently entered the US market and are in the early stages of client partnerships, the proportion of candidates disclosing veteran status in our dataset remains too small to draw meaningful conclusions. Publishing results from an underpowered analysis would risk producing misleading findings, which serves neither candidates nor clients.
Disability status presents an additional layer of complexity. Disability is not a single, homogeneous category. The CDC report meaningfully different prevalence rates across cognitive 13.9%, mobility 12.2%, hearing 6.2%, and vision 5.5% impairments, each of which may interact differently with a conversational AI assessment. Collapsing these into a single group for analysis would obscure more than it reveals. We are committed to doing this analysis properly, which requires both sufficient sample sizes within disability subgroups and a methodologically appropriate framework.
We intend to pursue both analyses once we have the evidence base to support them. We will not release findings until we are confident they are well-supported and defensible.
THE DOC RESPONDS
Let me caveat my response with an acknowledgement that it was not written for me, it was written for you, my friends.
However, it is written TO me, so I am going to take a side quest and air a grievance, and it goes a little something like this.
A SHORT SIDE QUEST
Dear Tech Leaders: before you write a response to some little peon industry analyst whom you only deem worthy of attention because her husband has a large bully pulpit, take 60 seconds and Google her. You might find out that said analyst has built systems, processes, and programs that have put tens of thousands of people with disabilities to work. That her work has been reviewed by the US Congress, that what she wrote wound up on the desk of a US President, and that she has spoken at the United Nations on International Day of Persons with Disabilities. You would also find that she did not just do work for the community. She is part of it.
So let’s dissect the question and respond to the parts so we can see the whole.
1The word missing from the answer is no
My question asked whether any adverse impact analysis has ever been run on Mochi outcomes by disability or veteran status, using voluntary self-identification data or any other method.
Maki answered a narrower question: whether US voluntary self-identification data is sufficient. Here is what is actually missing from the response: the word no. The actual answer, translated from the vendor: no analysis of any kind has ever been run, none is scheduled, and no threshold has been defined that would trigger one.
2The US market entry framing fails on geography
Maki is headquartered in France and has operated in Europe for years, serving major European enterprises.
France itself is not a data desert on disability. Per Agefiph and INSEE Employment Survey data for 2024, 3.3 million working-age people in France hold formal administrative recognition of a disability, 8.1% of the working-age (15-64) population.
Another 4 million people currently in employment report lasting activity restrictions from health conditions, 13.9% of the employed workforce (INSEE, 2024). Their unemployment rate runs at 12% vs 7% for the general population, and only 46% participate in the labor market at all, vs 75% overall.
And France does not treat disability employment as optional. The Obligation d’Emploi des Travailleurs Handicapés requires every employer with 20 or more employees, public or private, to employ recognized disabled workers at 6% of headcount.
FRANCE, BY THE NUMBERS
Unemployment rate
Labor market participation
Bars drawn to a common 0–100 scale. Sources: Agefiph, INSEE Employment Survey 2024.
Maki built its product, and chose its fairness testing scope of gender, age, and nationality, in the jurisdiction with one of the stronger disability employment mandates in the world. The omission predates the US market.
3The answer contradicts Maki’s own science page
Maki’s science page, which is live as I write this and archived in case that changes, advertises rigorous fairness checks, DIF and adverse impact under the 80% rule, at build-time and ongoing. It advertises inclusive design with support for neurodiverse candidates.
Pick one. Either those build-time fairness checks include disability, in which case Maki’s answer to my question understates what data exists, or they exclude it, in which case the marketing overstates what exists. Both cannot be true.
4Heterogeneity is the safe excuse
The heterogeneity argument has always been the “safe” excuse for companies uninterested in ensuring their solutions actually provide equitable access for diverse populations, but not brave enough to say, “hey, we just don’t give a shit.”
It’s too complicated, said the company that is selling investors and global TA leaders on the claim that it is sophisticated enough to be trusted with determining who should be considered at scale, based on an Al video interview.
Maki cites CDC subgroup prevalence to explain why disability is too complex to analyze as a single category. Look at which subgroups those are. Cognitive and hearing disabilities, and the speech differences that accompany many of them, are precisely the categories most likely to interact with a conversational voice AI assessment that scores the richness, accuracy, and complexity of the language a candidate produces.
And the single-category excuse was solved years ago: Section 503 requires federal contractors to track disability as one category against a 7% utilization goal, and OFCCP has never accepted “disability is heterogeneous” as a reason not to do the work.
5Waiting for organic disclosure is not the only method
That is why my question said “or any other method.” Vendors that want to know whether their product works for disabled users recruit disabled users and test. Structured studies with paid participant panels. Accessibility conformance testing. Controlled evaluations across speech and language differences.
Framing this as a sample-size problem assumes the only valid method is passive observation, and that assumption is false.
What Disability Bias Testing for AI Hiring Assessments Should Look Like
For Maki or any vendor who wants to write one: name the sample threshold that triggers the analysis. Name the date you expect to reach it. Name the methods you will use in the meantime that do not depend on disclosure rates. Commit to publishing the results either way.
Until then, “we intend to pursue both analyses” is a commitment not worth the lawyer who wrote it, and candidates with disabilities are being scored by a system that has never been tested on them, or if it has, no result has ever been made public.
I believe I can say for the community as a whole that we are really done with this answer. I have heard it, rebutted it, and built success on its ashes for more than a decade. Companies just have to be willing to do the work.
It isn’t rocket science. It’s commitment.
IN CASE YOU MISSED IT
Comments (0)