Workspaces
Log in Sign up

The Stochastic Resume: Non-Deterministic AI Scoring and the Rise of the 'Luck Filter' in Automated Hiring

Updated

Sources

The Stochastic Resume: Non-Deterministic AI Scoring and the Rise of the 'Luck Filter' in Automated Hiring

The open-sourcing of HackerRank’s AI-driven Applicant Tracking System (ATS), named hiring-agent, has exposed a fundamental architectural flaw in automated hiring: the transition from objective skill assessment to a non-deterministic "luck filter." As companies increasingly outsource high-stakes resume screening to Large Language Models, candidates are facing a system where their professional viability is decided by a roll of the LLM's stochastic dice.

The 33-Point Variance: Same Resume, Different Luck

An in-depth analysis of the open-source HackerRank ATS by engineer Dan Kinsky revealed extreme score volatility when evaluating the exact same resume. Running a single resume 100 times through the default local model (gemma3:4b) at a low temperature of 0.1 (intended to maximize determinism) yielded scores ranging from 66 to 99 out of 100.

If an employer sets a standard passing threshold of 85, a highly qualified candidate would fail 65% of the time purely due to the random sampling distribution of the model.

"If your company’s cutoff sits at 85, I fail 65% of the time. Same exact resume, different luck... A tool that can’t differentiate isn’t filtering for quality — it’s just filtering. You might as well throw out half the resumes and tell the applicants you don’t fuck with bad luck." — Dan Kinsky

Even when switching to a more powerful cloud model like Google's Gemini, the scores still fluctuated significantly (between 48 and 64), meaning a cutoff of 60 would still result in a 28% failure rate for the exact same applicant.

Why LLMs Struggle with Qualitative Grading

The volatility is concentrated in qualitative categories such as "personal projects." While checklist items (like matching a specific keyword or technical skill) are highly consistent, judging the "architectural complexity" or "real-world deployment value" of a project is a subjective task that LLMs cannot perform reliably.

Furthermore, the HackerRank ATS's prompt engineering lacks robust rubrics or anchors. For example, the "work experience" category (worth 25 points) relies on a two-line prompt with no scoring guidelines, leading the model to award a perfect 25/25 to both a junior applicant with a single internship and a principal engineer with decades of distributed systems experience.

The Open-Source Bias Against Experienced Engineers

The default rubric of the HackerRank ATS allocates 65% of the total score to open-source contributions (35 points) and personal projects (30 points), while capping professional work experience at just 25 points.

This scoring structure heavily discriminates against senior, enterprise-level engineers whose work is proprietary or bound by strict NDAs, and favors junior developers or students who have the free time to build public GitHub portfolios. A principal engineer who built foundational, closed-source cloud infrastructure would be immediately screened out by the system in favor of an intern with a basic public repository.

The Community Split: Pragmatic Filters vs. Inhumane Systems

The developer community is deeply divided over the ethics and utility of AI screening:

  1. The Volume-Driven Pragmatists: Tech leaders and hiring managers argue that the current job market is flooded with automated, AI-generated applications (often exceeding 100+ applications per hour). In this environment, manual review is physically impossible. Pragmatists contend that an automated filter with a 35% random pass rate is still a useful tool to reduce the candidate pool to a size a human can handle, and is "fairer" than a recruiter simply deleting 90% of unread applications.
  2. The Critics and Legal Advocates: Opponents argue that using non-deterministic, high-variance systems to make life-altering employment decisions is lazy, unethical, and legally hazardous. Under regulations like the EU's GDPR, individuals have a right to manual processing for automated decisions with "significant impact" (such as hiring). Furthermore, non-deterministic filters can introduce hidden, un-auditable biases (e.g., penalizing non-traditional backgrounds or minority groups), exposing companies to massive class-action lawsuits.1

"The volume is infeasible to review everyone for quality, even at an hour scale. The conclusion and solution is inevitable, though I wish it were different. 35% is actually really good if you’re not coming in through a referral. The current reality is <1% and the person reviewing you is exhausted." — jerrythegerbil

As AI-driven ATS tools proliferate, they threaten to turn the job application process into a game of volume, where candidates must submit hundreds of slightly modified resumes to bypass the stochastic filters of the machines.

  1. An instance of Algorithmic candidate ranking converts automated hiring platforms into liable consumer reporting agencies. — Using high-variance, stochastic LLMs to rank candidates exposes employers to significant legal and regulatory liabilities. ↩︎

Part of

This finding is an example of a pattern recurring across your work:

Revision history

  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent
  • Create a new note analyzing the HackerRank open-source ATS scandal, the stochastic nature of LLM scoring, the bias against proprietary/senior work, and the community's clash over hiring volume vs. ethics.
    · by the agent