Open-source playbook · 18 of 20 · For ops, finance and HR

Candidate Shortlist System

Hundreds of applications per role get skimmed in seconds each, and good people slip through.

The systemScores every application against the role’s scorecard and hands the recruiter a ranked shortlist with reasons.

Download the PDF Email it to your team

// The problem

Hundreds of applications. Only the early ones get a real read.

When a role gets busy, applications are read in the order they arrive, and the later ones get seconds each. A strong person who applied on day six can slip past before anyone reads their answers properly.

TodayRead in the order they arrive

Read in fullSkimmed in seconds

With the systemEvery one read against the same scorecard

Scored, with reasons, for a recruiter to review

Illustration, not data: one role, 412 applications, one recruiter.

// How it works

From 412 applications to a shortlist a person chose.

The system reads and ranks. The recruiter reviews and decides. Sample numbers for one role at a fictional company.

System
412applications in

Pulled from Greenhouse or Lever as they arrive. Nothing is skipped and nothing is filtered out.
System
412scored against the scorecard

Name, photo, age and address are removed first. Each criterion gets a score and the line from the application that earned it.
System
41strong118possible253lowranked into three bands

A reading order for the recruiter, not a decision. Nobody is rejected at this step, or at any step the system runs.
Recruiter
recruiter reviews every band

Reads the reasons, checks the quotes and goes through every low-ranked application. In this sample, 3 people came up from the lower bands.
Recruiter
18on the shortlist

Chosen by the recruiter and sent to the hiring manager with the evidence. Every decision is logged with its reason.

Removed before scoring

NamePhotoAge and birth dateGraduation yearsHome address

Never used or guessed: gender, race, religion, disability, age or any other protected characteristic.

// The scorecard

Scored on the job, with the evidence quoted.

Every role gets a scorecard before the first application is read: the criteria, their weights and what each score means. The system quotes the line behind every score, so the recruiter can check it in seconds. Sample role and applicant, fictional company.

Scorecard · Senior Customer Success Manager

Written by the hiring manager and the recruiter before the role opens

RenewalsHas owned renewals for a book of B2B software accounts
30%
OnboardingHas run onboarding for accounts with many users or sites
20%
Product dataUses product usage data to spot accounts at risk
20%
Working across teamsWorks with sales and product on expansions and feedback
15%
Written communicationClear, specific answers to the application questions
15%

How each criterion is scored

4Clear evidence, with a result
3Clear evidence
2Some evidence
1Mentioned, no detail
0Not in the application: ask in the screen call

Applicant 0147

Senior Customer Success Manager · applied on day 6

79of 100Strong match

Name, photo, age and address hidden from scoring

Renewals
“Owned renewals for 62 mid-market accounts and kept gross retention at 94% in 2025.”
Onboarding
“Led onboarding for a 1,200-seat hospital group across 9 sites.”
Product data
“Built a health-score sheet from usage exports that flagged 11 at-risk accounts a quarter early.”
Working across teams
“Worked with AEs on expansion calls.”
Written communication
“I’d start by meeting the ten accounts that renew before March, before I change anything.”

Ask in the screen callHow did you work with product on feature requests? Ask for one example with a result.

// The rules it runs by

It ranks. People decide.

Hiring is where an automated decision can do real harm, so these rules are part of the build. They aren’t settings someone can switch off.

Before you switch it on

Rules for AI in hiring differ by place. New York City’s Local Law 144, for example, requires an independent bias audit within the year before you use a tool like this, a public summary of the results, and notice to candidates. Have HR or legal check yours. This page isn’t legal advice.

It never rejects anyone
There is no reject button. It can’t move a candidate to another stage or send them anything. Every decision is a recruiter’s.
A person reviews every low score
The recruiter reads the reasons for every application ranked low, and moves people up when the system missed something.
Job-related criteria only
It scores what’s on the role’s scorecard and nothing else. No “culture fit”, no guesses about personality.
Personal details hidden
Names, photos, ages, dates that reveal age and addresses are removed before scoring. It never uses or infers gender, race, religion, disability or any other protected characteristic.
Every decision logged
Each score, the quote behind it, and the recruiter’s decision and reason are kept, so you can explain any outcome.
Checked against the law, audited for bias
HR or legal checks it against local hiring law before launch, and you run bias audits on a schedule after.

// For your tech lead

Can we build it? Here’s what it takes.

The scorecard matters more than the model. Write one per role with the hiring manager, run the system next to your normal process for one role, and compare.

Tools you’ll use

Job	Use	Or
Applications	Greenhouse or Lever (via their APIs)	Ashby, Workable
Hiding personal details	A redaction step in the workflow	Microsoft Presidio (open source)
Scoring against the scorecard	Claude (API)	ChatGPT, Gemini
Running the steps	n8n	Make, Zapier
Review and decision log	Notes in the ATS, plus a log of every score and decision	Google Sheets or Excel

Access it needs

Read access to applications, through a limited API key
Write access to ATS notes only: never stages, rejections or emails
An AI model account that doesn’t train on your data

Three ways to build it

Start simple1–2 days
The scorecard as a saved prompt in your approved AI workspace. The recruiter pastes in applications with personal details removed.
Production1–2 weeks
n8n pulls new applications, removes personal details, scores each one and writes the scores and quotes back to the ATS as notes.
Custom3–5 weeks
Adds a review screen, checks scores against past interview panels and builds the reports your bias audit needs.

Guardrails

It never rejects or contacts anyone; a recruiter makes every decision
A recruiter reviews every application it ranks low
Job-related criteria only; personal details hidden, protected characteristics never used or inferred
Every score, quote and decision is logged
Checked against local hiring law, such as NYC’s rules on automated hiring tools, and audited for bias

What to measure

Time from application to a first proper read
Low-ranked applications the recruiter moves up
Shortlisted people who pass the first interview
Selection rates across groups, in each bias audit

Who builds and runs it

A recruiter or talent lead who owns the scorecards, an engineer for the ATS connection and redaction step in weeks 1–2, and HR or legal to sign off the law check and audit plan. After that, the recruiter writes each role’s scorecard with the hiring manager.

Needs

01Company Brain

Next to build

20New-Hire Onboarding 02Meeting-to-Action

// Want help building it?

We’ll build your Candidate Shortlist System with your team.

In 30 minutes with our founder, we’ll look at your tools and your week, and work out what your first AI systems should be and how long they’ll take.

Book a Strategy Call

Free 30-minute call with our founder · No commitmentPallav Kaushish Co-Founder, AIxGrowth