Three mistakes that were reported

These three cases were reported by researchers and news writers. Each taught a check.

A hiring program. A company built a program to rate job applications. A resume lists a person's school and work. The program learned from ten years of past resumes, most from men. It rated a resume lower if the resume had the word "women's". The company changed the program. It was not sure the fix was enough. So it ended the project.

A face program. In 2018, researchers tested three face programs. Each said if a photo showed a woman or a man. Errors were highest for darker-skinned women, up to about 35 in 100. For lighter-skinned men, errors were under 1 in 100. The fault is in the programs, not in the groups tested. Training photos are the photos a program learns from. Two common test photo sets held mostly lighter-skinned faces. The makers did not share their training photos. The cause may be who was in them. A later test checked many programs. Most showed gaps between groups.

A health program. A program picked patients for extra care. It used past spending on a patient as a stand-in for illness. A stand-in is something you measure instead of the real thing. Some sick patients spent less. Cost or a far-away hospital can keep people from care. So the program scored those patients as less sick. Researchers reported this in 2019. A better measure made the program fairer.

The rule

Check who is in the data, what it copies, what it measures.

Then check again while the program is in use.

Data that includes many kinds of people is fairer. We call it balanced data.

Rank six patients two ways.