How a computer reads
A computer cannot read "She PLAYS well!" the way you do. How does it find the words?
- A message arrives: She PLAYS well!
- Tokenisation cuts it into words, called tokens.
- Lower-casing makes every letter small.
- Stemming chops word ends: plays becomes play.
- Then the program studies grammar and meaning.
It cuts the message into words, called tokens.
Natural language processing, or NLP, is AI for spoken and written language.
Lower-casing and stemming help the computer match PLAYS, plays and play.
Many programs also drop punctuation, such as ! and commas. Grammar is how words fit together. Meaning is what the words say. One word can mean two things. A date can be a fruit or a day on the calendar.
NLP is used in translation tools, chatbots, spam filters and voice helpers.
NLP cuts text into words and tidies them. Then it works out grammar and meaning.
Watch out!
Chopping word ends is rough. It can leave a piece that is not a word.
Switch the steps on. Find both words in all three messages. Watch one message change.
Loading lab…