Natural Language Processing
Senior 6 · ICT
Natural Language Processing
Lesson 1 of 5
22 min
Type "Is umuganda this Saturday?" into a phone and a well-built assistant understands that umuganda is Rwanda's monthly community work, held on the last Saturday of the month, and that you want a yes-or-no answer about a date. A child understands the question instantly. For a computer, the sentence is just a row of characters. Turning those characters into meaning, and meaning back into words, is the field called natural language processing. It is behind spam filters, search engines, phone keyboards that guess your next word, machine translation and every chatbot you will build in this unit.
Definition
Natural language: a language that people use with each other, such as Kinyarwanda, English, French or Kiswahili, as opposed to a programming language.
Natural language processing (NLP): the branch of computer science and artificial intelligence that makes computers analyse, understand and produce natural language, written or spoken.
Token: one unit of text after splitting, usually a word, a punctuation mark or a piece of a word. Tokenisation: splitting text into tokens.
Corpus (plural corpora): a large, organised collection of texts used to study language or to train NLP systems.
Before machine learning took over, most NLP systems processed text in fixed stages. Each stage is still worth knowing, because each solves a real problem.
| Step | What it does | Example |
|---|---|---|
| Tokenisation | Splits text into tokens | "Residents clean roads." → Residents, clean, roads, . |
| Lower-casing | Makes "Roads" and "roads" the same token | Roads → roads |
| Stop-word removal | Drops very common words that carry little meaning for the task | the, of, and |
| Stemming | Cuts words to a rough root by rules | studies → studi |
| Lemmatisation | Finds the dictionary form, using vocabulary and grammar | studies → study; better → good |
| Part-of-speech tagging | Labels each word as noun, verb, adjective… | clean (verb) roads (noun) |
| Named-entity recognition | Finds names of people, places, organisations, dates | "Huye" → place; "last Saturday" → date |
| Sentiment analysis | Judges whether a text is positive, negative or neutral | "The bus was late and dirty." → negative |
This program, run with python3, tokenises a short text about umuganda, removes stop words, applies a very crude stemmer and counts the most frequent stems. It uses only Python's built-in modules.
import re
from collections import Counter
text = ("Umuganda takes place on the last Saturday of every month. "
"Residents clean roads, build classrooms and plant trees. "
"After the work, residents meet to discuss community plans.")
# 1. Split into sentences, then into lower-case word tokens
sentences = re.split(r"(?<=[.!?])\s+", text)
tokens = re.findall(r"[a-z]+", text.lower())
# 2. Remove stop words: very common words that carry little meaning
STOP = {"the", "on", "of", "every", "and", "to", "after", "takes", "place"}
content = [t for t in tokens if t not in STOP]
# 3. A crude stemmer: strip a final "s" so "trees" and "tree" match
stems = [t[:-1] if t.endswith("s") and len(t) > 3 else t for t in content]
print(len(sentences), "sentences,", len(tokens), "tokens")
print(content[:8])
print(Counter(stems).most_common(3))
Output:
3 sentences, 27 tokens
['umuganda', 'last', 'saturday', 'month', 'residents', 'clean', 'roads', 'build']
[('resident', 2), ('umuganda', 1), ('last', 1)]The crude stemmer made "residents" into "resident", so the two mentions are counted together. It would also turn "class" into "clas" and "bus" would only survive because it has three letters, which is why real systems use proper lemmatisers from libraries such as spaCy or NLTK.
Worked example 1: why simple rules fail
A transport company scores customer comments by counting positive and negative words. Here is the method, and its results on three comments:
POSITIVE = {"good", "great", "clean", "helpful", "friendly"}
NEGATIVE = {"bad", "dirty", "slow", "rude", "late"}
def score(sentence):
words = sentence.lower().replace(".", "").split()
return sum((w in POSITIVE) - (w in NEGATIVE) for w in words)
for s in ["The bus was clean and the driver was friendly.",
"The bus was late and dirty.",
"The bus was not clean."]:
print(score(s), s)
2 The bus was clean and the driver was friendly.
-2 The bus was late and dirty.
1 The bus was not clean.| Approach | Main period | How it works | Strength and weakness |
|---|---|---|---|
| Rule-based | 1950s to 1980s | Experts write grammar rules and dictionaries | Predictable; breaks on anything the rules did not foresee |
| Statistical machine learning | 1990s to 2010s | Counts patterns in large corpora | Handles variety; needs a lot of labelled data |
| Neural networks and transformers | 2010s to today | Deep networks learn word meanings as vectors; the transformer design (2017) underlies today's large language models | Very fluent; can produce confident errors and is costly to train |
In unit 2 you met large language models as website generators. They are NLP systems: the same next-token prediction that writes HTML also answers questions, summarises and translates.
NLP works best for languages with huge amounts of digital text, above all English. Languages with less digital text are called low-resource languages, and Kinyarwanda has long been one of them. Kinyarwanda also builds long words from many parts: ndagukunda ("I love you") packs a subject, a tense marker, an object and a verb into one word. An English-style tokeniser that splits only at spaces treats every such combination as a separate, rare word. Progress depends on data: Google Translate added Kinyarwanda in 2020, and Rwandan volunteers have recorded many hours of spoken Kinyarwanda for Mozilla's open Common Voice speech dataset, which researchers use to train speech recognition.
CrashCourse: Natural Language Processing (Crash Course AI #7). Watch how words are turned into numbers (vectors) and why meaning depends on the words around them; compare with the word-counting program above.
Warning
Text data is often personal data. Messages, comments and chat logs can contain names, phone numbers and private problems. Before you collect or analyse them, get permission, remove what you do not need, and store them securely. Rwanda's Law N° 058 of 2021 on the protection of personal data and privacy applies to text just as it does to forms.
Tip
NLP in one sentence: split it, clean it, label it, use it. Tokenise, normalise, tag, then do the task. When an NLP system fails, check each stage in that order.
STOP until the top five stems tell you what the text is about.POSITIVE and NEGATIVE and score five comments of your own. Write two comments that the scorer gets wrong, and explain why.score so that a word straight after "not" counts with the opposite sign. Test it on "The bus was not clean."Check: your negation version scores "The bus was not clean." as negative, and you can name one comment it still gets wrong.
Practice
Sources and credits
This lesson adapts CC BY-SA material and is shared under CC BY-SA 4.0.