Natural Language Processing

Lessons

Senior 6 · ICT

Natural Language Processing

5 lessons

Lesson 3 of 5

Creating a Simple Chatbot in Python

25 min

Start with something you can fully explain

Before training anything, it pays to build a chatbot whose every decision you can trace by hand. The school office's list of common questions from the last lesson is enough: each question type becomes an intent, each intent gets a set of keywords and an approved answer, and the bot picks the intent whose keywords best match the message. This is a rule-based chatbot. It is limited, but it is honest: it never invents an answer, and when it fails you can see exactly why.

Definition

Keyword matching: choosing an intent by counting how many of its listed keywords appear in the user's message.

Normalisation: changing text into a standard form before comparing it, for example lower case with punctuation removed, so that "Fees?" and "fees" match.

Python set: an unordered collection with no repeats, written {"fee", "pay"}. The operator & gives the intersection of two sets: the items in both.

The design

Three data structures do all the work: a dictionary INTENTS that maps each intent name to its keywords and replies; a FALLBACK message; and the set of words in the message. The score of an intent is the size of the intersection of two sets, $|W \cap K|$ , where $W$ is the set of words in the message and $K$ the intent's keywords.

The program

Save this as schoolbot.py in your schoolbot folder. The school facts in the replies are illustrative; a real bot must use the school's approved information.

import random
import string

INTENTS = {
    "greeting": {
        "keywords": {"hello", "hi", "muraho", "morning", "afternoon"},
        "replies": ["Muraho! Ask me about fees, term dates, documents or visiting days."],
    },
    "fees": {
        "keywords": {"fee", "fees", "pay", "payment", "cost", "much"},
        "replies": ["The fee for one term is 95,000 Frw. Pay at the bank or by "
                    "mobile money, quoting the student's code."],
    },
    "term_dates": {
        "keywords": {"start", "starts", "begin", "begins", "term", "reopen", "date"},
        "replies": ["Term 1 begins on Monday 7 September. Boarders arrive the day before."],
    },
    "documents": {
        "keywords": {"document", "documents", "bring", "report", "certificate", "need"},
        "replies": ["Bring the last school report, a copy of the birth certificate "
                    "and two passport photos."],
    },
    "visiting": {
        "keywords": {"visit", "visiting", "parents", "sunday"},
        "replies": ["Parents may visit on the first Sunday of each month, 10:00 to 15:00."],
    },
    "goodbye": {
        "keywords": {"bye", "goodbye", "thanks", "thank", "murakoze"},
        "replies": ["Murakoze! Goodbye."],
    },
}
FALLBACK = ("Sorry, I did not understand. You can ask about fees, term dates, "
            "documents or visiting days.")


def normalise(text):
    """Lower-case, remove punctuation, return the set of words."""
    text = text.lower().translate(str.maketrans("", "", string.punctuation))
    return set(text.split())


def reply(message):
    """Return (intent, answer) for the intent whose keywords match best."""
    words = normalise(message)
    best, best_score = None, 0
    for name, intent in INTENTS.items():
        score = len(words & intent["keywords"])   # how many keywords appear
        if score > best_score:
            best, best_score = name, score
    if best is None:
        return "fallback", FALLBACK
    return best, random.choice(INTENTS[best]["replies"])


def chat():
    print("SchoolBot: Muraho! Type bye to finish.")
    while True:
        intent, answer = reply(input("You: "))
        print("SchoolBot:", answer)
        if intent == "goodbye":
            break


if __name__ == "__main__":
    chat()

Run it with python schoolbot.py and chat. The if __name__ == "__main__": line means the chat loop starts only when the file is run directly, so another file can import reply to test it.

Testing it automatically

Typing questions by hand every time you change a keyword is slow and easy to forget. A test script sends a fixed list of messages and prints the intent chosen for each. Save it as test_bot.py in the same folder and run it after every change.

from schoolbot import reply, normalise, INTENTS

tests = ["Hi, when does the term begin?",
         "What documents do I need to bring?",
         "How much are the fees?",
         "Can parents visit on Sunday?",
         "What is for lunch today?",
         "When do I need to pay?",
         "Murakoze!"]
for t in tests:
    intent, answer = reply(t)
    print(f"{t!r:40} -> {intent}")

Output:

'Hi, when does the term begin?'          -> term_dates
'What documents do I need to bring?'     -> documents
'How much are the fees?'                 -> fees
'Can parents visit on Sunday?'           -> visiting
'What is for lunch today?'               -> fallback
'When do I need to pay?'                 -> fees
'Murakoze!'                              -> goodbye

Worked example 1: trace one message

Message: "Hi, when does the term begin?"

Normalise: $W = \{\text{hi}, \text{when}, \text{does}, \text{the}, \text{term}, \text{begin}\}$ .
Scores: greeting $|\{\text{hi}\}| = 1$ ; term_dates $|\{\text{term}, \text{begin}\}| = 2$ ; every other intent 0.
The best score is 2 for term_dates, which is above 0, so the bot gives the term-dates reply. The greeting is ignored, which is acceptable here because the question matters more.

Worked example 2: right for the wrong reason

Message: "When do I need to pay?" The test says fees, which is correct. But trace it:

$W = \{\text{when}, \text{do}, \text{i}, \text{need}, \text{to}, \text{pay}\}$ .
fees: $|\{\text{pay}\}| = 1$ . documents: $|\{\text{need}\}| = 1$ . A tie.
The loop only replaces the best intent when the score is strictly greater, so the first intent in the dictionary with score 1 wins. fees comes before documents, so the answer was right only because of the order the intents were written in.
Fix: "need" is too general to be a keyword. Remove it from documents (keep "bring", "document", "documents", "certificate"), and re-run the tests. Now fees wins with 1 against 0, for the right reason.

IBM Technology: Generative vs Rules-Based Chatbots. Watch for the trade-off between control and flexibility, and decide which side of it the keyword bot in this lesson sits on.

Warning

Keyword bots are easily fooled. "I don't want to pay fees, I want to visit" contains "pay" and "fees" (2 points) and "visit" (1 point), so the bot answers about fees. Negation, word order and spelling mistakes ("feez") are invisible to it. The next lesson adds better matching; never let a keyword bot make decisions that matter, such as confirming a payment.

Tip

Write the test before the keyword. When a real user's question gets a wrong answer, first add it to tests, see it fail, then change the keywords until it passes, and check that all the old tests still pass.

✦

Hands-on: your school's bot

Copy schoolbot.py and test_bot.py into your environment and run both.
Replace the illustrative intents with the five or more intents from your list in the last lesson. Use real, approved answers or clear placeholders.
Add at least ten test messages, including two that should fall back.
Find one tie or one wrong answer by experiment, explain it with a trace like the worked examples, and fix it.
Ask a classmate to chat with the bot for three minutes. Add every question it failed on to the tests.

Check: every test message gets the intent you expect, and you can explain the score behind each one.

Practice

What does normalise("Fees, please!") return?
Compute the score of each intent for "Which documents must parents bring on visiting day?" Which intent wins?
Why does the program use score > best_score and not >=? What would change with >=?
Write a new intent uniform with suitable keywords and one reply, in the same format as INTENTS.
A user types "feez?". What happens, and why?
Explain the purpose of if __name__ == "__main__": in this program.
Evaluate the keyword bot for answering parents' questions about their children's marks.

Show answersHide answers

{"fees", "please"} (a set, so the order may differ).
documents: {documents, bring} = 2 (and "need" is not in the message); visiting: {parents, visiting} = 2; others 0. A tie: documents comes first in the dictionary, so it wins. Adding "day" to no list does not help; the bot cannot tell that the question is about documents for visiting day.
With >, the first intent with the top score is kept. With >=, the last intent with the top score would win ties. Either way the tie is settled by order, not meaning, which is why ties should be removed by better keywords.
For example: "uniform": {"keywords": {"uniform", "shirt", "skirt", "trousers", "shoes"}, "replies": ["The uniform list is on the notice board and from the bursar's office."]}
"feez" matches no keyword, every score is 0, and the bot gives the fallback message. Keyword matching needs exact words; spelling mistakes break it.
The chat loop runs only when the file is run directly. When test_bot.py imports reply, the loop does not start, so the tests can run without waiting for input.
Unsuitable. Marks are personal data that require identifying the parent securely, which the bot cannot do; and keyword matching could give the wrong reply. It is acceptable for general information only; marks need a secure, authenticated system.

Sources and credits

Chatbot, Wikipedia contributors, Wikipedia, licensed under CC BY-SA 4.0. Adapted by Ganzaa.
Set (abstract data type), Wikipedia contributors, Wikipedia, licensed under CC BY-SA 4.0. Adapted by Ganzaa.
Video: Generative vs Rules-Based Chatbots, IBM Technology, embedded from YouTube.
The programs, flowchart, worked examples, hands-on task and practice items were written and tested by Ganzaa. The school details in the replies are illustrative.

This lesson adapts CC BY-SA material and is shared under CC BY-SA 4.0.

1Start with something you can fully explain

2The design

3The program

4Testing it automatically

5Hands-on: your school's bot

Start with something you can fully explain

The design

The program

Testing it automatically

Hands-on: your school's bot