Training a Chatbot with ChatterBot and Improving Its Responses

1A bot that learns from examples

The keyword bot from the last lesson only knows the words you typed into its keyword lists. A retrieval-based bot works differently: you give it example conversations, it stores them, and when a new message arrives it finds the stored message that is most similar and gives the reply that followed it. You improve it by giving it more and better examples, not by editing rules. The Python library ChatterBot works this way, and in this lesson you train it on the school office's questions, measure where it goes wrong, and make it fail safely.

2Setting it up

Activate the virtual environment from lesson 2, then install ChatterBot and the small English language model of spaCy, the NLP library ChatterBot uses to tag words. The commands below were tested with ChatterBot 1.2.15 on Python 3.12; ChatterBot 1.2 needs Python 3.10 or newer.

(.venv) $ pip install chatterbot
(.venv) $ python -m spacy download en_core_web_sm

If you skip the second command, ChatterBot stops with the message The Spacy model for "English" language is missing, which tells you exactly what to run.

3Training

Save this as train_bot.py and run it once. It creates a database file, schoolbot.sqlite3, which is the bot's memory. Notice that each answer is trained with several ways of asking the question: this is the single most effective way to improve a retrieval bot.

from chatterbot import ChatBot
from chatterbot.trainers import ListTrainer

bot = ChatBot(
    "SchoolBot",
    storage_adapter="chatterbot.storage.SQLStorageAdapter",
    database_uri="sqlite:///schoolbot.sqlite3",       # the bot's memory, a file
    logic_adapters=[{
        "import_path": "chatterbot.logic.BestMatch",
        "default_response": "Sorry, I do not know that yet. Please ask the school office.",
        "maximum_similarity_threshold": 0.90,
    }],
)

# Each answer is trained with several ways of asking the question
FAQ = {
    "Term 1 begins on Monday 7 September.": [
        "When does term start?", "When do classes begin?",
        "When does school open?", "What is the first day of term?"],
    "The fee for one term is 95,000 Frw.": [
        "How much are the school fees?", "How much are the fees?",
        "What are the fees?", "How much does one term cost?"],
    "Bring your last report, a copy of your birth certificate and two photos.": [
        "What documents should I bring?", "Which documents do I need?",
        "What papers are needed to join?"],
}

trainer = ListTrainer(bot, show_training_progress=False)
for answer, questions in FAQ.items():
    for question in questions:
        trainer.train([question, answer])
print("Trained on", sum(len(q) for q in FAQ.values()), "question-answer pairs")

Output:

Trained on 11 question-answer pairs

4Asking, safely

Save this as ask_bot.py. It opens the same database in read-only mode and wraps every request in checks: empty or very long input, unexpected errors, and low confidence. Questions it cannot answer confidently are logged to unanswered.txt so that the office can add them to the training data.

from chatterbot import ChatBot

FALLBACK = "Sorry, I do not know that yet. Please ask the school office."

bot = ChatBot("SchoolBot",
              storage_adapter="chatterbot.storage.SQLStorageAdapter",
              database_uri="sqlite:///schoolbot.sqlite3",
              logic_adapters=[{
                  "import_path": "chatterbot.logic.BestMatch",
                  "default_response": FALLBACK,
                  "maximum_similarity_threshold": 0.90,
              }],
              read_only=True)          # users' messages never change what it knows


def answer(question, min_confidence=0.5):
    question = question.strip()
    if not question:
        return "Please type a question."
    if len(question) > 300:
        return "That message is too long. Please ask one short question."
    try:
        response = bot.get_response(question)
    except Exception:                      # never let the user see a crash
        return "Something went wrong. Please try again later."
    if response.confidence < min_confidence:
        with open("unanswered.txt", "a", encoding="utf-8") as log:
            log.write(question + "\n")     # for the office to review
        return FALLBACK
    return str(response)


if __name__ == "__main__":
    for q in ["When do classes start?", "What are the fees?", "What documents do I need?",
              "How much is the uniform?", "", "When does the school open?"]:
        print(f"{q!r:30} -> {answer(q)}")

Output, as run:

'When do classes start?'       -> Term 1 begins on Monday 7 September.
'What are the fees?'           -> The fee for one term is 95,000 Frw.
'What documents do I need?'    -> Bring your last report, a copy of your birth certificate and two photos.
'How much is the uniform?'     -> The fee for one term is 95,000 Frw.
''                             -> Please type a question.
'When does the school open?'   -> Term 1 begins on Monday 7 September.

Five answers are right. One is badly wrong: "How much is the uniform?" was matched to the fees question, because "How much is…" looks like "How much are…", and the bot was never taught anything about uniforms. It answered with confidence 0.70, above the threshold of 0.5, so the safety check did not stop it.

5Improving responses: a checklist

A warning from history: in 2016 Microsoft released a chatbot called Tay that learned from conversations with the public on Twitter. Within a day users had taught it to post offensive messages, and Microsoft took it offline. A bot that learns from unreviewed input learns whatever it is fed.

6Spelling tolerance for the keyword bot

Python's built-in difflib module can find the closest known word to a misspelt one. Added to the keyword bot from lesson 3, it lets "feez" match "fee":

from difflib import get_close_matches
from schoolbot import INTENTS, normalise

VOCABULARY = set().union(*(i["keywords"] for i in INTENTS.values()))

def correct(words):
    """Replace each unknown word by the closest keyword, if one is close enough."""
    fixed = set()
    for w in words:
        match = get_close_matches(w, VOCABULARY, n=1, cutoff=0.8)
        fixed.add(match[0] if match else w)
    return fixed

for msg in ["feez?", "documants to bring", "wen does term begn"]:
    print(f"{msg!r:22} -> {sorted(correct(normalise(msg)))}")

'feez?'                -> ['fee']
'documants to bring'   -> ['bring', 'documents', 'to']
'wen does term begn'   -> ['begin', 'does', 'term', 'wen']

"wen" was not corrected, because no keyword is close to it: "when" is not a keyword. The cutoff=0.8 sets how similar a word must be; lower values correct more words but also make more wrong corrections.

7Hands-on: train, break, improve

1. Install ChatterBot and the spaCy model in your environment. Run train_bot.py, then ask_bot.py.
2. Write a test list of 12 questions, including 4 the bot was never trained on. Record each answer and its confidence (print response.confidence).
3. Find the threshold that gives the fewest wrong answers on your test list. How many questions does it leave unanswered?
4. Add a uniform answer and at least two new phrasings for each existing answer. Delete schoolbot.sqlite3, retrain, and re-run the tests.
5. Compare the number of correct, wrong and refused answers before and after, in a table.

Check: after improvement, the bot gives no wrong answer on your test list, and every refused question appears in unanswered.txt.