Platform Game Programming with AI

Lessons

Senior 3 · ICT

Platform Game Programming with AI

4 lessons

Lesson 4 of 4

Teachable Machine and AI Ethics in Games

60 min

Control the game with your hand

So far your enemy follows rules you wrote. Now you will add a different kind of AI: a machine-learning model that you train yourself. Hold your hand flat to the camera and the player walks right; raise a fist and it jumps. Nobody writes a rule for what a fist looks like: the model learns it from examples. In S1 you learned that machine learning finds patterns in training data. Here you will collect the data, train the model, test it, and use its predictions in Scratch. Then you will look at the ethical questions any AI game raises.

Definition

Training data: the examples a machine-learning model learns from. Here, webcam pictures of each hand gesture.

Class (label): one category the model can recognise, such as right, jump or nothing.

Model: the result of training: a program that takes a new input (a camera picture) and predicts its class.

Confidence: how sure the model is of a prediction, from 0 (not at all) to 1 (completely), often shown as a percentage.

Teachable Machine: a free website from Google for training image, sound and pose models in the web browser, without writing code.

Part A: training the model

Open teachablemachine.withgoogle.com in Chrome and click Get Started → Image Project → Standard image model.
Rename Class 1 to right and Class 2 to jump. Click Add a class and name it nothing.
In the right class, click Webcam, allow the camera, and use Hold to Record while you show a flat hand, moving it a little and changing the distance. Collect about 150 pictures.
Do the same for jump (a fist) and for nothing (no hand: just you, or the empty background). Try to collect about the same number for each class.
Click Train Model and wait; keep the tab open while it trains.
Test it in Preview: the bars show the confidence for each class. Try different hands, distances and light.
Click Export Model, keep the Tensorflow.js tab, choose Upload (shareable link), then Upload my model. Copy the link that appears.

Training happens inside your browser. Google says the pictures stay on your computer unless you choose to save the project to Google Drive, and that uploading publishes the trained model, not the pictures. Anyone who has the link can use the model.

Part B: using the model in Scratch

The standard Scratch website has no Teachable Machine blocks. A free, experimental version of Scratch 3 called Stretch3 adds an extension called TM2Scratch that does.

In Chrome, open stretch3.github.io. Load your platformer: File → Load from your computer (first download it from Scratch with File → Save to your computer).
Click the Add Extension button (bottom left) and choose TM2Scratch. Allow the camera.
Add these scripts to the Player. Paste your link into the image classification model URL block.

Then change two tests in the main player script, so the AI and the keyboard both work:

Block diagrams drawn by Ganzaa. The teal blocks belong to the TM2Scratch extension.

Each time the model recognises a gesture with at least the threshold confidence, the matching when received image label script runs and sets the variable ai move. The player script then treats ai move like a pressed key. Keep your hand in the middle of the camera picture: TM2Scratch uses the centre of the image.

Worked example 1: testing the model

Kamanzi trained his model with his own hand only. He tested it 20 times per class, with three classmates' hands:

Shown	Predicted right	Predicted jump	Predicted nothing	Correct
right	18	1	1	18 of 20
jump	4	13	3	13 of 20
nothing	0	1	19	19 of 20

Accuracy: $\frac{18 + 13 + 19}{60} = \frac{50}{60} \approx 83\%$ . The weak class is jump (65%), mostly mistaken for right. Looking at his samples, he finds that all his fist pictures were taken close to the camera, in bright light, and only with his own hand. He adds 100 fist pictures from three classmates, at different distances and in dimmer light, retrains, and tests again: jump rises to 18 of 20.

Worked example 2: choosing a confidence threshold

With a threshold of 0.5, the player sometimes jumps by itself when Kamanzi scratches his face: the model is 55% sure it sees a fist. With 0.9, the player often ignores real fists, because confidence on a tilted fist is 80–85%. He settles on 0.8: the false jumps stop, and real fists still work. A threshold is a trade-off: higher means fewer false actions but more missed ones.

AI ethics in game programming

Every game that uses AI, or is made with AI tools, raises questions about people. Four matter most for your projects:

Issue	The question	In your platformer	Good practice
Fairness	Does the AI treat every player equally, and is the challenge fair?	A gesture model trained on one person's hand may fail for players with different hands, skin tones or lighting. An enemy faster than the player is unbeatable.	Train on varied data from many people; test with people who were not in the data; always keep a keyboard option; tune enemies so skill decides.
Privacy	What personal data does the game collect, and who sees it?	Webcam pictures of classmates are personal data.	Ask permission before recording anyone; keep samples on your computer; upload only the model; never share pictures of others. Rwanda's data protection law (Law N° 058 of 2021) applies to personal data.
Copyright	Who owns the sprites, music and code, including what an AI tool made?	Sprites copied from a commercial game, or music from a famous song, belong to others.	Make your own art, use the Scratch library, or openly licensed work with credit; credit remixed Scratch projects; say when an AI tool helped make art or music, and check the tool's terms.
Inclusive storytelling	Who appears in the game, and how?	A hero who always looks the same, or villains drawn from one group of people.	Let players choose a character; show varied people with respect; set stories in places your players know; avoid stereotypes.

Warning

Before you record anyone

Ask permission from every classmate whose hand or face will be recorded, and explain what the pictures are for.
Do not save the project to cloud storage with their pictures in it unless they agree.
A model trained on faces can be misused. In class, train on hand gestures or objects, not on recognising who someone is.

Google: Teachable Machine 2.0, making AI easier for everyone. Shows how people gather examples, train a model in the browser and use it in their own projects.

✦

Hands-on: an AI-controlled level

Train a Teachable Machine model with the classes right, jump and nothing (add left if you like). Collect samples from at least three people, with their permission.
Test the model with someone who was not in the training data: 10 tries per class. Record the results in a table and calculate the accuracy.
Improve the weakest class with more varied samples; retrain; test again.
Upload the model, open your platformer in Stretch3, add TM2Scratch and the scripts from Part B. Set a confidence threshold that avoids false actions.
Play one level using only gestures. Is it harder or easier than the keyboard? Why?
Write a short ethics statement for your game (5–6 lines) covering fairness, privacy, copyright and inclusive storytelling. Put it in the project's Notes and Credits.

Practice

Define training data, class, model and confidence.
List the main steps of training and exporting an image model in Teachable Machine.
Why does the model need a nothing class?
Explain the difference between the enemy's rule-based AI and the Teachable Machine model.
A model is tested 15 times per class on 4 classes, with 13, 11, 15 and 9 correct. Calculate the accuracy to the nearest whole per cent, and name the class to improve first.
What happens if the confidence threshold is set to 0.3? And to 0.99?
A pupil's gesture game works perfectly for her but badly for her friend in another classroom. Analyse three possible reasons.
A pupil wants to use an AI image generator to make all the sprites, copy a popular song as background music, and train a face model to recognise each classmate as a player. Evaluate the plan against the four ethical issues of this lesson.

Show answersHide answers

Training data: the examples a model learns from. Class: a category the model can recognise. Model: the trained program that predicts the class of a new input. Confidence: how sure the model is of a prediction, from 0 to 1.
Create an image project; name the classes; collect samples for each class with the webcam or uploads; Train Model; test in Preview; Export Model → Upload (shareable link) → Upload my model; copy the link.
Without it, every camera picture must be called right or jump, so the player would move even when no gesture is shown.
The enemy follows rules a programmer wrote (if distance < 120, chase). The Teachable Machine model learned its own pattern from labelled examples; nobody wrote rules for what a fist looks like.
$\frac{13 + 11 + 15 + 9}{60} = \frac{48}{60} = 80\%$ . Improve the class with 9 correct first.
0.3: the model acts even when unsure, so there are many false actions (the player moves by itself). 0.99: the model is almost never that sure, so real gestures are often ignored.
Any three: the training data came only from her own hand (different hand size, shape or skin tone); different lighting in the other room; a different background; a different camera or distance; the friend's hand not in the centre of the picture.
Copyright: the popular song belongs to others and cannot be used without permission; AI-generated sprites may be allowed, but she must check the tool's terms and say that AI made them. Privacy: a face model that identifies classmates collects sensitive personal data and needs everyone's permission; gestures would do the job without identifying anyone. Fairness: face models trained on few people may work worse for some players; a keyboard option is needed. Inclusive storytelling: check that the AI-generated characters show varied people without stereotypes. Overall: change the music and replace face recognition with gestures.

1Control the game with your hand

2Part A: training the model

3Part B: using the model in Scratch

4AI ethics in game programming

5Hands-on: an AI-controlled level

Control the game with your hand

Part A: training the model

Part B: using the model in Scratch

AI ethics in game programming

Hands-on: an AI-controlled level