Smart Robots

Lessons

Senior 6 · ICT

Smart Robots

5 lessons

Lesson 5 of 5

AI Ethics in Robotics

25 min

When a robot fails

In March 2018, in Tempe, Arizona, a test car driving itself for the company Uber struck and killed a woman who was walking her bicycle across the road at night. The US National Transportation Safety Board investigated. It found that the car's sensors had detected her several seconds before the crash, but the software kept changing its mind about what she was and did not predict that she would cross its path; the car's own emergency braking had been switched off during testing; and the safety driver, whose job was to take over, was looking at a phone. No single part failed alone. The design, the testing rules, the company's safety culture and a person's attention all played a part. Robots act in the physical world, so their mistakes can hurt people. Deciding how they should be built, tested, used and governed is the subject of robot ethics.

Definition

Ethics: the study of what is right and wrong, and of how we should act.

AI ethics (in robotics): principles and practices that make sure robots and AI systems are safe, fair, transparent and respectful of human rights, and that people remain responsible for them.

Accountability: the duty to explain a decision or outcome and to accept responsibility for it. Algorithmic bias: systematic errors in an AI system that disadvantage particular groups of people, often because of unrepresentative training data.

Principles for ethical robots

Many organisations have written principles for AI. In November 2021 the member states of UNESCO adopted the Recommendation on the Ethics of Artificial Intelligence, the first global standard of its kind. Its ideas, and those of most national policies, can be summarised in eight principles:

Principle	Question it asks	Example for a robot
Safety	Can it harm people, and how is harm prevented?	Emergency stop; speed limits near people; testing before release
Human oversight	Can a person understand, supervise and override it?	An operator can take control of a delivery robot at any time
Accountability	Who is responsible when it goes wrong?	A named operator, logs of every decision, a way to complain
Transparency	Do people know a robot or AI is involved, and roughly how it decides?	A sign: "This junction is controlled by cameras that count vehicles"
Privacy	What personal data does it collect, and why?	Count cars without storing faces or number plates
Fairness	Does it work equally well for everyone?	A people-detector tested on people of all skin tones, ages and sizes
Benefit and jobs	Who gains, who loses, and what happens to their work?	Retraining for workers whose tasks are automated
Sustainability	What does it cost the environment?	Energy use, batteries, repair and recycling

In Rwanda, the Law N° 058 of 2021 relating to the protection of personal data and privacy applies to any robot or system that collects personal data, such as camera images of people, and the government adopted a National Artificial Intelligence Policy in 2023 to guide the responsible use of AI.

Fiction and reality

The science-fiction writer Isaac Asimov imagined "Three Laws of Robotics" in a 1942 story: a robot may not injure a human; it must obey humans unless that would injure someone; it must protect itself unless that breaks the first two laws. They are a good story device but a poor engineering guide. "Injure" and "human" are not things a sensor can measure directly, a robot may not know the consequences of its actions, and the laws say nothing about privacy, fairness or who is responsible. Real robot safety is built from measurable requirements, standards (for example the international standards ISO 10218 for industrial robots and ISO/TS 15066 for collaborative robots), testing and law.

Bias: when "it works" means "it works for some"

In 2018 the researchers Joy Buolamwini and Timnit Gebru tested commercial face-analysis systems that classified people's gender from photos. The systems were wrong for less than 1% of lighter-skinned men but for up to about 35% of darker-skinned women. The cause was training and test data that under-represented darker-skinned faces. A robot whose camera recognises some people less reliably than others is not only unfair; it may also be unsafe for exactly those people. For robots used in Rwanda, training and testing data must represent the people who will actually meet the robot.

Who is responsible?

A robot cannot be put on trial. When one causes harm, responsibility lies with people and organisations, and usually with several of them: the designers who chose its rules and training data, the manufacturer who built and tested it, the operator who decided where and how to use it, and the person supervising it on site. Clear records (logs of what the robot sensed and decided), clear roles, and laws that say who must pay compensation make it possible to learn from failures and to put things right.

Worked example 1: an ethical review of a gate robot

A school proposes a robot at its gate that uses face recognition to record which students arrive late and to check their uniforms, and sends a daily list to class teachers. Review it in five steps.

Who is affected? Students (mostly children), parents, teachers, visitors who walk past the camera.
Benefits: accurate late records; less work for the teacher on duty.
Harms and risks: faces of children are sensitive personal data; a recognition error could punish the wrong student, and errors may be more frequent for some students; images could leak or be used for other purposes; students may feel constantly watched.
Rights and law: under Law N° 058 of 2021, collecting this data needs a lawful basis and clear purpose, minimal data, secure storage and the right to see and correct records; parents' consent is needed for children.
Safeguards and alternatives: a simpler method (a card scanned at the gate) records lateness without faces; if a camera is used, store no images, let a person check every flagged case before any punishment, publish what is collected and for how long, and review accuracy for all students.

Conclusion: as proposed, the risks outweigh the benefits; a card system with human checks meets the goal with far less harm.

Worked example 2: dividing responsibility

A sidewalk delivery robot, operated by a company, hits a child who runs out from behind a parked car. The robot's logs show its camera detected the child 0.4 s before impact; its maximum speed near pedestrians had been set by the operator to 10 km/h, above the manufacturer's recommended 6 km/h; the remote supervisor was watching twelve robots at once.

Operator: chose a speed above the recommendation and gave one supervisor too many robots. Major responsibility.
Manufacturer: allowed the speed limit to be raised above its own recommendation; should consider a hard limit.
Designers: should check whether the system could detect people emerging from behind obstacles earlier, for example with additional sensors.
Regulator: should set rules for speeds near pedestrians and for the number of robots per supervisor.

The logs made this analysis possible: without them, nobody could have learned what went wrong.

TED-Ed: The ethical dilemma of self-driving cars, Patrick Lin. Watch for how a decision programmed in advance differs from a split-second human reaction, and decide who should choose the rules: engineers, companies, governments or the public.

Warning

Discussing harm without causing it. When you research robot accidents, use official investigation reports and reputable news, not graphic videos or images. Discuss real people who were harmed with respect. The aim is to learn how to prevent harm, not to sensationalise it.

Tip

The "front page" test. Before deploying a robot, imagine a newspaper story about its worst plausible day. If you would not be comfortable explaining your design choices in that story, change them now.

✦

Hands-on: an ethics review board

In groups of four, take one system from this unit: the smart traffic controller, the hospital delivery robot, the cleaning robot that keeps learning, or your own design from lesson 3.
Each member takes a role: designer, operator, a member of the public affected by it, and a regulator.
Carry out the five-step review from worked example 1. Each role must name at least one concern that the others did not.
Agree on three safeguards and one thing the system must never do. Present your decision in two minutes, with your reasons.

Check: your review names who is affected, at least two harms, the relevant law or principle, and safeguards that reduce each harm.

Practice

Define accountability and algorithmic bias, with an example of each in robotics.
Name four of the eight principles in this lesson, and for each give one design feature that supports it.
Give two reasons why Asimov's Three Laws are not enough to make real robots safe.
What did the 2018 study by Buolamwini and Gebru find, and what was the main cause?
In the 2018 Tempe crash, identify three factors that contributed and who was responsible for each.
Which Rwandan law applies to a robot that records images of people, and name two things it requires.
A company proposes robots that decide by themselves whether to fire weapons in a war. Evaluate the proposal using at least three principles from this lesson.

Show answersHide answers

Accountability: the duty to explain and take responsibility for an outcome; e.g. an operator who keeps logs and answers complaints about its delivery robots. Algorithmic bias: systematic errors that disadvantage a group; e.g. a people-detector that misses darker-skinned pedestrians more often.
For example: safety (emergency stop), human oversight (manual override), privacy (no images stored), fairness (testing on diverse people), transparency (public signs), accountability (logs and a named operator).
Terms such as "injure" and "human" cannot be measured directly by sensors; a robot may not foresee consequences; the laws ignore privacy, fairness and responsibility; real safety needs measurable requirements, standards and testing.
Commercial face-analysis systems misclassified gender far more often for darker-skinned women (up to about 35%) than for lighter-skinned men (under 1%). The main cause was unrepresentative training and test data.
The software failed to classify the pedestrian correctly and predict her path (designers and the company); emergency braking had been disabled during testing (the company's testing decisions); the safety driver was distracted by a phone (the driver, and the company's supervision and safety culture).
Law N° 058 of 2021 relating to the protection of personal data and privacy. It requires, for example, a lawful basis and clear purpose, collecting only necessary data, secure storage, and respecting people's rights to access and correct their data.
It conflicts with human oversight (no person decides to take a life), accountability (no clear person is responsible for a wrongful killing), safety and fairness (recognition errors, including bias, could kill civilians), and transparency. Many states and organisations call for such weapons to be banned or strictly limited, and the question is discussed at the United Nations. Meaningful human control over the use of force should be required.

1When a robot fails

2Principles for ethical robots

3Fiction and reality

4Bias: when "it works" means "it works for some"

5Who is responsible?

6Hands-on: an ethics review board