Reinforcement Learning (CS 184) – Fall 2022

Module Topic: Reward functions and Ethical Implications
Module Author: Jenna L Donohue

Course Level: Upper-level undergraduate
AY: 2022-2023

Course Description: “Modern Artificial Intelligent (AI) systems often need the ability to make sequential decisions in an unknown, uncertain, possibly hostile environment, by actively interacting with the environment to collect relevant data. Reinforcement Learning (RL) is a general framework that can capture the interactive learning setting and has been used to design intelligent agents that achieve super-human level performance on challenging tasks such as Go, computer games, and robotics manipulation.
This course focuses on basics of Reinforcement Learning. The four main parts of the course are (1) multi-armed bandits, (2) Planning and Control in MDPs, (3) Learning in Large MDPs (function approximation), and (4) advanced topics.” 

Semesters Taught: Fall 2022-23

Tags

• disability (phil)
• The difference principle (phil)
• Mximin (phil)
• Least well off (phil)
• Structural injustice (phil)
• Fairness (phil)
• Utilitarianism (phil)
• Machine learning (CS)
• Reinforcement learning (CS)
• Reward function (CS)
• Utility function (CS)

    Module Overview

In this module, students were introduced to reward functions and their ethical implications. In particular we focused on and discussed how designing a reward function with an eye toward justice and the priority of the least well off might look differently from designing a reward function without those concepts in mind. The module was a mix of whole class discussion, whole class instruction, and group work. In particular, we discussed a RideShare case study and the ethical implications of setting a reward function in that context.

    Connection to Course Technical Material

Reward functions are one of the distinctive features of RL (Reinforcement Learning). They raise ethical questions in their own right, distinct from (but not wholly disconnected from) ethical questions raised by ML generally speaking. This course focuses on RL, so the connection between the ethical implications of reward functions and the course material is clear.

Goals

Module Goals

1. Recognize the importance of specifying a good / correct reward function.
2. Recognize that reward function specification is inherently value-laden.
3. Explain why different reward functions (in words and not yet formalized) benefit different groups, such as those in positions of power, those disadvantaged by the system itself, or the least well off in society, understood more generally.

    Key Philosophical Questions

1. How should we design reward functions for RL systems?
2. What considerations, including technical, societal, and ethical, should we take into account when we are designing our reward functions?
3. How does reward function design change if we think about those who will be made worst off by the adoption of the system? How does it change if we think about prioritizing those who are least well off in society more generally?

Materials

    Key Philosophical Concepts

• John Rawls & Theory of Justice
• least well off, maximin principle, difference principle
• structural injustice

    Assigned Readings

• Excerpt from Rawls, John (1985). “Justice as Fairness: Political not Metaphysical.” Philosophy and Public Affairs
• Excerpt from Rawls, John (2001). Justice as Fairness: A Restatement (pages 59 – 63)
• Hawkins, Andrews (2022). “Uber Doesn’t have to Provide Wheelchair-Accessible Vehicles in Every City, Judge Rules.” The Verge.
• Lu, Donna (2020). “Uber and Lyft Pricing Algorithms Charge More in Non-White Areas.” NewScientist.

Implementation

    Class Agenda

1. Brief introduction to Reward Functions
2. Case Study: Ride Share
3. John Rawls & Least Well Off
4. Return to Case Study and Apply Maximin

    Sample Class Activity

The students were asked to talk in groups about ethical considerations relevant to designing a reward function for an RL algorithm setting prices for a RideShare app. They were also asked to talk in groups about how to apply Rawls’ conception of priority for the least well off in this context. Group discussion of this case study was the main active-learning component of this module.

    Module Assignment

The assignment was a question on a problem set for the students. This method was chosen as a way of making the Embedded EthiCS lectures feel like an integral part of the course: the students are used to problem sets and have them regularly. Assignment Directions: Write a paragraph of 5-7 complete sentences addressing the following question. “Consider the hypothetical RL algorithm that sets dynamic prices for rides on a RideShare app that we discussed in class. Choose one concept and explain why it should or should not be incorporated into a just utility function. Be sure to explain why it does or does not favor the interests of the least well off and whether they would fare better under a different utility function as part of your answer. It is ok to set aside / ignore some of the complicating factors we discussed in class, such as other RideShare company’s prices, shareholder demands, etc. for the purposes of answering this question.”

Lessons Learned

The module was well-received, and participation was high. A few lessons learned:

• Reinforcement learning remains a very new area of study, so present-day applications are few and far between. In this module the Embedded EthiCS TA created a case-study for an application of the technology that doesn’t exist yet. This went mostly ok, but it was crucial to include a specification for both state and action when asking students to think about reward functions. Otherwise the conversation is a bit too abstract and disconnected from the course content.
• Students in this course were eager to talk and had a lot to say. Consider closing discussion a bit sooner to ensure sufficient time to discuss the application of the philosophy to the case study.