Auditing AI is accessible for to audiences from high school and above. It is currently assigned in university courses and because of its wide range of examples it can be useful in business; computer science; information science; science & technology studies; public policy; sociology; applied ethics; and media/communications.
Discussion Questions
Suitable for book groups and seminars.
Chapter One
- What factors beyond health should be prioritized in an automated kidney transplant allocation system and why? (e.g., youth? age? wealth?)
- Is “Crandall’s Complaint” (referring to CEO Robert Crandall) a reasonable complaint? Explain.
- If we agree that we need a contemporary version of a Civil Aeronautics Board “display bias” regulation for AI, what should it say? (Or, is it even possible?)
- Is the Department of Justice action against Facebook an example of effective regulation? Why or why not? If not, what could be improved?
Chapter Two
- What are parallels between fair housing committees of the past and today’s efforts to control AI?
- Attack or defend the statement: Auditing AI is pointless because it can’t open the black box.
- Is it possible to make auditing AI routine, but avoid creating a rubber stamp or “audit-washing?” Explain your reasoning.
- What makes an AI audit independent? What kind of AI audits need to be independent?
Chapter Three
- What stakes (or consequences) of AI are required to justify what kinds of AI auditing?
- What burdens can AI audits put on an AI system? How can these be mitigated?
- What is the significance of sock puppets in AI auditing? What advantages and challenges do they present?
- What are strategies for scaling an AI audit? When is this important?
Chapter Four
- What challenges does the muffin-Chihuahua problem present for auditing?
- Give an example of a plausible AI audit design (real or hypothetical) that seems strong–but also may be asking the wrong questions.
- What are some data problems that have derailed AI audits? Are these fixable?
Chapter Five
- Attack or defend the statement: We can’t perform AI audits here because if we find anything it will increase our liability.
- How do AI audits play a role in resistance and/or circumvention?
- Why are some well-known AI audits associated with abandoning a system entirely? How do AI audits sometimes lead to this result?
Chapter Six
- How can organizational culture be made consistent with auditing?
- Explain circumstances where it makes sense for a corporation or government agency to offer a bounty to encourage external AI auditors.
- How can an external, unapproved audit be ethical? (That is, it is an adversarial procedure that spends resources of the target and might have severe negative consequences for the target.) Or: Under what circumstances would it definitely be unethical?