I use “audits” and “lab experiments” to study the authoritarian politics of AI. In similar spirit to resume audits on gender bias and audits on government agencies, I use audit experiments to study the political biases of AI systems. In essence, audits entail poking at an institution repeatedly - sending resumes to employers, requests to government officials, or queries to AI systems, while varying some key attributes - such as the gender of the job applicant or whether the citizen is a member of the ruling party. In the case of my working paper on automated discrimination, I vary the ethnicity of criminal defendants to audit a commercial AI system used to assist judges in criminal sentencing and find that AI gives longer sentences when the defendants are of certain ethnicities, even when all relevant facts about the crime are held constant. In contrast to conventional audits, auditing AI systems has the advantage of not crowding out resources for people.

Lab experiment is kind of a misnomer here, as I’m not bringing people in a lab for some treatment. Instead, I recreate commercial AI systems (e.g., for censorship) and vary key components (e.g., the make-up of the training data) in the recreation to study how it changes the performance of the system. In my job market paper, for example, I train my own censorship AI models to approximate commercial models and find that 1) AI’s accuracy in censorship decreases with more pre-existing censorship and repression; 2) the drop in AI’s performance is larger during times of crisis, when people reveal their true preferences; but 3) the existence of the free world can help boost AI’s ability to censor. See a similar design in political science here.

By nature, experiments are difficult for some countries and topics. I hope to bring some new ideas for these difficult contexts.