What happens when the era of AI-powered surveillance coincides with an authoritarian assault on public education? The transparency needed to answer that question does not exist, which is why the Knight First Amendment Institute, where I work, has filed a lawsuit seeking information about how one school district uses surveillance systems to monitor student laptops.
It is estimated that millions of children — nearly half of K-12 students across the nation, according to a recent New York Times report —are subject to digital surveillance systems that can potentially monitor every word or phrase they type on school-issued laptops, tablets, and software. These software systems, supplied to school districts by private education technology companies like GoGuardian, Gaggle, and Lightspeed, scan students’ communications, internet searches, and assignments, searching for keywords or phrases that may indicate cyberbullying, thoughts of self-harm, or thoughts of harming others. If a student uses a certain word or phrase deemed inappropriate by the vendor or the school district using the technology, administrators are notified and, in some instances, police get involved. The systems also have filtering capabilities, which can restrict students’ access to certain websites and pages based on their content.
Stay up-to-date with the politics team. Sign up for the Teen Vogue Take
Given the widespread concerns about youth mental health, many people might view such digital surveillance as a godsend, a critical tool for combatting tragically high rates of youth suicide and depression, as well as school shootings. But that viewpoint likely rests on two assumptions: first, that the systems are as effective as the edtech industry claims, and second, that school districts are limiting the systems’ purview to content that relates to student safety.
It's not clear, however, that those assumptions have merit. The scant information we have about school surveillance suggests that the systems can lead to erroneous flags and can be used as a tool of overzealous surveillance by school administrators. Public records obtained by the Electronic Frontier Foundation have exposed several examples of students being flagged for innocuous visits to websites containing the text of Genesis from <a href="http://Bible.com" rel="nofollow">Bible.com</a>, classic literature like Romeo and Juliet, and publications about Martin Luther King Jr. and the Civil Rights Movement. In other cases, surveillance of students’ online activity has led to the unwanted disclosure of private details about their sexuality and the flagging or blocking of race-related content, according to a report from the Center for Democracy & Technology. And just last year, journalism students at a Lawrence, Kansas, high school successfully campaigned to get their files omitted from the purview of Gaggle, arguing that the school’s use of the technology would have a chilling effect on critical reporting on district staff and administration.
Despite the edtech industry’s claims that their artificial intelligence products have saved thousands of students’ lives, the Associated Press reports that no independent research has corroborated their efficacy. Industry data regarding the frequency and accuracy of the systems’ alerts is usually kept hidden behind closed doors, exclusively in the hands of the for-profit companies that develop and market them. And while school districts maintain that their use of these tools is aimed at deterring self-harm, cyberbullying, and other harmful situations, their ability to customize the standard list of keywords and blocked websites provided by edtech companies raises serious concerns about the extent of surveillance and censorship school administrators can carry out, and the potential impact on students’ privacy, speech, and associational rights. For example, school districts could easily block “Black Lives Matter” websites, as at least one district reportedly did in the past, and flag any research or discussion of sexual orientation. In a moment where many children are prohibited from discussing racial history, learning about gender identity, or reading banned books in the classroom, it is critical that they have alternative channels for exploration and expression, including the internet and messaging platforms. Digital surveillance endangers both the sanctity of their private communications and their freedom to access more accurate, complex, and engaging ideas than their schools might permit.