Apple and Google Face Pressure to Remove Controversial 'Nudify' Apps from App Stores

TL;DR

• San Francisco City Attorney David Chiu issued cease-and-desist letters to Apple and Google, demanding the removal of 13 AI face-swap apps that generate non-consensual nude images and the termination of business ties with their developers.
• Apple confirmed it has already removed three of the targeted apps and is terminating the developer accounts, while Google stated it has suspended all five apps referenced in the letter from Google Play.
• The legal action cites California laws against deepfake pornography, arguing the tech giants are "aiding and abetting" sexual abuse by profiting from in-app purchase fees generated by these harmful applications.

San Francisco Escalates Fight Against AI "Nudify" Apps

San Francisco has formally demanded that Apple and Google purge their app stores of controversial artificial intelligence applications capable of digitally stripping people in photos. On Thursday, City Attorney David Chiu dispatched legal cease-and-desist notifications to both tech giants, instructing them to eliminate 13 specific "face-swap" applications that predominantly facilitate the creation of non-consensual nude imagery, often targeting women and minors.

The directive marks a significant escalation in the legal battle over AI-generated deepfakes, shifting focus from the app developers to the platform holders themselves. Chiu’s office argues that by allowing these apps to monetize through in-app purchases, Apple and Google are complicit in the distribution of explicit deepfake content and must "sever" their business ties with the developers.

The Scope of the Violation

The 13 applications under scrutiny include eight listed on the Apple App Store and five on the Google Play Store. While many of these apps market themselves broadly as "face-swapping tools," their primary functionality becomes apparent once users engage with them, revealing the capability to generate sexual deepfakes.

These apps utilize in-app purchase systems, a revenue model from which the tech platforms receive a portion of the profits. The cease-and-desist letters assert that the companies have likely "made millions of dollars in fees" from apps that offer nudification services.

The problem is not limited to the 13 targeted apps. Reports indicate that approximately 100 apps have been identified with the capability to create non-consensual nude images, collectively downloaded around 480 million times. Furthermore, testing suggests that 70% of examined face-swapping apps can create "nudified" images, highlighting significant gaps in current content moderation processes.

Legal Basis: California’s Deepfake Laws

The legal pressure is grounded in California state laws that prohibit the creation and distribution of deepfake pornography. Chiu’s office contends that the apps violate these statutes by facilitating the generation of explicit images without consent.

"Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu stated in an interview with WIRED. The city attorney’s office has previously taken legal action against 16 popular deepfake websites, and this new move targets the platforms that host the software used to create such content.

The letters emphasize that the tech giants must cease "aiding and abetting" the distribution of this content. If Apple and Google fail to comply with the demands to remove the apps, the city attorney’s office may pursue further legal options to enforce compliance.

Apple and Google Respond

Both companies responded quickly to the directive, asserting that they already have policies forbidding such content.

An Apple spokesperson told TechCrunch that nudify apps are explicitly forbidden from the App Store. The company stated it has removed three of the apps in question and is currently terminating the developer accounts of the creators. Apple noted it is also in contact with four other developers that need to address policy violations or risk removal.

Google’s response was equally swift. A Google spokesperson confirmed that all five Play Store apps referenced in Chiu’s letter had been suspended from Google Play. The spokesperson added that the company investigates reported violations and takes "swift action," which has included suspending hundreds of violating apps and restricting search terms like "nudify" on their store.

Implications for Developers and Moderation

The San Francisco directive underscores the growing responsibility placed on platform holders to proactively moderate their ecosystems. The legal action highlights potential reputational risks for Apple and Google as trust in app safety declines amid ongoing violations.

For app developers, the message is clear: apps that facilitate sexual abuse or violate state laws regarding nudity will face immediate removal and potential legal repercussions. The city attorney’s office is urging a strategic shift in content moderation policies, suggesting that current reactive measures are insufficient to stop harmful technology from appearing in stores in the first place.

This case also raises the question of financial liability. By profiting from the fees generated by these apps, the platforms are seen as financially benefiting from the harm caused to victims of non-consensual deepfakes. The potential revenue loss for Apple and Google from removing these apps could reach $120 million, according to some estimates, though the primary goal of the legal action is the cessation of harm rather than financial penalty.

As the debate over content moderation in digital marketplaces continues, San Francisco’s move sets a precedent for how local governments may intervene to enforce state laws against AI-generated abuse. The outcome of this demand will likely influence future regulations and the operational strategies of the world’s largest technology companies.

AndroGuider Team

Articles written by the AndroGuider team. We try to make them thorough and informational while being easy to read.