India’s AI Deepfake Industry Exploits Women’s Identities on Scale, with Platforms and Regulators Mum
A 16-year-old content creator from a small town in India stumbled upon a disturbing AI-generated content goldmine: hundreds of social media accounts sharing nude images of women and young girls. These accounts use Chinese open-source AI models and Telegram bots powered by Unified Payments Interface (UPI), a digital payment system, to spread this content at scale.
The Rise of AI-Powered Deepfakes in India
India’s deepfake industry has grown exponentially over the past few years, with AI models like the Chinese-made DeepFaceLab and FaceSwap being widely used to create realistic fake videos and images. These models are often sourced from the internet, with many being open-source and free to use.
Bots & Payment: An Enabling Ecosystem
The AI-generated content is being shared on social media platforms and messaging apps, often using Telegram bots that are enabled with UPI payment systems. This allows bots to automatically send content and receive payments from users who want to access explicit material. The UPI system also enables instant payments, making it easier to monetize the spread of deepfakes.
Platforms and Regulators Turn a Blind Eye
Despite the alarming scale at which deepfakes are being exploited, platforms and regulators seem to be doing little to address the issue. The content creator from Belgaum described the ease with which they could find and access these accounts: “It’s like they’re waiting for someone to report them, and then they just take it down and open up another account.” The lack of action from platforms and regulators has left women and children’s identities being exploited and manipulated for profit.
What this means: The lack of regulation and oversight in India’s AI deepfake industry has created a toxic environment where women’s identities are being exploited at scale. Platforms and regulators must take immediate action to address this issue and ensure that users are protected from AI-generated content that harms and manipulates them.
