New Delhi, India – In the early hours of November 30 last year, as voters were lining up to cast their votes in parliamentary elections to choose the next government in India's southern state of Telangana, a seven-second video began circulating on social media.
In a post posted on X by the national opposition party, the Nationalist Congress Party, which was in the state at the time, KT Rama Rao, the leader of the Bharat Rashtra Samiti, which ruled the state, urged the people to vote in favor. It appeared that he was calling. Parliament.
The Congress shared this information widely in various WhatsApp groups “unofficially run” by the party, said a senior official who requested anonymity. It was eventually posted on the party's official X account, where it was viewed more than 500,000 times.
It was fake.
“Of course, it looks completely real, but it was generated by AI,” the Nationalist Party leader told Al Jazeera. “But the average voter wouldn't be able to tell the difference. Voting had begun. [when the video was posted] and there was no time [the opposition campaign] To limit the damage. ”
The well-timed deepfake is indicative of the deluge of AI-generated or manipulated media that has undermined a series of elections in Indian states in recent months, and which is now in the country's There is a risk that this will fundamentally shape the upcoming general election.
From March to May, India's approximately 1 billion voters will choose the next government in the largest election in the world and in history. The threat of deceptive AI-generated media came to the world's attention in January when sexually explicit images of artist Taylor Swift appeared on social media platforms. In November, India's Information Technology Minister Ashwini Vaishnaw called deepfakes a “threat to democracy,” and Prime Minister Narendra Modi expressed similar concerns.
But as useful artificial intelligence tools become more available, recent reports show that teams from across India's political parties, including Prime Minister Modi's Bharatiya Janata Party and the Congress, are deploying deepfakes to influence voters. The heads of 40 election campaigns told Al Jazeera. Some of the AI tools used to generate deepfakes are free, while others are available with a subscription for as little as 10 cents per video.
“Create recognition”
The Bharatiya Janata Party, perhaps India's most technologically sophisticated political party, has been at the forefront of using illusions in its election campaigns. Back in 2012, the party used 3D hologram projections of Mr. Modi to enable him to “campaign” in dozens of locations simultaneously. This strategy was widely deployed during the 2014 general election when Mr. Modi came to power.
There was little deception there, but in February 2020, Bharatiya Janata Party MP Manoj Tiwari became one of the first people in the world to use deepfakes for election campaigning. In his three videos, Tiwari addressed Delhi voters in Hindi, Haryanvi and English ahead of the national capital's legislative assembly elections, making him appeal to three different audiences in the multicultural city. Only the video in Hindi is real, the other two of him are deepfakes, using AI to generate his voice and words, and change his facial expressions and lip movements so that they don't look real just by looking at them. It made it almost impossible to detect that it's not there.
In recent months, the Dravida Munnetra Kazhagam (DMK), which rules the southern state of Tamil Nadu, has used AI to bring the party's iconic leader M. Karunanidhi back from the dead, as well as the former filmmaker and veteran politician. The realistic-looking video was used at campaign events.
Consultants and campaign managers now say the use of deepfakes could accelerate further in the 2024 election.
“Politics is about creating perceptions. Using AI tools [of voice and video modulation] You can flip that perception in a minute with just a click,'' said Arun Reddy, the tech-savvy Congress social media national coordinator who oversaw the party's Telangana elections. He added that his team was full of ideas for incorporating AI into campaigning, but didn't have enough “trained people” to implement them all.
Reddy, like other political parties, is strengthening its team.
“AI will have a huge impact on the creation of stories,” Reddy told Al Jazeera. “Political content manipulated by AI will increase many times more than before.”
“The campaign is going awry.”
Divyendra Singh Jadun, a 30-year-old from Pushkar, a desert town in western India, runs an AI startup, The Indian Deepfaker. His company, founded in October 2020, helped Congress leaders in Rajasthan to send personalized messages on WhatsApp and address each voter by name during the November assembly elections. It reproduced the voice of ministerial candidate Ashok Gehlot. Indian deepfakers are currently working with Sikkim Chief Minister Prem Singh Tamang's team to create holograms for upcoming campaigns. Sikkim is one of the smallest states in northeastern India, located in the Himalayas between India, Bhutan, and China.
It's a clean official job, he said. But in recent months, he has been overwhelmed by what he describes as “unethical demands” from political movements. “Political parties communicate indirectly through WhatsApp international numbers, Instagram burner handles or Telegram connections,” Jadoon told Al Jazeera in a phone interview.
He said the company rejected more than 50 such requests during the November election, when potential customers asked for video and audio to be altered to target political opponents, including pornography. It is said that he did. Jadun said that as a start-up company, they are especially careful to avoid legal troubles. “And this is a highly unethical use of AI,” he added. “But I know a lot of people who do it at a very low price and now have it readily available.”
During election campaigns in Madhya Pradesh in central India and Rajasthan in western India last November, police arrested Modi, Shivraj Singh Chouhan, Kailash Vijayvargia (both of the Bharatiya Janata Party) and Kamal. Multiple charges have been registered for deepfake videos targeting senior politicians including Mr. Nath (Congress). ). The production of deepfake content is often outsourced to private consulting firms that rely on social media networks led by WhatsApp to distribute them.
A political consultant who requested anonymity told Al Jazeera that large numbers of ordinary citizens without public profiles are registered on WhatsApp, making it difficult for anyone to directly track political parties, candidates, consultants, and AI companies. He said it was being used for election campaigns.
The consultant campaigned for both the Bharatiya Janata Party and the Congress six times in last year's assembly elections. “Deepfakes were primarily circulating in Rajasthan, where the network was operating on WhatsApp using the phone numbers of construction workers,” they said.
Meanwhile, AI-manipulated voices are a particularly valuable tool in small constituencies, and could be used to “arrange 'dark money' for elections or to blackmail someone into buying votes.” The consultant said that his own candidates were also targeted with such recordings. . Recordings typically mask candidates' voices as evidence of corruption.
“No political party considers voter manipulation by AI to be a sin,” they added. “It’s just part of the campaign strategy.”
India has 760 million internet users, more than 50% of the population and second only to China.
Among all the requests, one that stood out to Mr. Jadun was from a constituency in southern Rajasthan. Ahead of November's state election, a caller asks Jadoon, who did not specify his party affiliation, to alter a questionable but real candidate's video to create a realistic deepfake. did. The purpose is to claim that the original is a deepfake and that the deepfake is the original.
“The opposition had the offending video of the candidate and wanted to spread it on social media sooner to claim it was a deepfake,” he said, breaking out into an awkward laugh. “Political movements are getting weirder.”
Threats to election integrity
Anushka Jain, a policy researcher at the Goa-based Digital Futures Lab, said Indian law currently does not clearly define “deepfakes.” Police are attempting to deal with individual cases using a combination of laws against defamation, fake news or violations of personal integrity and information technology laws. But they often find themselves playing whack-a-mole.
“Police are prosecuting people because of the influence of deepfakes, not because of deepfakes themselves,” she said.
Analysts say the Election Commission of India (ECI), the autonomous body that conducts polls, needs to keep up with the changing nature of political campaigns.
In the days leading up to voting in last year's Telangana state elections, leaders of the ruling Bharat Rashtra Samithi made repeated statements. warned Their followers on social media must be on guard against deepfakes being deployed by the Congress party. They also complained to the ECI over the deepfake video shared by the Congress on the morning of the polls.
But the video Remaining Two parliamentary leaders aware of the issue told Al Jazeera.
Al Jazeera has contacted the ECI for comment, but has not yet received a response.
“Even if someone is misled into believing something and thereby changes their mind, that undermines the purity of the electoral process,” said SY Quraishi, India's former chief election commissioner. “Deepfakes have made the problem of spreading rumors during polls a thousand times more serious.”
Quraishi said deepfakes need to be curbed in real time to minimize the damage they do to India's democracy.
“ECI needs to take action before any damage is done,” he said. “They need to act faster.”
“The truth is just out of reach”
The Indian government is pressuring major tech companies, including Google and Meta, to take aggressive steps to curb deepfakes on their platforms. IT Minister Rajeev Chandrasekhar met with officials from these companies as part of his deliberations on the threat posed by deepfakes.
By asking the tech industry to take the lead, governments can avoid criticism that they are trying to selectively censor deepfakes or crack down on emerging AI technologies more broadly.
But by handing money to private companies, the government calls into question the sincerity of its intentions to regulate manipulative content, said an executive at the Internet Freedom Foundation of India, a leading technology policy think tank based in New Delhi. Director Prateek Wagre said. “It's mostly wishful thinking,” he said.
“The rise of AI is making the challenge even more complex,” Wagle said, noting that technology companies are failing to address existing content moderation issues. And current approaches to content moderation ignore the core of the problem, he said.
“You’re not solving the problem,” he said. “design [of algorithms] It's just flawed. ”
On February 16, major technology companies agreed at the Munich Security Conference to voluntarily adopt “reasonable precautions” to prevent artificial intelligence tools from being used to interfere with democratic elections around the world. signed. However, the vaguely worded agreement disappointed many supporters and critics.
YouTube announced that it will now use its privacy request process to request the removal of content generated or modified by AI that imitates an identifiable individual, such as a face or voice.
“I’m not very optimistic about the ability of platforms to detect deepfakes,” says Ravi Iyer, managing director of the Neely Center for Ethical Leadership and Decision Making at the University of Southern California Marshall School of Business. “With low digital literacy and increased video consumption, this poses a significant risk to the integrity of India's elections.”
Identifying all media that has been manipulated by AI is not a reasonable task, Iyer said, so companies need to redesign their algorithms to avoid polarizing content. “Companies have the money and resources and need to take reasonable steps to address the rise in deepfakes,” he said.
The Internet Freedom Foundation has published an open letter ahead of national elections calling on political candidates and political parties to voluntarily refrain from using deepfake technology. Wagle said he's not sure many people will bite, but it's worth a try.
Meanwhile, political movements are ramping up their AI arsenals, and some, like Reddy, Congress' national social media coordinator, admit the future looks bleak.
“Most of the people using AI are there to distort facts. They want to create a perception that is not based on truth,” Reddy said. “The penetration of social media in India, combined with the rise of AI, will prevent truth from reaching the people in elections.”
