Many deepfake apps address these concerns by being “on rails”, or restricted: users can only swap faces into a selection of scenes from pre-approved films or shows. But these restrictions are often the outcome of technological limitations rather than a deliberate security choice. In order to quickly generate high-quality face-swaps with one or a few user images, apps “pre-train” their generative models on a number of popular movie scenes, such as the twins from The Shining, or Sean Bean’s “one does not simply walk into Mordor” meme from The Lord of the Rings. As the technology becomes more powerful and pre-training less restrictive, developers might see a competitive advantage in opening up their apps to user-uploaded content in an “off-rails” approach.
Other technology companies offering potentially hazardous services such as lip synchronisation and voice synthesis have adopted policies to prevent their products from being misused – like individually vetting clients, and gaining permission from all parties whose likeness is being altered. Yet it’s difficult to imagine deepfake apps enforcing similar protocols, given their reliance on uptake by a large number of users eager for novel deepfake capabilities. As apps vie for users’ attention in a crowded market, it seems almost inevitable that they’ll “go off the rails.”
Sure, both Apple and Google have implemented bans on apps that create deceptive or malicious deepfakes from their app stores, and developers are working on security features to avoid falling foul of these policies. These include app-specific deepfake detection tools, automatically blocking p*rnographic or malicious content, and the watermarking of deepfakes generated by the app.
While developers’ readiness to address misuse of their apps is promising, deploying these security features poses several challenges. One is how developers roll them out in the first place. For detection tools to be effective at stopping malicious deepfakes, they would need to be widely adopted by the social media platforms and messaging apps – but no social media platform currently has deepfake detection in their media upload pipelines, and implementing detection on messaging apps like WhatsApp or Telegram would require monitoring users’ conversations, a significant change to these services’ current privacy-focused model.
Another is how reliable these security measures would be. A watermark would notify viewers that a video is fake, but developers might be reluctant to place one where it would obstruct the image entirely, meaning it could simply be cropped out of frame. Preemptively detecting and blocking malicious content would also prove difficult given the wide range of possible harms that could be wrought through this budding technology. Capturing the near-limitless variety of malicious uses is currently impossible to automate, while manual moderation would be unfeasible given the volume of content being generated online.
Given all this, what could be plausibly done to minimise deepfake apps’ misuse? One approach could involve the creation of an app safety framework for developers, including measures such as threat assessments, limited access without user authentication, or even moratoria on releasing new capabilities that lack harm-mitigation strategies. If such a framework were enforced by app stores and other stakeholders critical to an app’s success, it could help create a safety standard for deepfake apps that all developers would have to follow in order to be published.
A stronger reaction may involve new legislation that allows victims or authorities to hold developers to account if their deepfake apps are deemed open to, or intentionally designed for, misuse. This could entail fines and the payment of damages to victims, the removal of offending apps from app stores, or even criminal charges against the developers. What both these possible approaches share is that they target deepfake apps’ shift “off the rails” by either restricting who can access them and what capabilities are released, or punishing developers if this shift ends up causing harm.
There is clearly an inherent friction with the model deepfake apps are moving towards: the more open and powerful they become, the harder it is to prevent the harms they can cause. Striking the right balance of openness and safety will be essential to deepfake apps’ future success. If they stay their current course and continue to move “off the rails”, this balance will be difficult to achieve.
Henry Ajder is an adviser on deepfakes, disinformation and media manipulation. Nina Schick is author ofDeepfakes: The Coming Infocalypse
This article was originally published by WIRED UK
