Content moderation on social networks is becoming increasingly complex

Efforts by various technology companies to identify and remove violent, dangerous or misleading content on their platforms have already proven insufficient when social networks relied mainly on text, such as Facebook and Twitter, or images, as was the case with Instagram for a long time. Today, with millions of people consuming video almost continuously on the platforms, content moderation has become an unprecedented challenge for the companies themselves. This in itself is a complex task that technology companies have often downplayed in the past, largely because of the economic outlay it requires. Hiring moderators who are familiar with the language and context in which content is created is not only complicated but also costly, not least because the mental health of employees - who are exposed to potentially traumatic contentin particular - must be protected.

Automated moderation systems still have major technological limitations, which are particularly evident in video moderation, but these are becoming more common. In this context, artificial intelligence makes use of the study of images by linking them to a series of other images on which it has been "trained": in this way, it is possible, for example, to recognise whether a person is depicted naked or clothed in a photo or whether someone is holding a weapon. With videos, which are nothing more than a series of numerous reproduced images, this approach is more difficult to follow and proves insufficient. The many individual images that make up a video may not in themselves violate a platform's guidelines, but when combined they can become a harmful narrative. For example, a video circulated on TikTok claiming that a covid drug could be made from a grapefruit: while the series of images of the fruit did not violate the platform's guidelines, given the context in which they were placed, they undoubtedly resulted in uninformative content. «A system that detects a firearm in every video would flag the clip of two people discussing the value of an old rifle. At the same time, it would not recognise the video of a person being shot with a firearm out of frame», writes The Atlantic. «YouTube probably has the longest experience with automatic video moderation, but that does not change the fact that clips that violate its guidelines are viewed millions of times every day».

Streamers + Mods = Dynamic Duo

We’re launching Live Moderator on Instagram Live, where creators can assign a mod and give them the power to:

Report comments
Remove viewers from Live
Turn off comments for a viewer pic.twitter.com/S9j7s4dInB

— Instagram (@instagram) March 11, 2022

In the wake of TikTok's success, platforms - led by Instagram and YouTube - have invested heavily in video shorts, so much so that the shorts have become conceptually more complex, eventually evolving into three-dimensional memes in which audio, images and text are interwoven. This makes moderation even more complex, and not just for machines. «Footage of a desolate landscape with windblown weeds rolling around would not be problematic in itself, either for a human or a machine, any more than an audio clip of someone saying, 'Look how many people love you' would be. If you combine the two to insult someone and tell them that no one loves them, a computer will never understand that», The Atlantic says. The same thing happened with a video, much shot on TikTok, of American First Lady Jill Biden visiting people with cancer. Whistles and cackles could be heard in the background, which a user had added in post-production. This is why only 40 per cent of videos removed from a social network like TikTok are edited by an AI - the other controls (and we are talking millions) are still the work of flesh and blood employees, with all the consequences.

Platforms paradoxically tend to amplify videos with disturbing or inaccurate content because when they republish them, they only evaluate the number of user interactions, not the quality of those interactions. If people have watched a video repeatedly or commented on it because they found it unfair, for example, the platform will still rate that interaction as "positive" and then risk amplifying similar content in their feed. «Prolonged exposure to manipulated content can increase polarisation and reduce viewers' ability and willingness to distinguish between truth and fiction», the New York Times reports. Journalist Tiffany Hsu, an expert on the internet and misinformation, adds that «the danger of misleading content lies not necessarily in individual posts, but in the way it further damages the ability of many users to discern what is true and what is not». It is no coincidence, then, that author Clay A. Johson, in his book Information Diet, compares the current production of low-quality information to junk food: content that is tempting but of little "nutritional value" to the mind.