Disinformation and current challenges for the industry

March 11, 2021

This policy paper describes tactics, adversarial behavior and challenges for the platform industry.  

It is based on the output of our technology during and after the US 2020 elections. The five major challenges include: 

1. Verified accounts, pages and channels 

2. Live streaming 

3. Highly partisan news media 

4. Elected officials 

5. Domestic terrorism

Context

In the past few years, platforms heavily invested in developing community guidelines, trust and safety teams and intelligence desks. Community guidelines set out what content is allowed on the platform and what is not allowed. Those guidelines are necessary references towards users when platforms remove violative content. There are indisputable cases such as violations against child safety. But there are also disputable cases such as borderline content that allows different views on a political issue.

The stakes are high after major social media platforms were used by foreign state-actors to interfere in the US 2016 presidential election, for supporting the genocide of the Rohingya in Myanmar in 2016 and 2017 and the incitement of a deadly storming of the US Capitol on January 6, 2021 aiming to block elected officials from certifying the outcome of the US 2020 presidential election.

People in the industry agree that disinformation as intentional distribution of false, misleading or deceptive information should not have a home on the platforms. However, when it comes to defining what constitutes disinformation, some platforms have a hard time to make difficult trade-offs between free speech and violative behavior. Even if people find a common ground on what disinformation is, breaking the definition down into labeling functions and appropriate metrics for automated systems in order to identify disinformation in the data as it streams is challenging. Particularly in an real-time product environment that scales across languages and is intertwined with dependent systems. When it comes to disinformation, context matters. And context is hard to retrieve or to apply. Disinformation is not only a challenge for policy teams. Disinformation is a challenge for data engineers.

Download