Assessing political news quality: An automated comparison of political news quality indicators across German newspapers with different modalities and reach

by Nicolas Mattis

Thesis Supervisor: dr. Anne Kroon

The research puzzle

In an ideal democracy, news media should provide citizens with high-quality political information, so that they can best perform their civic duties (Strömbäck, 2005). However, the transformation of traditional newspaper markets through new technological affordances and increased economic pressures has prompted various scholars to voice concerns over a decrease in journalistic news quality (e.g. Humprecht & Esser, 2018; Jungnickel, 2011; McManus, 2009), that is said to be particularly pronounced among newspapers with weaker financial bases such as online (Burggraaff & Trilling, 2017; Humprecht & Esser, 2018; Jacobi Kleinen-von Königslöw & Ruigrok, 2016) or regional outlets (Masini et al., 2018). While various studies have examined news quality and its determinants, their comparability suffers from different selections and operationalisations of key indicators.

In light of this, this study set out to a) develop a comprehensive framework for automatically assessing various political news quality indicators in a resource-efficient and scalable manner (see Figure 1) and b) map the current state of political news quality in Germany across different modalities (online vs. print) and levels of reach (national vs. regional). Thus, its goal was to not only provide up to date insights into the quality of the German newspaper market, but also to facilitate future comparative research that makes use of recent methodological advancements and examines over-time developments in a resource-efficient and scalable manner.

Data & Methods

To explore the current state of political news quality in Germany, this study analysed a total of 11,491 political news articles from six major German newspapers (Der Tagesspiegel, Die Welt, Die Süddeutsche, Aachener Teitung, Rheinische Post, Stuttgarter Zeitung) that had been published over a 7-week period between April and June 2020. It employed a combination of several computational methods, in order to tap into four indicators of news quality (see Figure 1) and compare them across newspapers’ reach and modality. Specifically, it drew on dictionary-based methods and named entity recognition with SpaCy to assess actor diversity, emotionality and readability, and used a supervised machine learning (SML) approach to assess impartiality.

Figure 1. Framework for automatically assessing news quality.

Methodological innovation

Most studies that examine news quality rely on manual content analysis, as it allows for the nuanced measurement that is necessary when trying to capture high-level constructs. However, due to limited resources, such approaches tend to only examine a limited number of media markets, newspapers, and articles. In order to overcome this issue, this study used manually coded training material to train and cross-validate binary classifiers that assess three impartiality indicators (see Figure 1). In doing so, it developed a means to automatically assess a high-level construct such as impartiality that future studies can use and build on ( Thereby, this study addressed Boumans and Trilling’s (2015) call for more automated journalism studies and shows the potential that especially SML approaches hold for future research into news quality.

Main Findings

In line with previous arguments about commercialisation (e.g. Burggraaff & Trilling, 2017), print news exhibited higher degrees of news quality than online news. Specifically, print news:

  • relied on a more diverse set of actors in their articles,
  • reported with higher degrees of impartiality, and
  • used a less emotional and less negative reporting style

Contrary to the expectation that regional newspapers’ quality suffers from fewer available resources that can be allocated to quality reporting, regional news scored higher on two of the four political news quality indicators under study than national news. Specifically, regional newspapers:

  • reported in a style that was less emotional and negative than national newspapers,
  • and relied on a more diverse set of actors in their reporting
Implications for future research

This study added to the literature and can guide future research into political news quality in several ways. First, it provides a comprehensive framework for the automated measurement of news quality that might facilitate future comparative research. Second, it showed that SML provides a promising avenue for news quality research, as it allows scholars to scale-up the measurement of high-level constructs. Third, it points to the need for new ways of newspaper classifications that go beyond a mere distinction by reach and modality, as the aggregated differences across newspapers were rather minor and somewhat obscured the at times stark differences that existed between individual newspapers.


Boumans, J. W., & Trilling, D. (2016). Taking stock of the toolkit: An overview of relevant automated content analysis approaches and techniques for digital journalism scholars. Digital Journalism, 4(1), 8–23.

Burggraaff, C., & Trilling, D. (2017). Through a different gate: An automated content analysis of how online news and print news differ. Journalism.

Jacobi, C., Kleinen-von Königslöw, K., & Ruigrok, N. (2016). Political News in Online and Print Newspapers: Are online editions better by electoral democratic standards?. Digital Journalism, 4(6), 723-742.

Jungnickel, K. (2011). Nachrichtenqualität aus Nutzersicht. Ein Vergleich zwischen Leserurteilen und wissenschaftlich-normativen Qualitätsansprüchen. M&K Medien & Kommunikationswissenschaft, 59(3), 360-378.

Masini, A., Van Aelst, P., Zerback, T., Reinemann, C., Mancini, P., Mazzoni, M., … & Coen, S. (2018). Measuring and explaining the diversity of voices and viewpoints in the news: A comparative study on the determinants of content diversity of immigration news. Journalism Studies, 19(15), 2324-2343.

McManus, J. H. (2009). The commercialization of news. In The handbook of journalism studies (pp. 238-254). Routledge.Strömbäck, J. (2005). In search of a standard: Four models of democracy and their normative implications for journalism. Journalism studies, 6(3), 331-345.