The Effect of Humour in Political Messaging: An Investigation Combining Fine-Tuned Neural Language Models and Social Network Analysis
by Janice Butler
Thesis supervisor: Dr. Damian Trilling
“Remember when Trump forgot Mike Pence’s name and then tried to have him killed?” Whether with irony, wit, sarcasm or benevolence, humour in politics brings things to the point, can evoke strong emotions or – as in this example – polarize like nothing else. The dearth of research on measurement of humour would seem to imply – for such a fascinating and powerful aspect of communication – that we are far from solving this comic conundrum.
The study aimed to implement an optimized humour detection system (HDS) based on neural language models (NLMs) using the denoising autoencoder (e.g. BERT and DeBERTa) and autoregressive patterns (e.g. GPT-x and XLNet), thereby solving 2 tasks discerning between 9 humour-types and 6 degrees of humour.
A wide variety of NLMs with differing architectures and sizes were fine-tuned on the annotated data, with the statistically best models (according to the F1-metric) being chosen for use in the next phase. Fine-tuning was carried out through varying the optimizer (AdamW or Adafactor), the initial learning-rate, the warm-up period and learning decay type, number of training epochs, batch-size, as well as the fundamental NLM-type chosen and the size of model. The cloud service www.wandb.com was used extensively to capture and manage results, visualize training in real time and organise the project artefacts.
Figure 1 – Hyperparameter Evaluation
Large numbers of tweets from heterogeneous groups of politicians, political journalists and (as a control group) comedians and satirical sources were taken, subjected to HDS-analysis and, in combination with the metadata from each tweet, provided answers to which type and what degree of humour achieved most propagation for the actor.
More than 420,000 political as well as 250,000 non-political social media posts were analysed. The resonance in social networks was correlated with the type and degree of humour of each posting to create a picture of which types and intensities of humour are most frequently and most successfully employed by politicians, political journalists and comedians.
The results of the project let us conclude that NLMs – with appropriate fine-tuning – do enable good automated differentiation between the 9 chosen humour-types and a moderately good estimation of the degree of humour in a text. It was additionally possible to replicate results of previous researchers of the simpler binary humour-categorization task.
Figure 2 – MP Tweets: Mean Propagations per Humour Type (Standard Error of Mean)
GPT-2-derived models (of type AR) are considerably less precise in predicting humour-type, although XLNet (also of type AR) produced thoroughly usable results.
Humour invariably provides an effective boost to message propagation and heightened comedic intensity maximises propagation in all groups analysed. When politicians use humour they prefer sarcasm to all other types, but message propagation succeeds for them better with ironic humour. Benevolent humour is least used, but produces the second best propagation effect for politicians.
Political journalists – like politicians – prefer to use sarcastic humour, but they achieve most resonance with cynical remarks.
There is an almost remarkably clear picture indicating that more intense humour enhances social media propagation, although serious messaging, albeit in smaller proportions, is frequently quite effective. Humour is not always the ideal communication tool, is somewhat sparingly used by politicians (< 9% of all tweets) and even more so by journalists (< 4%). Of the 5% most liked postings, the majority in every group – including comedians – were non-humorous. The results though show, an increase in usage of humour would seem to be possible and no bad thing.
Figure 3 – Numbers of MP Tweets per Humour Type
The automated measurement of humour-type and degree enables processing of very large amounts of text in a very short space of time. The approaches to network analysis allow an immediate impression of relative take-up by the information consumers.
My interpretation of the research is that NLMs fine-tuned for these tasks are uncannily good at recognising humorous language patterns and the reliable recognition of humour-types demonstrates this. Nevertheless, type of humour has more to do with certain language patterns than does the degree of humour, which ultimately is very reliant on the cleverness of the message and goes even deeper into the language, requiring general knowledge and often topical information, too.
The difference in language is analogous to a political speech-writer, who on the one hand should include the important policy points being “sold” by the politician, but ideally also packages these points using rhetorical devices. It would seem that the methods presented here successfully recognise the stylistic patterns of humourists without necessarily deciphering the inner meaning. Nonetheless, this ability to recognise “the packaging” is a definite and important step forward in automated content analysis.
The methodological approach can also be envisaged for completely different tasks (other than humour detection) that centre on natural language understanding, because other fine-tunings are equally possible, based on training data coded with other task-specific annotations. Many ideas for improvement in precision of data-annotation and in inference accuracy are already obvious and described in the thesis.