Did a politician break a promise? : Automating election promise fulfilment tracking
Härkönen, Markus (2026)
2026
Tietojenkäsittelyopin maisteriohjelma - Master's Programme in Computer Science
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Hyväksymispäivämäärä
2026-07-30
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-202607298612
https://urn.fi/URN:NBN:fi:tuni-202607298612
Tiivistelmä
Citizens in democracies need more visibility to what their representatives in politics promise and whether they act towards fulfilling those promises. When promises and their corresponding actions become more visible, it is easier for citizens to judge how politicians perform. This helps citizens make more informed voting decisions and gives direction for their own political activity such as directly contacting politicians or introducing citizen's initiatives for the parliament.
This thesis helps political researchers automate parts of the promise-fulfilment evaluation. Automation scripts and Large Language Models (LLMs, AI) gather the evidence, and the human validates or rejects the evidence and makes their own conclusions on the fulfilment.
The scripts followed the pattern a human would use in such research: 1) Extract election promises and actions from source texts, 2) list which actions are related to which promises, and 3) evaluate whether promises were fulfilled based on the action evidence.
The pipeline extracted 523 promise records from four out of five government-forming party programmes and 383 evidence quotes. The pipeline created 444 valid links between the promises and evidence quotes. 135 promises reached an LLM-generated fulfilment verdict and 74.2 % of the promises were left undetermined due to a lack of linked evidence. On a 60-record stratified sample, two annotators judged 27/60 and 17/60 verdicts to be correct. Manual validation by the two annotators confirmed one case where a promise was acted against, reaching a ``negative fulfilment'' verdict by the large language model used in the promise fulfilment tracking pipeline. However, one case does not prove pipeline performance nor government performance.
Doing such a workflow manually takes considerable resources, which might be a reason why not many academic research papers were found on political promise fulfilment during the thesis writing process. That is why this thesis looked for and found ways to add automation to the three steps with the help of AI. The conclusion of this thesis is that AI/LLMs can build a promise fulfilment tracking pipeline, but reliability of the promise fulfilment evaluation by a local LLM was not proven. This thesis did not create a reliable tool for tracking election promise fulfilment. Such tool will need a comprehensive manual validation of the source and output texts as well as continued development to increase reliability. However, this work shows that creating such pipeline is possible when reliability issues are resolved.
This thesis helps political researchers automate parts of the promise-fulfilment evaluation. Automation scripts and Large Language Models (LLMs, AI) gather the evidence, and the human validates or rejects the evidence and makes their own conclusions on the fulfilment.
The scripts followed the pattern a human would use in such research: 1) Extract election promises and actions from source texts, 2) list which actions are related to which promises, and 3) evaluate whether promises were fulfilled based on the action evidence.
The pipeline extracted 523 promise records from four out of five government-forming party programmes and 383 evidence quotes. The pipeline created 444 valid links between the promises and evidence quotes. 135 promises reached an LLM-generated fulfilment verdict and 74.2 % of the promises were left undetermined due to a lack of linked evidence. On a 60-record stratified sample, two annotators judged 27/60 and 17/60 verdicts to be correct. Manual validation by the two annotators confirmed one case where a promise was acted against, reaching a ``negative fulfilment'' verdict by the large language model used in the promise fulfilment tracking pipeline. However, one case does not prove pipeline performance nor government performance.
Doing such a workflow manually takes considerable resources, which might be a reason why not many academic research papers were found on political promise fulfilment during the thesis writing process. That is why this thesis looked for and found ways to add automation to the three steps with the help of AI. The conclusion of this thesis is that AI/LLMs can build a promise fulfilment tracking pipeline, but reliability of the promise fulfilment evaluation by a local LLM was not proven. This thesis did not create a reliable tool for tracking election promise fulfilment. Such tool will need a comprehensive manual validation of the source and output texts as well as continued development to increase reliability. However, this work shows that creating such pipeline is possible when reliability issues are resolved.