Lynn Kaack leverages machine learning approaches for public policy analysis

Kaack and her co-author Sebastian Sewerin found that large and systematic assessments of climate policies were lacking as manual analyses of public policies were costly and labour-intensive. This inspired the creation of POLIANNA, a dataset of 20,577 annotated spans, drawn from 18 EU climate change mitigation and renewable energy policies.  

Uncertainty around how to measure effectiveness of climate policy action 

Climate change continues to be one of the biggest challenges of our time and public policy remains insufficient in reaching the commitments of the 2015 Paris Agreement. There has been a shift towards focusing on national policies over international agreements, which makes it even more challenging for researchers to measure and evaluate the effectiveness of public policy actions. It is difficult to establish the causal link between the specific policy design and their impact on climate change mitigation. There have been numerous studies identifying ‘policy design elements’ that may determine effectiveness, including the policy scope, actors, sectors covered, or instrument type. However, as noted in the study, much of these analyses do not evaluate specific design elements of policies at scale. This can be due to the challenge of needing a lot of resources for collecting the data (ie. trained staff, access to government documents, country experts) to manually code the policies. 

Utilizing text as data approaches 

An alternative to manually coding policies is utilizing text-as-data approaches that leverage machine learning and AI to help scale up data gathering to produce datasets of policy design elements. The paper showcases a training dataset that can serve to aid in reaching the goal of automatising the analysis of climate policy design elements.   

Kaack’s team has developed a coding scheme that reflects key policy design elements and is suitable for machine-learnable tasks. This was then used to create the policy design annotations (POLIANNA) dataset, which read relevant EU climate and energy policies and developed a scheme for annotating the instrument types, policy design characteristics, and technology and application specificity at the level of text spans. The POLIANNA training dataset includes 18 EU policies containing 412 articles, i.e., subsections dividing the EU legal acts, comprising 20,577 annotated spans. 

Kaack’s team envisions this dataset to aid in building further tools to help with manually coding large scale policy texts. 

Read Lynn Kaack’s paper in Scientific Data. 

The Hertie School is not responsible for any content linked or referred to from these pages. Views expressed by the author/interviewee may not necessarily reflect the views and values of the Hertie School.