Evaluating Cyber Security Dashboards for Smart Cities and Buildings: Enhancing User Modeling with LLMs

Abstract

Designing effective cybersecurity visualization has become a crucial component of cyber defense strategies in many domains and industrial environments. Human behaviour, modeling and input are major aspects of designing visualization systems. Yet, the task of evaluating these developed visualization systems is both time-consuming and challenging, and it is often prone to cases where user evaluation is limited owing to a lack of different stakeholders and end users during the design process. Recognizing the potential of advanced Generative Artificial Intelligence and Large Language Models (LLMs), our study aims to explore their capabilities in evaluating web-based security visualization tools and dashboards, particularly in the context of smart cities and buildings. We study and compare the feasibility of using various LLMs available today, for conducting usability testing, serving as an additional resource given the limited availability of human participants. In particular, we focus on three different LLMs: Bing Chat, ChatGPT-4 and ChatGPT-4o. While each had its strengths and drawbacks, our findings revealed that the results obtained had a strong correlation with human test subjects. LLMs can be a valuable aid during evaluation, by offering in-depth insights and evaluations, tailored to the specific requirements of smart buildings, cities and automation cybersecurity. Moreover, our research and findings also reveal that LLMs can similarly be used for the evaluation of a wide range of other visual systems for industrial environments.