The decision-maker moment: Rich findings to invite rich questioning
LLMs that have been so thoroughly optimized can be used for forecasting and related analyses. Here, as before, the key is iteration. Different at this stage, however, must be the focus on the decision-maker. Exploring key questions about cybersecurity function, transformations, and relevant exogenous factors inevitably has to be couched in terms understood by decision-makers.
A key takeaway from the UCP study is that LLM outputs must be dissected and analyzed to understand points of convergence and divergence. Doing so allows planners to place their own weight on variables that appear critical in determining the shape of some suppositions versus others.
Then, so armed, planners can inject these findings directly into decision-maker briefings as an alternative to just directly reporting on the outputs of a few AI models. In other words, it is the cross-comparative analysis of how LLMs come to individually interesting conclusions that matter, rather than the generated scenarios or suggestions themselves.
The bottom line: Avoiding the AI CISO
When it comes to using LLMs effectively for cybersecurity planning, the bottom line is clear: Planners and executives must avoid the AI CISO. Simply put, the AI CISO concept describes circumstances where an organization uses AI without effectively incorporating humans into not only the decision-making loop, but also conversations about underlying ethical, methodological, and technical practice.
The result would be the rise of AI systems as de facto decision-makers. Not Skynet or HAL 9000, of course, but support systems to which we delegate too much of what goes into decision-making.
This recent study and others like it lay out initial best practices for accomplishing this. They make the case that effective use of LLMs for robust forecasting and analysis means having humans in the loop at every stage of deployment.
More importantly, they make the case that this engagement has to reflect the full range of human expertise — from specialist know-how to investigative skills and marketing savvy — to get the most out of the machine.