Repository navigation
Scientific paper proposal: When AIOps Become "AI Oops": Subverting LLM-driven IT Operations via Telemetry Manipulation - #3128
Conversation
|
Readme is not correctly formatted Got: ['Assignment Proposal', 'Title', 'Names and KTH ID', 'Deadline', 'Category', 'Description', 'Relevance'] |
|
Readme is not correctly formatted Got: ['Assignment Proposal', 'Title', 'Names and KTH ID', 'Deadline', 'Category', 'Description', 'Relevance'] |
|
We would like to do feedback on your paper if that is possible. |
Of course :) |
Paper FeedbackOverall, the group discussed an interesting and relevant security problem: attacker-controlled telemetry and how it could feed into LLM-based AIOps to drive an unsafe automated response. The group clearly kept their focus on how attacker-controlled telemetry could interact with LLM-based operation decision making to drive automated remediation. The attack demonstration of perturbing error messages and telemetry showed the attack idea well. They also covered attack design, defense, evaluation, limitations, related work and real world applicability. Evaluation of CriteriaRelevance to DevOpsThe report analysed by the group explains the risks associated with using AIOps agents in IT operations, specifically focusing on the vulnerability of telemetry data to manipulation by attackers and the resulting harmful remediations. Conceptual ExplanationThe group explains the problem in a clear and understandable way, by comparing traditional direct prompt injection to this indirect telemetry manipulation, explaining how attackers use adversarial reward-hacking to trick an AIOps agent. The use of AIOps is well explained and they make it clear on why it’s an important issue. Technical AnalysisThe group did a good job of explaining the high-level attack workflow, including scanning services, identifying endpoints, generating telemetry-injection payloads, and attempting to influence remediation. The discussion of the defensive mechanism also provided useful technical insight. However, one area of improvement would be if the presentation went more depth in explaining a technical component. The criterion asks for analysis of a key technical component in detail. Rather than describing several components briefly, it would be better to select maybe one component and explain its operation in greater technical detail, e.g. the AIOps Shield. Critical EvaluationLimitations are defined and explained. The group explained how the attacker needs to influence telemetry, how the defense only addresses the telemetry-injection path, and how the experiments use simulated environments. These are good observations and demonstrate awareness that the experimental results should not automatically be generalized to all production AIOps systems. Related WorkRelated work were discussed, or at least closely related topics. Indirect Prompt Injection and Spotlighting, and they explained the basic difference between them and the paper's approach. This is a good starting point because it demonstrates that the paper is part of a broader research area involving attacks and defenses against LLM-based systems. ReflectionThe final discussion of who should care about the paper was a strong part of the presentation. The group identified organizations using autonomous AIOps agents with the ability to recommend or execute remediation as particularly relevant, while noting that the approach may be unnecessary for small projects without autonomous AIOps. This could be developed further by considering the trade-off between automation and security. For example, when is autonomous remediation worth the risk introduced by trusting telemetry? Narrative and SlidesThe slides that the group provided and presented, helps to explain the concepts discussed. It followed a reasonable progression from AIOps and the problem, to the attack, defense, evaluation, limitations, related work, and practical implications. The slides were also not filled with a lot text so the listeners could focus on listening on what was said rather than just reading everything from the screen. SpeechThe group delivered an engaging presentation. They held a good speech speed consistency throughout the whole presentation. Clear annunciation and spoke in a way that made it possible to understand what they were saying. TimingThe presentation was within the mandatory time slot. CertificationWe certify that generative AI, incl. ChatGPT, has not been used to write this feedback. Using generative AI without permission is considered academic misconduct. |
4 participants
Assignment Proposal
Title
When AIOps Become "AI Oops": Subverting LLM-driven IT Operations via Telemetry Manipulation
Names and KTH ID
Deadline
Category
Description
Modern IT operations increasingly use LLM-based AIOps agents to analyze telemetry, diagnose incidents, and suggest or execute remediation. This creates a new security problem because the telemetry that guides the agent can be influenced by an attacker and may cause the agent to make harmful decisions.
The paper we have chosen to present, “When AIOps Become “AI Oops”: Subverting LLM-driven IT Operations via Telemetry Manipulation”, presents an approach for studying and exploiting this attack surface in AIOps systems. Instead of relying on direct prompt injection, the paper introduces AIOpsDoom, an automated attack framework that uses reconnaissance, crawling, and fuzzing to inject attacker-controlled content into logs, metrics, or traces. The injected content is then crafted as adversarial reward-hacking, making a malicious remediation appear to be a plausible solution to the incident. The paper also proposes AIOpsShield, a defense that identifies untrusted telemetry during a setup phase and sanitizes it before it reaches the AIOps agent.
During the presentation we cover the problem of securing LLM-driven AIOps, explain the AIOpsDoom attack workflow and AIOpsShield defense in detail, and discuss the evaluation. We also critically evaluate the assumptions and limitations of the approach with related approaches and reflect on when telemetry sanitization is useful, when it may be insufficient, and who should rely on it.