Barracuda has reported on how generative artificial intellegence (AI) is being used to create and distribute spam emails and craft highly persuasive phishing attacks. These threats continue to evolve and escalate — but they are not the only ways in which attackers leverage AI. Security researchers are now seeing threat actors manipulate companies’ AI tools and tamper with their AI security features in order to steal and compromise information and weaken a target’s defenses. Learn more in this Threat Spotlight.
Email attacks targeting AI assistants
AI assistants and the large language models (LLMs) that support their functionality are vulnerable to abuse.
Barracuda’s threat analysts have found attacks where malicious prompts are hidden inside benign-looking emails. This malicious payload is designed to manipulate the behavior of the target’s AI information assistants.
For example, a recently reported — and fixed — vulnerability in Microsoft 365’s AI assistant, Copilot, could allow anyone to extract information from a network without authorization. Threat actors can exploit this to collect and exfiltrate sensitive information from a target.
They do this by leveraging the ability of internal AI assistants to look for and collate contextual data from internal emails, messages and documents when answering queries or completing tasks.
- First, the attackers send one or more employees a seemingly harmless email containing a concealed and embedded malicious prompt payload.
- This email needs no interaction from the user and lives benignly in their inbox.
- When the employee asks the AI assistant for help with a task or query, the assistant scans through older emails, files and data to provide context for its response. As a result, the AI assistant unwittingly infects itself with the malicious prompt.
- The malicious prompt could then ask the AI assistant to silently exfiltrate sensitive information, to execute malicious commands or to alter data.
Weaponized emails also try to manipulate AI assistants by corrupting their underlying memory or data retrieval logic. This includes emails with exploits targeting vulnerabilities in RAG (Retrieval-Augmented Generation) deployments. RAG is a technique that enables the LLMs to retrieve and incorporate new information beyond their training model.
Such attacks can lead to AI assistants making incorrect decisions, providing false information, or performing unintended actions based on corrupted data.
Tampering with AI-based protection
Attackers are also learning how to manipulate the AI components of defensive technologies.
Email security platforms are being enhanced with AI-powered features that make them easier to use and more efficient, including features such as auto-replies, ‘smart’ forwarding, auto-triage to remove spam, automated ticket creation for issues, and more. This is expanding the potential attack surface that threat actors can target.
If an attacker successfully manipulates these security features, they could:
- Manipulate intelligent email security tools to autoreply with sensitive data.
- Abuse AI security features to escalate helpdesk tickets without verification. This could lead to unauthorized access to systems or data, as attackers could exploit the escalated privileges to perform malicious activities.
- Trigger workflow automation based on a malicious prompt. This could lead to the execution of harmful actions, such as deploying malware, altering critical data, or disrupting business operations.
Casting doubt on reality
Identity confusion and spoofing
When AI systems operate with high levels of autonomy, they can be tricked into either impersonating users or trusting impersonators. This can lead to:
- ‘Confused Deputy’ attacks: This involves an AI agent with higher privileges performing unauthorized tasks on behalf of a lower-privileged user, such as an attacker.
- Spoofed API access: This involves existing AI-based integrations with Microsoft 365 or Gmail, for example, being manipulated to leak sensitive data or send fraudulent emails.
Cascading hallucinations: trusting the untrue
As mentioned above, email attacks targeting AI assistants can try to manipulate the assistant’s functionality. This could lead the assistant to summarize a user’s inbox, generate reports, and set the calendar — but based on false or manipulated data.
In such cases, a single poisoned email could:
- Mislead task prioritization. For example, sending “urgent” emails from fake executives
- Skew summaries and recommendations
- Influence critical business decisions based on hallucinations
How email defenses need to adapt
Legacy email gateways, traditional email authentication protocols such as SPF or DKIM and standard IP blocklists are no longer enough to defend against these threats. Organizations need an email security platform that is generative-AI resilient. This platform should include:
- LLM-aware filtering: It should be able to understand email context (topic, target, type, etc.), tone and behavioral patterns in addition to the email content.
- Contextual memory validation: This helps to sanitize what AI-based filters learn over time and can prevent long-term manipulation.
- Toolchain isolation: AI assistants need to operate in sandboxes, with measures in place to block any unverified action based on a received email prompt.
- Scoped identity management: This involves using minimal-privilege tokens and enforcing identity boundaries for AI integrations.
- Zero-trust AI execution: Just because an email claims to be “from the CEO” doesn’t mean the AI should automatically act on it. Tools should be set to verify everything before execution.
The future of email security is ‘agent-aware’
The AI tools being used within organizations are increasing built on agentic AI. These are AI systems capable of independent decision-making and autonomous behavior. These systems can reason, plan and perform actions, adapting in real time to achieve specific goals.
This powerful capability can be manipulated by attackers, and security measures must shift from passive filtering to proactive threat modeling for AI agents.
Email is a great example. Email is becoming an AI-augmented workspace, but it remains one of the top attack vectors. Security strategies need to stop seeing email as a channel. Instead, they need to approach it as an execution environment requiring zero-trust principles and constant AI-aware validation.
How Barracuda Email Protection helps defend against AI attacks
Barracuda’s integrated cybersecurity platform is purpose-built to meet the dual challenge of AI-based attacks and attacks targeting AI components.
Our email protection suite combines intelligent detection, adaptive automation, and human-centric design to help customers outpace AI-powered threats.
This includes:
- Advanced AI-based detection: Barracuda uses behavioral AI and natural language processing (NLP) to spot social engineering even without obvious malware or links. It catches impersonation, business email compromise (BEC), and tone-shift anomalies that traditional filters miss.
- Defense-in-depth: Barracuda covers every stage of the kill chain from phishing prevention to account takeover detection and automated incident response, closing the gaps that attackers exploit.
- Real-time threat intelligence: With data from a global detection network, Barracuda rapidly adapts to evolving threats like prompt injection, RAG poisoning and AI hallucination abuse.
- User training and awareness: Technology alone isn’t enough. Barracuda empowers employees to recognize AI-powered phishing through ongoing awareness training because trust is the new vulnerability.
For more information: https://www.barracuda.com/solutions/ai-in-cybersecurity
This article was originally published at Barracuda Blog. Learn more about current threat trends by reviewing past Threat Spotlight articles.
Photo: Premreuthai / Shutterstock
This post originally appeared on Smarter MSP.