Email Security
AI Security Threats Ranked By What's Actually Exploited
Part 2 of 6: AI Security Market Analysis Series
Most AI security threat rankings use OWASP categorization or theoretical severity scores. This creates prioritization problems when teams try to map generic risk frameworks to their actual deployment environments and budget constraints.
This analysis ranks threats differently: by what's actively exploited in production systems right now, combined with the operational and financial impact when you get hit. Some threats score high on both likelihood and impact. Others are technically severe but rarely exploited outside targeted campaigns. The difference matters when you're deciding what to fix first with limited budget and engineering time.
The threat landscape splits into three tiers based on exploitation patterns observed in 2024-2025 and the maturity of available defenses.
These threats are being exploited at scale right now. Detection and defense solutions exist but require careful implementation to be effective.
Attackers distribute malicious models through Hugging Face, PyTorch Hub, and similar repositories that see over 2 billion monthly downloads. The attack works by embedding backdoors in model pickle files during serialization. When an organization downloads and loads the model, the backdoor executes with the privileges of the loading process.
ReversingLabs identified over 100 malicious models in 2024-2025 using this technique. The "nullifAI" attack they documented is particularly concerning because it exploits three zero-day vulnerabilities in Picklescan, the scanner most teams rely on for model security validation. When your security tooling has exploitable vulnerabilities, you have a foundational problem.
The financial impact when this succeeds runs $2-5 million in incident response costs, not counting intellectual property theft or data exfiltration that may not be discovered for months. Organizations downloading models from public repositories without validation are exposed.
Defense requires model provenance tracking, sandboxed evaluation environments, and verification tooling that goes beyond basic pickle scanning. HiddenLayer and Protect AI offer commercial solutions. Open source alternatives exist through Garak combined with custom validation pipelines, but require 2-3 weeks of engineering work to implement properly.
Traditional phishing relied on content patterns that security tools could detect—grammatical errors, generic greetings, suspicious links. LLMs changed the economics and quality threshold. Attackers now generate hyper-personalized phishing emails with perfect grammar, contextually appropriate content, and legitimate-looking sender information. The same techniques work for voice (deepfake audio) and video (deepfake video calls).
The scale is measurable. Research indicates 82.6% of current phishing campaigns incorporate AI-generated content, up 53.5% from 2024. Success rates hit 60% against targets that previously showed resistance to traditional phishing. The Arup case demonstrated the financial risk: $25 million lost in a single deepfake video conference attack.
Traditional Secure Email Gateways miss 70-85% of these attacks because they rely on content signatures and pattern matching. The attacks look clean at the content layer. Defense requires behavioral analysis that identifies anomalies in communication patterns, sender reputation systems that track historical behavior, and authentication controls that verify sender identity independent of content.
Solutions exist but require a budget. Be sure to scope out the best behavioral AI platforms from vendors like Abnormal, Darktrace, and others depending on deployment size. For organizations focused specifically on email, platforms like AegisAI provide targeted defenses optimized for this threat class. The alternative is accepting that a significant percentage of sophisticated phishing will reach users and investing in detection and response capabilities instead.
Indirect prompt injection happens when AI systems automatically process untrusted content from emails, documents, or web pages that contain malicious instructions. The user doesn't need to do anything—the AI reads the content and follows the embedded commands.
Microsoft Copilot got hit with this in June 2025. The EchoLeak vulnerability (CVE-2025-32711) exploited how Copilot auto-processed Markdown reference links in emails. An attacker could craft an email that, when processed by Copilot, would execute their instructions instead of the user's intended query. Microsoft spent over $2 million in emergency engineering time fixing it and broke three integrations during the patch cycle.
The attack works because LLMs fundamentally struggle to distinguish instructions from data when both are presented as text. Architectural mitigations help—sandboxing tool access, separating planning from execution, implementing least-privilege principles. Runtime guardrails from vendors like Lakera and Prompt Security provide detection, typically achieving 90-95% accuracy, but that remaining 5-10% still represents hundreds of attacks at enterprise scale.
Open source alternatives exist through NeMo Guardrails and similar projects, but require ongoing engineering investment to maintain and tune. Either way, this is a defense-in-depth problem where you need both architecture and detection working together.
These threats appear in the wild but typically target high-value organizations or specific use cases rather than opportunistic campaigns.
Attackers can reconstruct proprietary model weights through systematic API queries without needing direct access to training infrastructure. The technique works by analyzing model outputs across carefully constructed input sets, then using those observations to approximate the underlying parameters.
Research demonstrated that OpenAI's embedding layer could be extracted for under $20 in API costs. For organizations that invested millions in training proprietary models, this represents IP theft at scale. The attack is economically rational when the target model provides significant competitive advantage.
Detection is possible but incomplete. Rate limiting helps but sophisticated attackers distribute queries across multiple accounts and time periods. Monitoring for systematic probing patterns catches obvious attempts. Watermarking and fingerprinting techniques can prove theft after the fact but don't prevent it.
The defense currently available is primarily detection-based. No vendor offers strong prevention guarantees. Organizations training high-value proprietary models need to assume some level of extraction risk and plan accordingly through IP protection strategies, legal frameworks, and competitive moats that don't depend solely on model secrecy.
Contaminated training data creates backdoors that persist through the model lifecycle. The attack works by injecting adversarial examples into training corpora—often through contributions to open datasets, GitHub repositories, or web scraping targets.
Research from 2024-2025 demonstrated that 250 poisoned documents can backdoor models ranging from 600 million to 13 billion parameters, with attack cost remaining roughly constant regardless of model size. DeepSeek's DeepThink-R1 learned backdoors from poisoned GitHub repositories. Grok 4 acquired a universal jailbreak trigger ("!Pliny") from contaminated Twitter data in its training set.
Both were flagship releases from well-funded labs with security programs. The poison persisted for months before discovery. When discovered, the fix requires model recall and retraining at costs exceeding $5 million plus reputational damage.
Defense requires data provenance tracking, deduplication to prevent adversarial repetition, quality filtering based on source reputation, and adversarial example detection during training. Commercial solutions from TrojAI and Robust Intelligence are emerging but still maturing. Most organizations building custom models are implementing defensive data pipelines themselves.
These threats matter for specific deployment patterns but aren't universal priorities.
Multi-tenant infrastructure escapes, jailbreaks and alignment failures, data leakage through embeddings, and similar issues affect organizations in particular circumstances—running shared AI infrastructure, deploying customer-facing chatbots, or using RAG over sensitive documents.
Coverage of these appears in Part 3 when we map vendor solutions to specific deployment scenarios. The key point is that not every organization faces every threat equally. Prioritization depends on your architecture, use cases, and risk tolerance.
The sequence that works based on threat priority and available solutions:
Start with supply chain security. The attack is well-understood, exploitation is active, and effective solutions exist commercially and open source. This is weeks 1-8 of any AI security program.
Address synthetic social engineering if you're seeing measurable phishing impact. The solutions work but require a budget. If email isn't a primary attack vector for your organization, this can wait.
Implement zero-click injection defenses if you're deploying AI assistants that auto-process user content. The combination of architectural controls and runtime guardrails is necessary but not sufficient—you're managing risk, not eliminating it.
Model extraction and training data poisoning matter primarily for organizations building proprietary models. If you're consuming commercial APIs or using pre-trained models without custom training, these drop in priority.
Part 3 maps these threats to the vendor landscape and identifies which solutions actually address which problems versus marketing claims.
Part 3 publishes : AI Security Vendor Landscape: 6 Categories Explained.
