How to Safely Source AI Models from Public Repositories: Lessons from a Supply Chain Attack

Overview

In early 2025, a malicious repository on Hugging Face named Open-OSS/privacy-filter impersonated OpenAI's legitimate Privacy Filter release. Before takedown, it logged over 244,000 downloads and reached the platform's number one trending spot—likely through artificial inflation of likes and downloads. The repository contained a hidden infostealer malware targeting Windows systems, raising urgent questions for enterprises about how they validate and integrate AI models from public registries. This tutorial breaks down the attack, explains how to detect such threats, and provides a practical guide to securing your AI supply chain.

How to Safely Source AI Models from Public Repositories: Lessons from a Supply Chain Attack — Source: www.infoworld.com

Prerequisites

Basic familiarity with AI/ML model deployment and Hugging Face platform.
Understanding of Python scripting, base64 encoding, and command execution (PowerShell, bash).
Familiarity with common security concepts: supply chain attacks, malware analysis, sandboxing.
A development or test environment where you can safely analyze model files (e.g., isolated VM).

Step-by-Step Guide to Understanding and Mitigating AI Model Supply Chain Attacks

1. Analyzing the Attack Vector: How the Malicious Model Worked

Hugging Face repositories include a model card (README), optional code files, and serialized model weights. The Open-OSS/privacy-filter repository copied the legitimate model card almost verbatim, but included a file called loader.py. This script first executed decoy code to appear as a normal model loader, then initiated a concealed infection chain.

Infection chain details:

The loader.py script disabled SSL verification and decoded a base64 string to reveal a URL hosted on jsonkeeper.com.
It fetched remote payload instructions and passed them to PowerShell.
PowerShell downloaded an additional batch file from an attacker-controlled domain and created a scheduled task mimicking a legitimate Microsoft Edge update process to establish persistence.
The final payload—a Rust-based infostealer—targeted Chromium and Firefox-derived browsers, Discord local storage, cryptocurrency wallets, FileZilla configurations, and system information. It also attempted to disable Windows Antimalware Scan Interface (AMSI) and Event Tracing for Windows (ETW), while checking for sandbox or virtual machine environments to evade analysis.

The attack leveraged JSON keeper as a command-and-control (C2) channel, allowing attackers to rotate payloads without modifying the repository.

2. Identifying Malicious Repositories: Red Flags to Watch For

When sourcing models from Hugging Face or similar platforms, inspect the following:

Unusual file names or code files. The fake repository included loader.py and start.bat—these are not standard for a model designed to be loaded via transformers or diffusers.
Discrepancies in the model card. The README diverged by instructing users on Windows to run start.bat or on Linux/macOS to run python loader.py. Legitimate models rarely require manual execution of scripts.
Artificially inflated metrics. The repository gained 244K downloads and 667 likes in under 18 hours—numbers that researchers flagged as likely inflated by bots.
Pickle files. Previous attacks have hidden malicious code inside Pickle-serialized model files that bypass Hugging Face's scanners. Look for .pkl or .pt files that may contain unsafe deserialization payloads.

3. Implementing Protective Measures for Your AI Pipeline

To defend against supply chain attacks from public model repositories, adopt these practices:

Prefer safe serialization formats. Hugging Face recommends using safetensors instead of .pickle or .pt files. Safetensors are not vulnerable to arbitrary code execution during deserialization.
Scan all code and model files before use. Use static analysis tools to detect base64-encoded URLs, SSL verification disabling, suspicious PowerShell commands, or known C2 indicators. For Python files, review imports and unexpected os.system or subprocess calls.
Run models in isolated environments. Use containers (Docker) or virtual machines with no network access or with restricted egress rules. Never execute untrusted scripts directly on production machines.
Verify the source and maintainer. Check the repository owner's history, other repositories, and social presence. Impersonators often have recently created accounts or a single repository.
Use a model registry with security scanning. Some platforms (e.g., Hugging Face Hub's scanning) are improving, but you should implement your own scanning pipeline that checks for known malicious patterns.

4. Monitoring and Incident Response

If you suspect a model has been downloaded or executed from a malicious repository:

Check for scheduled tasks with names like MicrosoftEdgeUpdateTaskMachine or similar—attackers often mimic legitimate system tasks.
Review PowerShell logs for suspicious download commands (e.g., Invoke-WebRequest to unknown domains).
Look for network connections to jsonkeeper.com or other paste/JOSN hosting services used as C2.
Sweep for the presence of additional batch files or scripts in temporary directories.
If the Rust infostealer deployed, assume credentials, browser data, and cryptocurrency wallets are compromised. Reset passwords, rotate API keys, and notify affected users.

HiddenLayer also identified six additional malicious repositories in the same campaign—monitor Hugging Face advisories and security blogs for indicators of compromise (IOCs).

Common Mistakes to Avoid

Trusting trending repositories without verification. Popularity can be artificially inflated; always inspect code and history.
Ignoring code files in model repositories. Models often come with preprocessing scripts or loaders—these can hide malicious code.
Running containers with privileged access or full network. A sandboxed environment with minimal permissions limits damage.
Bypassing SSL verification in code. If a loader script disables SSL, it’s a major red flag—attackers often do this to avoid certificate checks on their malicious endpoints.
Failing to update security policies. The AI supply chain is still less mature than traditional software supply chains. Regularly review your organization’s model sourcing and validation policies.

Summary

The Open-OSS/privacy-filter incident shows how quickly malicious AI models can spread through public repositories, reaching hundreds of thousands of downloads within hours. The attack leveraged social proof (trending, likes) and a disguised loader.py to execute a multi-stage infection that ultimately deployed a Rust infostealer. Enterprises must adopt layered defenses: verify repository authenticity, inspect all code and model files, use safe serialization formats like safetensors, and run models in isolated environments. As AI adoption grows, supply chain security tools and practices must catch up to protect sensitive credentials and internal systems from similar threats.

Tags: