Home / Critical Brief / No. 116
No. 116 Pillar 02 · Verifiable AI

A fake OpenAI model hit #1 trending on Hugging Face

publisher provenance never verified before execution

Model Supply Chain Code ProvenanceIdentity & Auth
Incident date
2026-05-07
Published
2026-07-31
Authors
Lemma Critical Team
Related Pack
Pack A · Incident Response

TL;DR

On May 7, 2026, HiddenLayer disclosed a malicious Hugging Face repository, Open-OSS/privacy-filter. It typosquatted OpenAI’s legitimate openai/privacy-filter release, copied the model card nearly verbatim — including the link to OpenAI’s official documentation — and passed as legitimate. Only the README was swapped, instructing users to run start.bat on Windows or loader.py on Unix-like systems. The loader.py fetches a PowerShell command from a remote server and executes an infostealer. The repository reached #1 trending, 244,000 downloads, and 667 likes in under 18 hours, though those numbers were almost certainly artificially inflated. Detection through scanning and trending monitoring worked after the fact. What was missing is the layer that verifies, before execution, whether the model’s publisher is really OpenAI.

What happened

  • The malicious repository Open-OSS/privacy-filter typosquatted OpenAI’s legitimate openai/privacy-filter release, copied the model card nearly verbatim, and even carried the link to OpenAI’s real model card PDF to look legitimate.
  • It diverged from the legitimate project in exactly one place — the README — which told users to clone the repo and run start.bat (Windows) or python loader.py (Linux/macOS) directly.
  • After running decoy code to look like a real loader, loader.py fetches a PowerShell command from a remote server and ultimately executes a credential-harvesting infostealer.
  • Before removal, the repository reached #1 trending, roughly 244,000 downloads, and 667 likes in under 18 hours. (Attribution points to a Chinese-speaking actor, but that is not this Brief’s focus.)

The attack succeeds through the following chain.

  1. A user looks for openai/privacy-filter and lands on Open-OSS/privacy-filter, near-identical in name and appearance. The model card is a verbatim copy, down to the link to OpenAI’s official documentation.
  2. Reputation signals — #1 trending, 244K+ downloads, hundreds of likes — are taken as grounds that it is legitimate.
  3. Following the README, the user runs start.bat or loader.py.
  4. loader.py fetches a PowerShell command from a remote server, the infostealer executes, and browser credentials, tokens, and wallets are stolen.

Timeline — disclosure and response

  • 2026-05-07: HiddenLayer identifies malicious code in Open-OSS/privacy-filter and discloses. At the time the repository sat among the platform’s top trending, with over 240,000 downloads.
  • 2026-05-07: HiddenLayer reports its findings to Hugging Face’s security team.
  • 2026-05-08: Hugging Face confirms a terms-of-service violation and removes the repository.

Note: the facts here rest on HiddenLayer’s primary research and established media reporting. The download and like counts were almost certainly artificially inflated and do not represent the real scale of cloning or execution. The majority of the 667 accounts that liked the repository followed mechanical naming patterns — firstname-lastname### (504) and adjectivenoun#### (153) — indicating bot-driven manipulation of the trending list. This Brief focuses not on the attackers’ attribution or motives, but on the structure in which a model’s publisher provenance is not independently verified before execution.

Response and industry movement after disclosure are as follows.

  • HiddenLayer treated any Windows host that cloned the repository and executed the files as compromised, recommending reimaging and rotation of all credentials.
  • It also identified a set of malicious repositories under another account using the same command-retrieval URL, noting this is likely part of a broader supply-chain operation targeting open-source ecosystems.

Why it wasn’t stopped

The failure here is not that scanning was weak, nor that users were careless. It is that the trust signals a model hub provides — trending rank, download count, likes, the appearance of the model card — prove nothing about the model’s publisher provenance, that is, whether OpenAI actually published it. Detection worked: the malicious code was identified by scanning, and the repository was removed after the report. What was missing sits earlier — verification, at the instant the user executes, that the model artifact and its publisher are legitimate.

The user executed without any means to independently verify the publisher’s identity. The verbatim model card, the link to official documentation, the #1 trending rank — all looked like a “legitimate OpenAI model.” But looking right and ranking high is not proof of being the publisher one claims to be. In this case the reputation signal itself became the attack surface: a bot-inflated trending rank stood in for legitimacy and drew users toward execution.

Download and like counts are meant to serve as cues to a model’s trustworthiness. But they can be generated mechanically and the ranking can be manipulated. Looking like “many people use this” is not proof that “a legitimate publisher released it.”

The same structure connects to Brief 101 (Paysafe fake payment SDK) and Brief 004 (Megalodon GitHub supply chain), where artifacts impersonating legitimate packages or releases were pulled in before their provenance could be checked, and to Brief 010 (Claude Code leak lure), where a legitimate-looking lure went unverified before execution. Each shows that an artifact “looking legitimate” and its publisher and contents being verified right now are separate questions.

What proof would have changed

Proof-as-auth inserts a layer, ahead of each individual action in which a model is pulled from a hub and executed, that independently verifies the provenance of the artifact and its publisher. Rather than letting trending rank, download count, likes, or the appearance of a model card stand in for legitimacy, it checks — before execution completes — whether this model is an untampered artifact published by the claimed publisher. If the answer is “provenance unknown” or “publisher does not match,” the pull and the execution are held in advance.

The design Lemma offers for this primitive is as follows.

  • Pre-execution verification of publisher provenance: bind pulling and executing a model not to a match in name or appearance, but to the publisher’s verifiable provenance — so a typosquat posing as openai/privacy-filter is separated out as lacking the legitimate publisher’s provenance before execution.
  • Provenance replacement of reputation signals: remove manipulable reputation signals — trending rank, download count, likes — from the basis for judging legitimacy, and replace them with provenance proof, excluding bot-inflated numbers from standing in for legitimacy.
  • Binding of artifact integrity: bind not the appearance of the model card or README, but the integrity of the model artifact itself to verifiable provenance, so an artifact disguised as legitimate through copied metadata is separated out as lacking proof before execution.
  • Provenance verification built into CI paths: embed provenance verification into the CI and automation paths that pull models from a hub, stopping a model that lacks provenance before it is pulled or executed.

Lemma is not a product that scans and removes malicious code, nor one that governs a hub’s reputation ranking. Its scope is to independently verify the provenance of the artifact and its publisher before a model is pulled and executed, excluding the passage of typosquats and disguised artifacts before execution. Detection (identifying malicious code, removing the repository, sharing IOCs) and pre-execution proof (a record that independently verifies publisher provenance and artifact integrity before pull and execution) are complements, not substitutes. See Proof-as-Auth: sign in without sending your key (Lemma, 2026-05) and Verifiable AI; the view that binds publisher and artifact provenance also connects to Verifiable Origin.

Sources

Proof-as-Auth: sign in without sending your keyVerifiable AIVerifiable OriginBrief 101 (Paysafe fake payment SDK)Brief 004 (Megalodon GitHub supply chain)

This material is a structured analysis of public information; it is not an audit, diagnosis, or recommendation for any specific organization.

Cite this Brief

Lemma Critical Team. (2026).
"A fake OpenAI model hit #1 trending on Hugging Face — publisher provenance never verified before execution".
Lemma Critical Brief No.116. Lemma / FRAME00, Inc.
https://lemma.frame00.com/critical/briefs/116-open-oss-privacy-filter-fake-model/
Related BriefsRelated
No. 115 AI applicant-screening discrimination suit against Workday: the adverse action was taken, but whether the decision was independently verified and authorized was never recorded AI Bias / Harm No. 099 Agentjacking: an AI coding agent trusted a single fake error report as its "resolution steps" and ran the attacker's commands AI Decision Integrity No. 098 BioShocking: convince an AI browser "it's a game" and it drops its guardrails and hands over credentials AI Decision Integrity
Lemma

If it can't be verified,
it doesn't enter your operation.

Lemma attaches cryptographic proofs to data and AI execution, so the receiving side can confirm authenticity without asking the issuer. Detection stays; a proof layer is added in front of it.

Talk to enterprise See the verification center Subscribe to Critical Weekly →