Untrusted Models: AI Supply-Chain Risk for Defenders

Teams download a model the way they download a CSV. Two of the three usual ways to load one execute code written by whoever published it, before a single inference runs. MITRE ATLAS tracks this as machine-learning supply-chain compromise, and it is the least exotic AI risk on the board - software provenance with tensors attached.

Where the code execution lives

1. Pickle-based checkpoints. .pt, .pth, .bin and .ckpt files are usually Python pickle streams. Unpickling is not parsing: the format has opcodes that import a module and call it, which makes loading an untrusted checkpoint equivalent to running its author's script as your serving user. Recent PyTorch releases default torch.load to weights_only=True, which refuses those constructs. The realistic failure is a developer who hits a load error and passes weights_only=False to make it go away.
2. Repo-supplied model code. Passing trust_remote_code=True to a from_pretrained call imports Python modules from the model repository itself. It is a documented feature and a complete execution path; the flag name is the whole security review.
3. Everything shipped beside the weights. Conversion scripts, custom serving handlers and a requirements file all run in your environment with your service account's rights.

The safer artifact is safetensors: a flat tensor container with a JSON header, no callables, memory-mappable. When a repo offers both formats, pin the safetensors file explicitly rather than letting the loader choose.

Pin the thing you actually reviewed

Model repos are mutable, like a git branch. Referring to a model by name alone means "whatever is in the default branch today", including whatever it becomes next week. Pin to a commit revision, mirror the artifact into your own registry, and record the digest.

model = AutoModel.from_pretrained(
    "internal-mirror/summarizer",
    revision="9f3c1ad24b0e7c5d8a1f6b2e4c7d9013a5b8e6f2",  # a commit, not a branch name
    trust_remote_code=False,
)

Then keep an inventory row per model: source, revision, digest, license, approver. It is the SBOM question with a different noun, and someone will eventually ask it in an audit.

Detect the load going wrong

A malicious checkpoint has to do something once it runs, and on a serving host that something is loud. Alert on a shell or downloader spawned under the Python process, on outbound connections to anything but your registry and telemetry, and on writes to SSH keys, cron or unit files by the serving user.

title: Shell Or Downloader Spawned By Python On A Model Serving Host
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    ParentImage|endswith:
      - '/python'
      - '/python3'
    Image|endswith:
      - '/sh'
      - '/bash'
      - '/curl'
      - '/wget'
  condition: selection
falsepositives:
  - Training and build jobs that legitimately shell out
level: medium

Scope that to the inference fleet. On a data-science workstation it is pure noise; on a host whose only job is serving a pinned model it is close to silent. Pair it with egress allowlisting on the GPU nodes, which is the highest-value control available here - a checkpoint that cannot reach the internet is mostly just a crash.

Scanning helps, and then it stops

Pickle scanners read the opcode stream and flag imports of os, subprocess, builtins.eval or socket. Run one at the gate into your registry, because a checkpoint referencing those is not a checkpoint.

What scanning cannot see is a backdoored weight set - a model that behaves normally except on a trigger input. No file inspection finds it, and evaluation only covers behaviours you thought to test. That gap is the argument for provenance over inspection: prefer models whose training you can attribute, keep the previously approved revision warm, and make rollback a one-line change instead of a project.