LiteLLM compromised again through its own release pipeline

LiteLLM has suffered a second software supply-chain compromise in seven months. The difference from March changes which deployments need investigation.

In March, malicious PyPI releases exposed users to credential theft, while the official proxy image was unaffected. On October 21, version 1.107.0 delivered malware through both Python wheels and official container releases.

This time, a clean source tag did not describe the software users received. The release process itself had become part of the attack.

What changed since March

LiteLLM's March incident report identified versions 1.82.7 and 1.82.8 as compromised. The team linked that incident to its Trivy scanning dependency and described unauthorized package uploads that bypassed the official release workflow. It subsequently introduced a revised CI/CD pipeline with isolated environments and additional release gates.

The October incident crossed a different boundary.

March 24	October 21
Malicious PyPI versions 1.82.7 and 1.82.8	Malicious release 1.107.0
Official proxy Docker image unaffected	PyPI wheels and official container artifacts affected
Unauthorized uploads reportedly bypassed the release workflow	Compromised developer access altered and approved the release workflow

This comparison does not establish the same attacker or the same initial intrusion method. It establishes that a deployment path spared in the first incident was affected in the second. Reusing March's exposure assessment would miss October's container users.

The approval existed. The attacker could satisfy it.

LiteLLM's October account says three engineers' GitHub accounts were involved, apparently through stolen GitHub CLI OAuth tokens. One account changed and triggered the release workflow; another approved production publication. The team has not established how the tokens were stolen.

Its timeline records branch protection rejecting a workflow change, followed by a privileged override five seconds later. Credentials associated with a third account later deleted workflow records.

The security question is therefore more specific than whether a project requires approval. Does an approval represent an independent decision when an attacker controls several identities? Can an ordinary developer credential also override the rule meant to constrain that developer?

Separate usernames provide limited protection if their sessions can be stolen together. A stronger release design restricts who can change workflow definitions, makes bypass authority exceptional, and records changes outside the administrators' immediate control. Production approval should bind to the exact source revision and artifact being promoted, rather than merely authorizing a workflow run.

Those are design requirements, not a claim that any single control would have prevented this incident.

The official tag and the shipped artifact diverged

Socket's analysis found that container build provenance named Berri-AI/litellm, with a hyphen, instead of the legitimate BerriAI/litellm. The recorded source revision contained the startup payload files found in a malicious wheel. The official source tag did not contain those files.

That is why reviewing the public repository or seeing a familiar registry name is insufficient. The review must reach the artifact that will execute.

Build provenance can expose a mismatch, but only if deployment policy checks its contents. Merely collecting an attestation leaves the decision to someone else. A release gate should reject an unexpected source repository, an unapproved workflow revision, or an artifact whose digest differs from the reviewed build.

Likewise, a signature answers a narrower question than whether software is safe. A compromised authorized build process may produce malicious output. This is not a claim that the October artifacts passed a particular signature verification check. It is why signer identity, permitted source, build configuration, and final contents need separate checks.

Digest pinning helps preserve an approved artifact. Pinning a malicious digest preserves the malicious artifact just as reliably.

Python startup was enough

Upwind identified seven affected wheels and three payload files: cook.pth, cook.py, and sitecustomize.py. The startup hooks can activate the malicious code when Python starts in the affected environment, without an application explicitly importing LiteLLM.

That changes incident response. Do not begin by running the affected environment's Python or pip executable to ask which version is installed. Inspect package metadata, image records, and filesystem evidence with trusted tooling from outside that environment.

Socket describes both credential collection and a separate remote-Python loader. Its observations did not establish an active second-stage destination at analysis time. The loader's presence is evidence of capability, not proof that it successfully ran a later payload on every affected host.

Wiz also reported subsequent use of stolen AWS credentials, including identity checks and model-service enumeration. This is evidence of downstream abuse in observed environments, not evidence that every installation lost the same secrets.

The practical exposure boundary is what the compromised process could read or authenticate to. A gateway with provider keys, a database connection, and a cloud workload identity needs a broader investigation than an isolated test environment with no credentials.

Distribution was contained. Installed copies still matter.

LiteLLM reports removing the malicious wheels and removing or repointing the affected image tags. Its reported PyPI window was 11:39 to 13:23 UTC on October 21; the container distribution window extended to 14:20 UTC. It says the private enterprise registry was unaffected.

Removal limits further distribution. It does not erase a downloaded wheel, replace a running container, or revoke credentials already copied elsewhere.

Floating tags make historical evidence important. A deployment configured with latest, main-latest, or main-stable may have resolved to the malicious image during the incident and resolve to something else now. Checking the tag today cannot reconstruct what ran yesterday.

Use resolved image digests and deployment history. Include jobs that finished and containers that have already disappeared. A short-lived CI runner can expose credentials even when no affected workload remains online.

Respond without restarting the payload

Treat this as a possible host or workload compromise, rather than a routine dependency update.

1. Stop affected deployments and isolate suspect workloads. Block 1.107.0 artifacts from being promoted. Preserve relevant filesystem, process, and network evidence before replacing environments.
2. Establish what actually executed. Check lockfiles, package metadata, CI logs, image digests, and registry caches using trusted tools. Distinguish a stored download from an environment that started Python with the malicious files installed.
3. Rebuild from reviewed artifacts. Replacing the litellm package directory alone may leave startup files at the environment's top level. Do not treat uninstalling the package as proof that a background process has stopped or the host is clean.
4. Revoke exposed access from a clean system. Inventory secrets readable by the affected process, including provider credentials, database access, developer sessions, and cloud identities. For temporary cloud credentials, follow the provider's session-containment procedure rather than only replacing a static key.
5. Look for use after theft. Review provider and cloud audit records for unexpected callers, model access, and permission discovery. Replacing the workload does not stop an attacker using credentials elsewhere.

Useful investigation leads include cook.pth, cook.py, unexpected sitecustomize.py content, and traffic to www[.]litellm[.]dev/healthcheck. The domain is defanged here. Do not visit it. A filename alone is not conclusive, and absence of these indicators is not a clean bill of health.

What a second compromise should change

After one incident, recovery often ends with credential rotation and a repaired pipeline. After a second, the organization needs to test the assumptions behind that repair.

Can an attacker with developer access change what the release job builds? Can the same compromise supply the approval? Will deployment reject a lookalike repository even when the build succeeds? Can investigators recover audit evidence after workflow records are deleted?

For downstream AI teams, there is a separate test. If a model gateway executes malicious dependency code before it handles a request, what prevents that code from reading unrelated credentials or sending them out?

Keep the gateway's permissions narrow. Avoid mounting developer home directories or unrelated secrets into its runtime. Separate build-time access from production identities, and monitor unexpected file access and outbound connections independently of the gateway's own logs.

Prompt filters cannot inspect a package before Python starts. Request-level guardrails cannot revoke a stolen cloud session. AI infrastructure needs release controls and runtime containment as well as controls on model behavior.

The October incident shows why the recovery criterion must go beyond restoring a familiar version number. Teams need evidence that the artifact they approved is the artifact running, and that compromising it does not grant access to every credential around it.

Published October 22, based on incident disclosures available late October 21 UTC. The investigation into the initial credential theft remains ongoing.