Your LLM can change its answers while it's still running

Security reviews of language-model applications usually begin above the model: prompts, tools, retrieval, authentication, and generated output. The model file itself is often treated as static input. It is verified before deployment, opened by the inference server, and then forgotten.

That assumption can fail while the server is still running.

I built a proof of concept that modifies selected weights in a GGUF file used by llama-server. Under the default memory-mapped loading path, later requests reflect the new values without restarting the process, injecting code, attaching a debugger, or writing directly to process memory.

This is not a flaw in the Transformer architecture. It is a failure of runtime integrity caused by the interaction between file permissions, memory mapping, and shared storage.

The threat model

The attacker needs write access to the GGUF file used by the running server. They do not need root access or control of the inference process.

That precondition should be prevented in a production deployment, but it is easy to create accidentally. Common examples include a model directory mounted read-write into several containers, local experiments running beside an inference service, automated model-management tools, and weak separation between model storage and application workloads.

The attack does not cross a permission boundary. It shows why write access to a loaded model must be treated as equivalent to write access to the behavior of the service.

Why a file change reaches a running model

By default, llama-server can load GGUF data using mmap. The operating system maps file-backed pages into the process address space instead of copying the entire model into a private buffer.

When another process writes to the same file, the kernel updates the corresponding cached pages. The inference process may then observe the modified values on later reads, even though it never reopened or reloaded the model.

From the server's perspective, the mapping is read-only. From the system's perspective, the underlying file remains shared state. Those are different guarantees.

Running llama.cpp with --no-mmap prevents this specific propagation path because the model is loaded into private memory. It does not make writable model storage a good design, but it changes this experiment from a live modification into an offline one.

Targeting the final projection

Randomly corrupting weights usually produces a broken model, not controlled behavior. The proof of concept instead modifies output.weight, the final matrix used to convert the hidden state into token logits.

In simplified form, generation ends with:

logits = hidden_state @ output.weight

Rows of this matrix correspond to vocabulary tokens. Increasing the magnitude of selected rows raises the chance that those tokens dominate the final distribution. This gives the attacker a direct, if crude, way to influence generated text.

I tested the technique on TinyLlama 1.1B Chat in Q4_K_M GGUF format. Its output.weight tensor is stored as Q6_K. The proof of concept understands the GGUF structure rather than searching for a byte pattern. It reads the header and metadata, follows tensor descriptors and alignment, locates output.weight, and calculates the absolute file offsets of the relevant quantization blocks.

In Q6_K, a block represents 256 values in 210 bytes. Its final two bytes store an fp16 scale value usually called d. Multiplying this scale increases every dequantized value in that block. Applying the change across the blocks belonging to one output row amplifies the logit for the corresponding token without rebuilding the model.

This implementation is deliberately format-specific. A different quantization type has a different block layout and requires a different modification strategy.

Controlling a sequence requires more than one token

The target string in my experiment was Pwned. TinyLlama tokenized it into three tokens: [349, 1233, 287].

Boosting all three by the same factor produced poor results. Autoregressive generation changes the context after every token. The first token must win before the target prefix exists, the middle token competes in a different distribution, and the final token may already be likely once the prefix has been generated. Excessive amplification also causes repetition and general response degeneration.

The script therefore applies different multipliers by position. The first token receives a base value, middle tokens receive a stronger value, and the final token receives a lower value. This is a heuristic rather than a universal formula, but it produced a much cleaner result than uniform amplification.

I evaluated the modification through both /completion and /v1/chat/completions. Testing both matters because chat templates alter the prompt seen by the model and therefore change the natural probabilities of the target tokens.

The complete experiment

The proof of concept follows a reproducible sequence:

1. Send baseline prompts to the unmodified model.
2. Parse the GGUF file and locate the target tensor rows.
3. Save the original scale values to a JSON backup.
4. Modify the selected Q6_K scale fields and flush the writes.
5. Repeat the prompt set against the same running server.
6. Restore the original values from the backup.

The process remains healthy throughout the test. There is no restart, crash, or obvious infrastructure alert. Only the generated behavior changes.

This is what makes the issue operationally interesting. Standard monitoring may show a healthy container, stable memory use, and successful HTTP responses while the model has stopped behaving like the artifact that was originally approved.

Limits of the proof of concept

The current implementation relies on several specific conditions:

• the running service uses the writable GGUF file through mmap;
• output.weight exists as a separate tensor rather than being tied to the input embeddings;
• that tensor uses the Q6_K layout supported by the parser;
• the attacker already has write access to the model artifact.

The modification is also intentionally obvious. Forcing a word into many responses is useful for demonstration, but not subtle. More interesting research would ask whether smaller changes can alter one category of answers, weaken a refusal boundary, or bias a classifier while leaving ordinary regression prompts unchanged.

Protecting runtime model integrity

The first control is simple: inference workloads should not share writable model storage with less trusted components. Mount model directories read-only, copy approved artifacts into a private runtime location, and separate model management from model execution.

Hashing the model before startup is useful but incomplete. A startup hash proves what was loaded at one moment. It does not prove that the backing file remained unchanged. Periodic integrity checks, immutable storage, and alerts on writes to active model artifacts provide stronger guarantees.

Behavioral regression tests add another layer. A small reference prompt set can detect large shifts, although subtle manipulation may stay within normal sampling variance. File integrity and behavioral monitoring solve different parts of the problem and should not be treated as substitutes.

The broader lesson is that an open-weight model is part of the live system, not just a deployment asset. Its format, permissions, loading strategy, page-cache behavior, and surrounding volume mounts all affect the integrity of inference.

The reproducible environment and attack implementation are available in the llm-inference-tampering repository.