--- title: "If your model has a public API, it can be stolen" description: "A public prediction endpoint can become a labeling service, an unofficial inference backend, or a training source for a replacement model." date: 2025-11-14 category: "technical" tags: ["security","ai","api","llm"] published: true paywall: false --- An AI feature does not need to expose weights to expose the value of a model. If an attacker can submit inputs and collect outputs, the interface itself can become a source of training data. Facebook's Automatic Alt Text is a simple example. It generates descriptions such as "two people in a restaurant" or "a dog in a park" to make images accessible to blind and visually impaired users. The feature was not designed as a commercial image-captioning API, but its inputs and outputs have the same basic shape as one. A determined user could upload a large image collection, collect the descriptions, and use the service as a free labeling pipeline. The underlying model remains private, yet part of its capability has effectively been made available for another purpose. ## An endpoint can become a dataset Consider a company that spends months building a classifier for one narrow task: detecting whether a user is trying to move a conversation away from a marketplace. The production endpoint accepts text and returns a decision. An attacker can generate varied messages, submit them to the endpoint, and store the labels. After enough queries, those input-output pairs become a dataset for training a substitute model. The substitute does not need identical weights or architecture. It only needs to reproduce the original system's decisions well enough for the attacker's use case. This is usually called model extraction, model stealing, or functional imitation. The exact term matters less than the cost asymmetry. The owner pays for data collection, experimentation, evaluation, and failed iterations. The attacker pays mainly for queries and a smaller training run. The risk depends strongly on the API. A binary classifier exposes less information per request than an endpoint returning calibrated probabilities, embeddings, explanations, or full token distributions. Rich outputs can make extraction dramatically more efficient. ## Theft is not the only form of abuse Some public AI interfaces can also be turned into unofficial inference backends. Research demos and comparison platforms may provide expensive models at no direct cost to the user. Automation can redirect those resources into unrelated applications. This does not copy the model. It copies access to its capability and transfers the operating cost to someone else. From the provider's perspective, the result may be just as damaging: increased infrastructure spend, degraded service, and a commercial workload hidden inside what appears to be ordinary public use. These two attacks are related but distinct: - **Inference abuse** repeatedly uses the original service instead of paying for an intended API. - **Model extraction** records outputs and trains a replacement that can later run independently. A defensive system should know which one it is trying to stop. Daily quotas may limit free inference while still allowing a slow extraction campaign spread across many accounts. ## Why normal rate limits are not enough Obvious scraping is easy to block. Careful extraction looks more like legitimate use. The attacker can distribute requests over time, vary phrasing, rotate accounts, and focus on inputs that reveal the decision boundary. A narrow classifier may require surprisingly few well-chosen queries. If the service has many normal users, malicious requests can disappear into the background traffic. Simple volume limits therefore address only the cheapest version of the attack. More useful signals include: - repeated exploration of small variations of the same input; - unusually broad coverage of the input space; - many low-value accounts showing coordinated behavior; - systematic probing near a classifier's decision boundary; - traffic that collects outputs but does not behave like the intended product workflow. The last signal is particularly valuable. A real user usually follows a product journey. An extractor is interested in the model, not the product around it. ## Designing a safer public AI feature There is no universal control that makes extraction impossible. If useful information is returned to a client, some of that information can be learned. The practical goal is to raise the cost and reduce the amount of information disclosed. Start by returning only what the feature needs. Do not expose probabilities, embeddings, explanations, internal labels, or alternative predictions unless the product genuinely uses them. Apply quotas at several levels, including account, organization, payment method, network, and device signals. Monitor coordinated patterns instead of treating every identity in isolation. For high-value models, add extraction-focused evaluation to the threat model. Estimate how accurately a substitute can be trained from the outputs available under realistic quotas. This produces a more useful security measure than asking whether the weights themselves can be downloaded. Finally, separate public demonstrations from production APIs. A demo should have a deliberate budget and abuse model. If it exposes the same capability as a paid endpoint with weaker controls, it is not merely a marketing surface. It is an alternative route to the model. ## The broader point A public AI interface exposes more than a feature. It exposes examples of the model's behavior, and behavior is exactly what another model can learn. Not every endpoint can be cloned economically, and a substitute may fail outside the region covered by its queries. Still, teams should treat model outputs as valuable information. Protecting weights while offering unlimited predictions is like protecting source code while publishing an oracle that answers every relevant question about the program.