Short answer
Local AI means running a language model on the company's own server or in an environment it controls, so data does not leave it. It lets you use AI on price lists, technical drawings or customer data without sending them to a service abroad. Under UK GDPR, sending personal data to a provider outside the UK can be a restricted transfer that needs a legal basis and safeguards. Setting up local AI comes down to three jobs: choosing the model, sizing the hardware and managing access.
Key points
- With a local set-up, documents, questions and answers stay inside the company network.
- Under UK GDPR, sending personal data to a provider outside the UK can be a restricted transfer that needs safeguards.
- Open-weight models (such as Llama, Qwen, Mistral and Gemma) can run on your own hardware.
- Not every job needs the largest model: classification and summarising often work with a mid-sized one.
Contents
What is local AI?
Local (on-premise) AI means downloading a language model's weights to the company's own hardware and running it there. Questions, documents and answers do not leave the company network. A set-up usually has three layers:
- Model: a language model whose weights are openly published. Choose models whose licence allows commercial use.
- Serving layer: the software that runs the model on the server. Ollama is an easy start for small teams and trials, while tools such as vLLM are built for multi-user load.
- Document search (RAG): company documents are indexed so the model can find the relevant one when answering and show it as the source.
How does it differ from cloud models?
| Criterion | Cloud API | Local model |
|---|---|---|
| Where data goes | The provider's servers, often abroad | The company network |
| Time to start | Minutes | A few weeks with hardware and set-up |
| Model strength | The strongest models are usually in the cloud | Enough for many jobs, can trail on the hardest ones |
| Cost structure | Pay per use | Hardware investment and maintenance |
| Dependence | Exposed to the provider's price and policy changes | You control the model and version |
| Internet | Required | Can run on an isolated network |
The two can be combined. Many companies process sensitive documents with a local model and use a cloud model for general work that contains no personal data. Which data may go where should be set out in a written rule beforehand.
What does UK GDPR mean for AI services?
The Information Commissioner's Office (ICO) treats sending personal data to a recipient outside the UK as a restricted transfer. It is allowed when it is covered by UK adequacy regulations, by appropriate safeguards such as the International Data Transfer Agreement, or by one of the limited exceptions. The ICO updated its international transfers guidance in January 2026.
- If prompts or documents sent to an AI provider contain personal data, data protection law applies to that processing.
- Each purpose (answering, summarising, training) needs a lawful basis and must be explained to the people concerned.
- Where the provider processes data outside the UK, the transfer rules apply on top of that.
A local model removes the transfer question for the data it handles, because nothing leaves your network. The rest of data protection law, such as lawful basis, transparency and security, still applies.
What hardware do you need?
Three things decide the hardware: the size of the model, how many people use it at once and how fast answers need to come back.
- GPU memory: where the model has to fit. As a rough rule, at 4-bit quantisation each billion parameters needs about half a gigabyte of memory, plus working headroom.
- Model size: models with 7 to 14 billion parameters are often enough for classification, summarising and document search. Larger models do better on long, complex texts but multiply the hardware needed.
- Concurrent use: a one-person trial and a twenty-person team do not fit on the same server. Memory and compute grow with the number of users.
- CPU-only: small models run without a GPU, but slowly. That can suit batch jobs that run overnight.
The most reliable way to find the right size is to try two or three models on your real documents and measure the quality. Benchmarks on paper do not show how a model performs on your work.
How do you set it up, step by step?
Write down the use case and data class
Which job, with which documents, by whom? Do the documents contain personal data, trade secrets or only general information?
Choose the model with real documents
Try two or three models with suitable licences on your own examples and measure accuracy and speed.
Size the hardware
Choose the server based on the model, the number of users and the response time you expect.
Install the serving layer and lock down access
Make the model server reachable only from the internal network and add authentication.
Build document search with permissions
The model should only find documents the user is already allowed to see. Accounting files should not come back as answers to the sales team.
Plan logging, monitoring and updates
Record who asked what and which document was used. Update models and software in a test environment first.
A hybrid set-up: documents inside, computation by choice
Running everything inside is not always necessary. A practical middle path separates two planes. Documents, the index and the audit log stay in the building in every case. Only the model's computation request goes out, in cases the company has approved in advance and to a provider it has chosen. That outgoing request can count as an international transfer and should be covered by contract and recorded.
We describe this set-up in detail on our on-premise page. How agents work in such an environment is in our AI agent guide.
Frequently asked questions
Is a local model as good as ChatGPT?
The strongest cloud models usually lead on hard tasks. For jobs such as classifying documents, summarising and searching company files, open models often give good enough results. A trial with your own documents should make the decision.
What is Ollama?
Ollama is a tool that downloads open-weight language models with a single command and runs them on a computer or server. It is an easy start for trials and small teams. For multi-user load, server-focused tools such as vLLM are preferred.
Does it work without the internet?
Yes. As long as the model and the documents are on the same network, no internet connection is needed. Where security is critical, an air-gapped server with no outside connection can be set up.
Does a local model make us GDPR compliant?
It removes the international transfer question for the data it processes. Lawful basis, transparency, retention and security still have to be handled like any other processing.
Sources
- A guide to international transfersInformation Commissioner's Office
- The ICO's 2026 updated international transfer guidance: decoding the new UK regimeKennedys
- OllamaOllama
- vLLM documentationvLLM
Ask your assistant
Open the article in an AI assistant to apply it to your own situation.
About this article
Entrida Technology & Consulting builds AI, automation and software for SMEs from Istanbul. The facts in this article rest on the sources listed at the end. If you spot an error, write to info@entrida.com.tr and we will correct it and move the updated date forward.
This article in other languages