What hardware you actually need to run AI over your company's documents

The question always arrives in the same shape, at the same moment: somebody has realised you can let an AI read your internal documents without sending them anywhere, everyone likes the idea, and immediately afterwards somebody asks what the hardware costs.

The honest answer is that it depends, but "it depends" does not close a budget. So let us try to be more useful: below are the orders of magnitude, what actually changes as you add resources, and — more importantly — how to measure your real requirement on your own documents instead of trusting a table you found online, this one included.

Three jobs consume resources, and not in the same way

A RAG system does three distinct kinds of work, and conflating them is the most common cause of bad sizing.

Indexing happens once per document. It reads the file, extracts the text, splits it into chunks and computes embeddings. This is what occupies the system while you load your historical archive, and it is the work that benefits most linearly from a GPU: double the throughput, halve the wait. It is also the only one where, if a night's run is slow, nobody notices.

Retrieval happens on every question and is nearly free. Turning a question into a vector and finding the nearest passages takes milliseconds even on large archives, and depends on RAM far more than on the GPU.

Generation happens on every question and is the part users experience as "fast" or "slow". Here the GPU is not a multiplier: it is the difference between an answer that scrolls as you read it and one that makes you watch a cursor.

From which follows the single most useful rule in this article: if you have to choose where the money goes, put it where generation happens. Indexing can wait for the night. A person in front of a screen cannot.

Without a GPU: when it makes sense and when it does not

It makes sense in two cases. The first is evaluation: you want to know whether answer quality on your archive is good enough before buying anything, and for that speed is irrelevant. The second is genuinely occasional use — a few questions a day, from a few people, over a modest archive.

It does not make sense once the system has to enter an office's daily workflow. Not because it fails, but because a tool that answers slowly gets abandoned: people go back to searching by hand, and the project is written off as a failure when the problem was only the hardware.

One operational detail worth knowing: on a machine with a GPU, a broken CUDA setup makes the work fall back to CPU silently. The system keeps working, but indexing goes from seconds to minutes, and you find out weeks later. The Community engine has a setting for exactly this, EMBEDDINGS__REQUIRE_GPU=true, which fails the start rather than degrading quietly. In production, set it.

With a GPU: what to look at

The number that matters is VRAM, not teraflops. The language model has to be loaded into video memory: if it does not fit, it either runs partly on CPU with a collapse in performance, or does not start at all.

A useful order of magnitude: a quantised model in the 7B–14B range — the right range for answering over your own documents — sits comfortably in a 16 GB GPU alongside the embedding model, which takes a share of its own. Below 12 GB you start trading away how much context you can afford. Above 24 GB you have headroom for larger models or for serving more conversations at once.

There is a load-order detail that explains an otherwise baffling problem: the inference engine sizes its own VRAM allocation from how much it finds free at startup. So the embedding model has to be loaded first — otherwise inference reserves memory that then is not there. The Community engine already enforces that order, but if you are assembling a stack yourself it is exactly the kind of thing that will cost you an afternoon.

RAM and disk: less dramatic, not negligible

Sixteen gigabytes of RAM is the workable minimum; thirty-two is comfortable and lets the vector database hold larger indices in memory, which is what keeps retrieval instant as the archive grows.

On disk, the most frequent surprise concerns the first run. The Community engine downloads roughly twelve gigabytes of language model, embedding model, vector database, inference engine and OCR data, verifying each against a declared sha256 before using it. That is not a coffee-break download on a slow line, and it needs planning. Every later start is immediate, because a component is re-fetched only when missing or failing its check.

After that, space grows with the archive: the source documents, the vectors and the application database. These are modest numbers next to the initial twelve gigabytes, but it is worth putting the data directory on a dedicated volume before the first run: moving it afterwards restarts every download from scratch, because nothing exists yet at the new path.

The serious method: measure instead of estimating

Everything above is an order of magnitude. Your real requirement depends on things no table can know: how long your documents are, how many are scans that need OCR, how many people will ask questions in the same hour.

Which is why the engine can time itself:

./i3k-rag-engine --bench /path/to/a/real-document.pdf

It produces a Markdown report with the time spent in every stage — extraction, chunking, embedding, index write, prefill, decode — and charts of where it goes. Reading that report tells you immediately whether you are short of GPU, short of CPU, or whether the bottleneck is OCR and the GPU is beside the point.

There is also a mode that records a real session, --bench-live, reporting on shutdown: useful for understanding behaviour under real use rather than on one well-chosen file.

The practical advice is obvious and almost nobody follows it: benchmark on your own documents, the ugly representative ones. Not the clean ten-page PDF, but the crooked two-hundred-page scan that represents half your archive. That is the real workload.

A sensible buying path

If you have to reach a decision, this order works nearly every time:

1. Install on what you already have, GPU or not. It exists to answer the one question that matters most: are the answers on our documents useful?
2. Run the benchmark on real, representative documents. Now you have your own numbers.
3. Buy the GPU against those numbers, not against a generic table — this one included.
4. Postpone every optimisation until a user complains about something specific.

The mistake we see most often is swapping the first two steps with the third: buying hardware from a quote, and only then finding out whether the use case held up. The software is a free download; the GPU is not. Trying it first only costs time.

In short

• The bottleneck people feel is nearly always generation, so that is where the GPU goes.
• VRAM matters more than raw throughput: 16 GB is a good starting point for the model range suited to this task.
• 16 GB of RAM is the minimum, 32 GB the comfortable point.
• Budget roughly 12 GB of download on the first run, and choose the data volume before you do it.
• In production, fail the start when the GPU is missing rather than discovering it from the timings.
• Above all: measure on your own documents. The benchmark is inside the binary, and it takes ten minutes.