What you'll end up with

By the end of this guide you'll have a private AI assistant that:

  • Runs 100% on your Windows machine — no internet connection required after setup
  • Can read and answer questions about your PDFs, Word documents, and spreadsheets
  • Looks like a normal chat interface in your browser
  • Costs nothing to run after the one-time setup

Minimum requirements

You don't need a powerful computer for basic document Q&A. Check the following:

  • Windows 10 or Windows 11 (either works)
  • 16 GB RAM — open Task Manager, click the Performance tab, and look at Memory. If you have less than 16 GB, the setup will still work but will be slower.
  • 30 GB free disk space — check in File Explorer under This PC
  • Internet connection for setup — you'll download software and one AI model. After that, everything runs offline.

That's it. No special graphics card required.

The two tools you'll install

AnythingLLM is your main application. It gives you a clean, familiar chat interface — similar to ChatGPT — but everything runs privately on your machine. It handles document uploads, organizes your files into workspaces, and lets your team ask questions across hundreds of documents.

Ollama runs quietly in the background. Think of it as the engine that powers the AI. AnythingLLM uses it to do the actual thinking — you won't interact with it directly after setup.

Step 1 — Install Ollama

  1. Go to ollama.com and click Download for Windows.
  2. Run the installer (OllamaSetup.exe). Click through the prompts — no custom settings needed.
  3. Once installed, Ollama runs silently in your system tray (bottom-right of your taskbar). You won't see a window open.

Verify it worked: Open your browser and go to http://localhost:11434. You should see the text Ollama is running. If you see that, you're good.

Step 2 — Download an AI model

Ollama needs an AI model to function. For confidential document work on a standard business PC, Mistral 7B is the best balance of quality and speed.

  1. Click the Start menu and search for Command Prompt. Open it.
  2. Type the following and press Enter: 
    ollama pull mistral
  3. Wait for the download to complete — it's about 4 GB, so this may take a few minutes on a standard connection.
  4. Once done, type exit and close the Command Prompt. You won't need it again.
Lighter option: If your PC has less than 16 GB RAM, use ollama pull phi4-mini instead — it's smaller and faster on modest hardware, with slightly less reasoning depth.

Step 3 — Install AnythingLLM

  1. Go to anythingllm.com and click Download — Desktop App.
  2. Run the installer and follow the prompts.
  3. Open AnythingLLM from your Start menu or desktop.

On first launch, a setup wizard walks you through configuration. Here's what to choose:

  • LLM Provider: Select Ollama
  • Ollama Base URL: Leave it as http://localhost:11434 (this is already filled in)
  • Model: Select mistral (or whichever model you downloaded in Step 2)
  • Embedding Model: Select any option that says "local" — this handles document indexing and also runs on your machine

Click through the rest of the wizard using the defaults.

Step 4 — Upload your documents

AnythingLLM organizes documents into Workspaces — think of these as separate rooms, one per project or client.

  1. Click New Workspace and name it (e.g., "Client Contracts 2026" or "Finance Reports").
  2. Click the Upload icon inside the workspace.
  3. Drag and drop your files — PDFs, Word documents (.docx), spreadsheets (.xlsx), or plain text files all work.
  4. AnythingLLM processes each file and builds a private index. This happens locally; nothing leaves your machine.

You can create as many workspaces as you need and switch between them at any time.

Step 5 — Start asking questions

Click into any workspace and use the chat box. Try questions like:

  • "What are the payment terms in the uploaded contracts?"
  • "Summarize the key risks from the Q3 financial report."
  • "Which documents mention the NDA expiry date?"
  • "List every action item from the uploaded board minutes."

The AI will answer based only on the documents in that workspace. It won't guess or pull from the internet — everything is grounded in your actual files.

What this setup cannot do

This guide gets you to a working private document assistant quickly. There are things it doesn't cover:

  • Running a more powerful model that requires a graphics card (GPU)
  • Connecting AnythingLLM to your existing business tools via API
  • Setting up automated workflows (e.g., auto-summarize new contracts)
  • Multi-user server deployment for a whole team

If your needs grow beyond document Q&A for a single user or small team, see the companion guide: Advanced LocalAI Deployment on Windows for Enterprise Teams — which covers GPU acceleration, API integration, and autonomous agent workflows.

Keeping your data private

Before you start uploading sensitive documents, verify your setup is fully offline:

  • ☐ Ollama is running — check http://localhost:11434 shows "Ollama is running"
  • ☐ AnythingLLM is set to use Ollama, not any cloud provider
  • ☐ Your Windows Firewall is active (Settings → Windows Security → Firewall)
  • ☐ You have not entered any API keys for OpenAI, Anthropic, or similar services in AnythingLLM settings

With these confirmed, your documents never leave your machine. There is no cloud sync, no telemetry on document contents, and no external inference calls.

Common problems and fixes

AnythingLLM says it can't connect to Ollama

Check that Ollama is running in your system tray. If it's not there, re-launch it from the Start menu. Then go to AnythingLLM Settings → LLM and click Test Connection.

The AI is very slow to respond

This is normal on PCs with less than 16 GB RAM or no graphics card. Try switching to a lighter model: open Command Prompt, run ollama pull phi4-mini, then update the model in AnythingLLM Settings → LLM.

I uploaded a document but the AI doesn't seem to know about it

Make sure you uploaded the file inside the correct workspace — the AI only sees documents in its own workspace. Also check that the file finished processing (you'll see a green checkmark in the uploads panel).

The model gives vague or wrong answers

Rephrase the question to be more specific, and reference the document by name if possible (e.g., "In the Acme contract, what is the termination clause?"). Larger models handle ambiguous questions better — consider upgrading to Mistral 7B if you're on the lighter Phi-4 Mini.