Home>AI Courses>Private AI on Your Own Computer>Lesson 5

Private AI on Your Own Computer

Module 2 — Getting Started

Lesson 5: Chatting with Your Own Documents, Privately

Last updated: 2026-10-09

The most common reason to run AI locally is to ask questions about files you can't upload anywhere: contracts, medical letters, client records, internal reports. Here's how document chat works on your own machine, and where it falls short.

Lesson 5: Chatting with Your Own Documents, Privately

Blackboard

Your files, your machine

12 min
Concept 1

How document chat works

Most document chat uses retrieval-augmented generation (RAG). Your files are split into chunks and turned into numbers that capture meaning (embeddings), stored on your computer. When you ask a question, the app finds the most relevant chunks and hands them to the model with your question. The model answers from those passages.

Concept 2

Tools that do it locally

AnythingLLM pairs with Ollama or LM Studio and organises documents into workspaces. Jan lets you upload files into Projects. Open WebUI offers a browser interface for a local engine. Our Windows guide walks through Ollama plus AnythingLLM step by step, with no graphics card required.

Concept 3

Where it falls short

Retrieval can miss the passage that matters, especially in tables, scanned PDFs and long documents. Questions that need the whole document at once ('what are all the deadlines in this contract?') are harder than finding one fact. Ask the tool to quote and cite the passage, and open it to check anything important.

Concept 4

Keep the whole chain local

Document chat is only private if every part is local: the model, the embedding model, and the storage. Some tools default to a cloud embedding or a cloud model. Check the settings for each, and do the offline test with a document question, not just a chat question.

Pause and Think

Which documents would you never paste into a cloud chatbot, but would love to ask questions about?

Flashcards

Key Terms

Click or press Enter to flip a card. Review these before the practice question.

Additional Resources
Practice Question

Your local document app answers questions well, but you find its embedding setting points to a cloud service. What does that mean?

Does not count for certificate
Choose one answer, use the hint if needed, then check your answer.
Use the hint before checking if you need a nudge.