Imagine an employee stranded at an airport after a cancelled flight, asking the company’s AI assistant if they can expense a hotel. The company has thousands of documents covering travel, expenses, insurance, employee benefits, and regional policies. Finding the specific policy that addresses this situation is a challenge for an LLM chatbot. The question mentions a “cancelled flight,” while the policy refers to “involuntary travel disruption.” Other documents discuss hotels but apply to different countries, and an older policy may contain outdated information. The LLM must find information that matches the question’s meaning, belongs to the correct policy, and remains valid today.
This is known as the retrieval problem behind many LLM applications. To solve this, LLMs use a process called retrieval-augmented generation (RAG). An embedding model converts the question into a numerical representation, which a search system uses to find promising passages. The application supplies the original text to the LLM, which then uses those passages to explain the policy and cite the exact sources.
For a large collection of documents, retrieval offers a way to select a manageable amount of information. A vector database supports the retrieval part by storing and searching numerical representations of content. This division of responsibilities is crucial because if the application retrieves an outdated policy, even a capable language model will produce an outdated answer. If it retrieves a general hotel-booking rule while missing the cancellation exception, the answer may sound reasonably correct while overlooking the important condition. Reliable answers therefore depend on what happens before generation.
Documents are divided into smaller units called chunks. One chunk might describe hotel reimbursement, while another explains approval requirements. Chunking creates a balance between precision and context. A small chunk may focus tightly on the question but leave out an exception. A large chunk may preserve the exception while including several unrelated policies. For example, a passage saying, “Accommodation expenses are reimbursable following a cancellation,” and the next sentence adding, “This applies only when accommodation is not provided by the airline,” could cause the assistant to give an incorrect answer despite having the relevant text. A useful chunk therefore preserves a complete idea wherever possible. Section headings and limited overlap between neighboring chunks can help retain context.
Embeddings make it possible to search for related meaning even when the wording differs. An embedding model takes a passage and produces a vector: a list of numbers, often containing hundreds or thousands of decimal values. Passages with related meanings occupy nearby positions. The application embeds the question into the same space as the document chunks, allowing it to compare the question’s vector with stored vectors to identify nearby passages. The query and document embeddings must come from compatible encoders. Each searchable record also needs a connection to the original text. The vector helps locate a passage, but the LLM needs the words themselves to interpret the rule. Therefore, a record should connect an embedding with a chunk identifier, the passage text, or its location. Metadata such as document ID, section, effective date, and version are also present.
The search system needs a precise definition of “nearby.” This definition comes from a distance or similarity metric, which compares two vectors and assigns a score. There are different metrics around this: Cosine similarity compares their directions while ignoring their lengths. Euclidean distance measures the straight-line distance between their endpoints, so vector length can influence the result. Dot product reflects both alignment and length. Normalization rescales vectors to length one. When both query and document vectors are normalized, dot product equals cosine similarity. Euclidean distance then produces the same ranking, although its scores differ. Dot product also works with unnormalized vectors when that matches the embedding model’s design. The choice of the metric should follow the embedding model’s intended use. Choosing a metric simply because it is popular can change the ranking in unintended ways. A similarity score should also be interpreted carefully. A score of 0.85 does not mean that a passage has an 85% probability of answering the question correctly. It describes a mathematical relationship between representations.
For more detailed information, you can check out the original article here.
