如何使用向量存储检索数据

¥How use a vector store to retrieve data

Prerequisites

本指南假设你熟悉以下概念：

¥This guide assumes familiarity with the following concepts:

可以使用 .asRetriever() 方法将向量存储转换为检索器，从而更轻松地将它们组合成链式结构。

¥Vector stores can be converted into retrievers using the .asRetriever() method, which allows you to more easily compose them in chains.

下面，我们展示了一个检索增强生成 (RAG) 链，它使用以下步骤对文档执行问答：

¥Below, we show a retrieval-augmented generation (RAG) chain that performs question answering over documents using the following steps:

初始化向量存储
¥Initialize an vector store
从该向量存储创建一个检索器
¥Create a retriever from that vector store
编写问答链
¥Compose a question answering chain
提问！
¥Ask questions!

每个步骤都有多个子步骤和可能的配置，但我们将介绍一个通用流程。首先，安装所需的依赖：

¥Each of the steps has multiple sub steps and potential configurations, but we'll go through one common flow. First, install the required dependency:

tip

请参阅此部分包含有关安装集成包的一般说明。

¥See this section for general instructions on installing integration packages.

npm
Yarn
pnpm

npm install @langchain/openai @langchain/core

yarn add @langchain/openai @langchain/core

pnpm add @langchain/openai @langchain/core

你可以下载 state_of_the_union.txt 文件此处。

¥You can download the state_of_the_union.txt file here.

import * as fs from "node:fs";

import { OpenAIEmbeddings, ChatOpenAI } from "@langchain/openai";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import {
  RunnablePassthrough,
  RunnableSequence,
} from "@langchain/core/runnables";
import { StringOutputParser } from "@langchain/core/output_parsers";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import type { Document } from "@langchain/core/documents";

const formatDocumentsAsString = (documents: Document[]) => {
  return documents.map((document) => document.pageContent).join("\n\n");
};

// Initialize the LLM to use to answer the question.
const model = new ChatOpenAI({
  model: "gpt-4o",
});
const text = fs.readFileSync("state_of_the_union.txt", "utf8");
const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
const docs = await textSplitter.createDocuments([text]);
// Create a vector store from the documents.
const vectorStore = await MemoryVectorStore.fromDocuments(
  docs,
  new OpenAIEmbeddings()
);

// Initialize a retriever wrapper around the vector store
const vectorStoreRetriever = vectorStore.asRetriever();

// Create a system & human prompt for the chat model
const SYSTEM_TEMPLATE = `Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
----------------
{context}`;

const prompt = ChatPromptTemplate.fromMessages([
  ["system", SYSTEM_TEMPLATE],
  ["human", "{question}"],
]);

const chain = RunnableSequence.from([
  {
    context: vectorStoreRetriever.pipe(formatDocumentsAsString),
    question: new RunnablePassthrough(),
  },
  prompt,
  model,
  new StringOutputParser(),
]);

const answer = await chain.invoke(
  "What did the president say about Justice Breyer?"
);

console.log({ answer });

/*
  {
    answer: 'The president honored Justice Stephen Breyer by recognizing his dedication to serving the country as an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. He thanked Justice Breyer for his service.'
  }
*/

API Reference:

OpenAIEmbeddings from @langchain/openai
ChatOpenAI from @langchain/openai
RecursiveCharacterTextSplitter from @langchain/textsplitters
MemoryVectorStore from langchain/vectorstores/memory
RunnablePassthrough from @langchain/core/runnables
RunnableSequence from @langchain/core/runnables
StringOutputParser from @langchain/core/output_parsers
ChatPromptTemplate from @langchain/core/prompts
Document from @langchain/core/documents

让我们逐步了解这里发生了什么。

¥Let's walk through what's happening here.

我们首先加载一段长文本，然后使用文本分割器将其分割成更小的文档。然后，我们将这些文档（也使用传递的 OpenAIEmbeddings 实例嵌入文档）加载到我们的向量存储 HNSWLib 中，并创建索引。
¥We first load a long text and split it into smaller documents using a text splitter. We then load those documents (which also embeds the documents using the passed OpenAIEmbeddings instance) into HNSWLib, our vector store, creating our index.
虽然我们可以直接查询向量存储，但我们将向量存储转换为检索器，以便以正确的格式返回检索到的文档，供问答链使用。
¥Though we can query the vector store directly, we convert the vector store into a retriever to return retrieved documents in the right format for the question answering chain.
我们初始化一个检索链，我们将在后面的步骤 4 中调用它。
¥We initialize a retrieval chain, which we'll call later in step 4.
我们提出问题！
¥We ask questions!

后续步骤

¥Next steps

现在你已经学习了如何将向量存储转换为检索器。

¥You've now learned how to convert a vector store as a retriever.

有关特定检索器、关于 RAG 的更广泛教程的深入介绍，请参阅各个部分；有关如何使用基于任何数据源创建你自己的自定义检索器，请参阅本节。

¥See the individual sections for deeper dives on specific retrievers, the broader tutorial on RAG, or this section to learn how to create your own custom retriever over any data source.

如何使用向量存储检索数据

API Reference:

后续步骤​

后续步骤