Skip to main content

WebLLM

Compatibility

仅在 Web 环境中可用。

¥Only available in web environments.

你可以使用 LangChain 的 WebLLM 集成直接在 Web 浏览器中运行 LLM。

¥You can run LLMs directly in your web browser using LangChain's WebLLM integration.

设置​

¥Setup

你需要安装 WebLLM SDK 模块以便与本地模型通信。

¥You'll need to install the WebLLM SDK module to communicate with your local model.

tip

请参阅 此部分包含有关安装集成包的一般说明。

¥See this section for general instructions on installing integration packages.

npm install -S @mlc-ai/web-llm @langchain/community @langchain/core

用法​

¥Usage

请注意，首次调用模型时，WebLLM 将下载该模型的完整权重。这可能高达数 GB，并且可能无法满足你应用的所有终端用户的使用需求，具体取决于他们的网络连接和计算机配置。虽然浏览器会缓存该模型的未来调用，但我们建议使用尽可能小的模型。

¥Note that the first time a model is called, WebLLM will download the full weights for that model. This can be multiple gigabytes, and may not be possible for all end-users of your application depending on their internet connection and computer specs. While the browser will cache future invocations of that model, we recommend using the smallest possible model you can.

我们还建议在调用和加载模型时使用 独立的 Web Worker，以免阻塞执行。

¥We also recommend using a separate web worker when invoking and loading your models to not block execution.

// Must be run in a web environment, e.g. a web worker

import { ChatWebLLM } from "@langchain/community/chat_models/webllm";
import { HumanMessage } from "@langchain/core/messages";

// Initialize the ChatWebLLM model with the model record and chat options.
// Note that if the appConfig field is set, the list of model records
// must include the selected model record for the engine.

// You can import a list of models available by default here:
// https://github.com/mlc-ai/web-llm/blob/main/src/config.ts
//
// Or by importing it via:
// import { prebuiltAppConfig } from "@mlc-ai/web-llm";
const model = new ChatWebLLM({
  model: "Phi-3-mini-4k-instruct-q4f16_1-MLC",
  chatOptions: {
    temperature: 0.5,
  },
});

await model.initialize((progress: Record<string, unknown>) => {
  console.log(progress);
});

// Call the model with a message and await the response.
const response = await model.invoke([
  new HumanMessage({ content: "What is 1 + 1?" }),
]);

console.log(response);

/*
AIMessage {
  content: ' 2\n',
}
*/

API Reference:

也支持流媒体。

¥Streaming is also supported.

示例​

¥Example

要查看完整的端到端示例，请查看 此项目。

¥For a full end-to-end example, check out this project.

¥Related