Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 hrz
|
d4343cf691 | ||
|
hrz
|
7a711564f1 | ||
|
hrz
|
1df75f6882 | ||
|
hrz
|
ba5e074bce | ||
|
hrz
|
bb392930b8 | ||
|
hrz
|
cdb4c34c4c | ||
|
hrz
|
36e34994ba |
No files matched your search
@@ -22,7 +22,6 @@
|
||||
<a href="./docs/readme/README_vi.md"><img alt="Tiếng Việt" src="https://img.shields.io/badge/Tiếng Việt-DFE0E5"></a>
|
||||
<a href="./docs/readme/README_de.md"><img alt="Deutsch" src="https://img.shields.io/badge/Deutsch-DFE0E5"></a>
|
||||
<a href="./docs/readme/README_pt_BR.md"><img alt="Português (Brasil)" src="https://img.shields.io/badge/Português (Brasil)-DFE0E5"></a>
|
||||
<a href="./docs/readme/README_es.md"><img alt="Español" src="https://img.shields.io/badge/Español-DFE0E5"></a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">
|
||||
<img alt="GitHub Contributors" src="https://img.shields.io/github/v/release/xinnan-tech/xiaozhi-esp32-server?logo=docker" />
|
||||
</a>
|
||||
@@ -170,8 +169,8 @@ Spearheaded by Professor Siyuan Liu's Team (South China University of Technology
|
||||
#### 🚀 部署方式选择
|
||||
| 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **最简化安装** | 智能对话、单智能体管理 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 本地跑 `FunASR` 较耗内存，全部走 API 时资源占用低 | - |
|
||||
| **全模块安装** | 智能对话、多用户管理、多智能体管理、智控台界面操作 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 本地跑 `FunASR` 较耗内存，另需数据库资源，全部走 API 时资源占用低| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **最简化安装** | 智能对话、单智能体管理 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - |
|
||||
| **全模块安装** | 智能对话、多用户管理、多智能体管理、智控台界面操作 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
常见问题及相关教程，可参考[这个链接](./docs/FAQ.md)
|
||||
|
||||
@@ -213,7 +212,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
| 工具名称 | 位置 | 使用方法 | 功能说明 |
|
||||
|:---:|:---|:---:|:---:|
|
||||
| 音频交互测试工具 | main》digital-human》index.html | 在 `main/digital-human` 执行 `python start.py` 后访问 `http://127.0.0.1:8006/index.html` | 测试音频播放和接收功能，验证Python端音频处理是否正常 |
|
||||
| 模型响应测试工具 | main》xiaozhi-server》performance_tester.py | 执行 `python performance_tester.py` | 测试ASR(语音识别)、LLM(大模型)、VLLM(视觉模型)、TTS(语音合成)等核心模块的响应速度 |
|
||||
| 模型响应测试工具 | main》xiaozhi-server》performance_tester.py | 执行 `python performance_tester.py` | 测试ASR(语音识别)、LLM(大模型)、VLLM(视觉模型)、TTS(语音合成)三个核心模块的响应速度 |
|
||||
|
||||
> 💡 提示：测试模型速度时，只会测试配置了密钥的模型。
|
||||
|
||||
@@ -233,10 +232,10 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
| 知识库 | 支持RAGFlow知识库，让大模型判断需要调度知识库后再回答 |
|
||||
| 工具调用 | 支持客户端IOT协议、客户MCP协议、服务端MCP协议、MCP接入点协议、自定义工具函数 |
|
||||
| 指令下发 | 依托MQTT协议，支持从智控台将MCP指令下发到ESP32设备 |
|
||||
| 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理；界面支持中文简体、中文繁体、英文、德语、越南语、巴西葡萄牙语显示 |
|
||||
| 测试工具 | 提供模型响应性能测试工具和音频交互测试工具 |
|
||||
| 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理；界面支持中文简体、中文繁体、英文显示 |
|
||||
| 测试工具 | 提供性能测试工具、视觉模型测试工具和音频交互测试工具 |
|
||||
| 部署支持 | 支持Docker部署和本地部署，提供完整的配置文件管理 |
|
||||
| 插件系统 | 支持功能插件扩展、自定义插件开发；新增插件文件在下次设备连接时自动加载，改动已加载插件的代码需重启服务 |
|
||||
| 插件系统 | 支持功能插件扩展、自定义插件开发和插件热加载 |
|
||||
|
||||
### 正在开发 🚧
|
||||
|
||||
@@ -282,8 +281,8 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
|
||||
| 使用方式 | 支持平台 | 免费平台 |
|
||||
|:---:|:---:|:---:|
|
||||
| 接口调用 | EdgeTTS、科大讯飞、火山引擎、腾讯云、阿里云及百炼、CosyVoiceSiliconflow、TTS302AI、CozeCnTTS、CustomTTS、OpenAITTS、MinimaxTTS | EdgeTTS、CosyVoiceSiliconflow(部分) |
|
||||
| 本地服务 | FishSpeech、GPT_SOVITS_V2、GPT_SOVITS_V3、IndexStreamTTS、PaddleSpeechTTS | IndexStreamTTS、PaddleSpeechTTS、FishSpeech、GPT_SOVITS_V2、GPT_SOVITS_V3 |
|
||||
| 接口调用 | EdgeTTS、科大讯飞、火山引擎、腾讯云、阿里云及百炼、CosyVoiceSiliconflow、TTS302AI、CozeCnTTS、GizwitsTTS、ACGNTTS、OpenAITTS、灵犀流式TTS、MinimaxTTS | 灵犀流式TTS、EdgeTTS、CosyVoiceSiliconflow(部分) |
|
||||
| 本地服务 | FishSpeech、GPT_SOVITS_V2、GPT_SOVITS_V3、Index-TTS、PaddleSpeech | Index-TTS、PaddleSpeech、FishSpeech、GPT_SOVITS_V2、GPT_SOVITS_V3 |
|
||||
|
||||
---
|
||||
|
||||
@@ -299,8 +298,8 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
|
||||
| 使用方式 | 支持平台 | 免费平台 |
|
||||
|:---:|:---:|:---:|
|
||||
| 本地使用 | FunASR、SherpaASR、SherpaParaformerASR、VoskASR | FunASR、SherpaASR、SherpaParaformerASR、VoskASR |
|
||||
| 接口调用 | FunASRServer、火山引擎、科大讯飞、腾讯云、阿里云、百度云、OpenAI ASR、GroqASR、Qwen3ASRFlash | FunASRServer |
|
||||
| 本地使用 | FunASR、SherpaASR | FunASR、SherpaASR |
|
||||
| 接口调用 | FunASRServer、火山引擎、科大讯飞、腾讯云、阿里云、百度云、OpenAI ASR | FunASRServer |
|
||||
|
||||
---
|
||||
|
||||
@@ -308,9 +307,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
|
||||
| 使用方式 | 支持平台 | 免费平台 |
|
||||
|:---:|:---:|:---:|
|
||||
| 接口调用 | 声纹识别服务（voiceprint-api） | - |
|
||||
|
||||
声纹识别需要单独部署 [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) 服务（基于 3D-Speaker），并依赖 MySQL，详见[声纹识别启用指南](./docs/voiceprint-integration.md)
|
||||
| 本地使用 | 3D-Speaker | 3D-Speaker |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -28,22 +28,16 @@ wget https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/sherpa-o
|
||||
tar xvf sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01.tar.bz2
|
||||
|
||||
# 方法2：使用ModelScope
|
||||
# 这份仓库里只有 epoch-12 的模型，文件名里的 avg 是 avg-2，不是 avg-1
|
||||
# 照抄下面「配置模型文件」的命令时，把 epoch-99-avg-1 换成 epoch-12-avg-2（avg 一起换，只换 epoch 数字会找不到文件）
|
||||
# int8 只决定你选方案一还是方案二，不是方法2 的前提：这份仓库里 epoch-12 的 fp32 三件套齐全，
|
||||
# 用方法2 + 方案一（精度优先）照样能配出四个必需文件
|
||||
pip install modelscope
|
||||
python -c "
|
||||
from modelscope import snapshot_download
|
||||
snapshot_download('pkufool/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01', local_dir='./sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01')
|
||||
snapshot_download('pkufool/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01', cache_dir='./models')
|
||||
"
|
||||
```
|
||||
|
||||
运行时只认 `models/` 目录下的四个固定文件名（`encoder.onnx`、`decoder.onnx`、`joiner.onnx`、`tokens.txt`），也不搜子目录（判定见 `wakeword_runtime/core/detector_assets.py`）。文件留在下载目录里不做拷贝改名，启动会报 `No valid model directory found`。两种方法下载完都要接着做下面「配置模型文件」那节的拷贝改名。
|
||||
|
||||
#### 2. 配置模型文件
|
||||
|
||||
模型包下载后包含以下文件（epoch-99 那几行只有官方包有，ModelScope 那份只有 epoch-12）：
|
||||
模型包下载后包含以下文件：
|
||||
|
||||
```
|
||||
sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/
|
||||
@@ -51,9 +45,7 @@ sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/
|
||||
├── encoder-epoch-12-avg-2-chunk-16-left-64.onnx
|
||||
├── encoder-epoch-99-avg-1-chunk-16-left-64.int8.onnx # 速度优先
|
||||
├── encoder-epoch-99-avg-1-chunk-16-left-64.onnx # 精度优先
|
||||
├── decoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx # 速度优先
|
||||
├── decoder-epoch-12-avg-2-chunk-16-left-64.onnx
|
||||
├── decoder-epoch-99-avg-1-chunk-16-left-64.int8.onnx # 速度优先
|
||||
├── decoder-epoch-99-avg-1-chunk-16-left-64.onnx # 精度优先
|
||||
├── joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx # 速度优先
|
||||
├── joiner-epoch-12-avg-2-chunk-16-left-64.onnx
|
||||
@@ -95,8 +87,7 @@ cd sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01
|
||||
# 创建模型目录
|
||||
mkdir -p ../models
|
||||
|
||||
# 复制速度优先的epoch-99 三件套：encoder 和 joiner 用 int8，decoder 用 fp32
|
||||
# 官方包里 decoder 也有 int8 版，但量化对 decoder 收益不大，一般仍用 fp32
|
||||
# 复制速度优先的epoch-99 int8三件套
|
||||
cp encoder-epoch-99-avg-1-chunk-16-left-64.int8.onnx ../models/encoder.onnx
|
||||
cp decoder-epoch-99-avg-1-chunk-16-left-64.onnx ../models/decoder.onnx
|
||||
cp joiner-epoch-99-avg-1-chunk-16-left-64.int8.onnx ../models/joiner.onnx
|
||||
@@ -107,10 +98,9 @@ cp tokens.txt ../models/tokens.txt
|
||||
|
||||
**注意事项**:
|
||||
|
||||
- **精度怎么选**：encoder、joiner、decoder 三件套都有 int8 和 fp32 两版，官方包和 ModelScope 那份都是这样；官方文档的说法是量化对 decoder 收益不大，所以方案二里 encoder 和 joiner 用 int8、decoder 仍用 fp32
|
||||
- **int8 只决定选方案一还是方案二**：运行时只按上面的文件名找模型，不看你拷的是 int8 还是 fp32，所以方法2（ModelScope）配方案一（精度优先）照样能跑
|
||||
- **不要混用 fp32 与 int8**：三个模型文件必须保持一致的精度
|
||||
- **优先选择 epoch-99**：比 epoch-12 训练更充分，精度更高
|
||||
- **必需文件**：`encoder.onnx` + `decoder.onnx` + `joiner.onnx` + `tokens.txt` 四个，`keywords.txt` 不在其中，它不用自己准备，首次启动会用配置里的唤醒词自动生成
|
||||
- **必需文件**：`encoder.onnx` + `decoder.onnx` + `joiner.onnx` + `tokens.txt` + `keywords.txt`
|
||||
|
||||
### 最终模型文件结构
|
||||
|
||||
@@ -121,7 +111,7 @@ wakeword_runtime/models/
|
||||
├── encoder.onnx # 编码器模型（重命名后）
|
||||
├── decoder.onnx # 解码器模型（重命名后）
|
||||
├── joiner.onnx # 连接器模型（重命名后）
|
||||
├── tokens.txt # 拼音 Token 映射表（227行版本）
|
||||
├── tokens.txt # 拼音 Token 映射表（228行版本）
|
||||
├── keywords.txt # 关键词配置文件（首次启动自动生成）
|
||||
└── keywords_raw.txt # 可选，runtime 不依赖
|
||||
```
|
||||
@@ -204,14 +194,12 @@ python start.py
|
||||
|
||||
### 首次使用
|
||||
|
||||
1. 先按上面「模型下载（必需）」那节把模型包下载解压，再按「配置模型文件」那节把模型放到 `models/` 目录，该目录下要有 `encoder.onnx`、`decoder.onnx`、`joiner.onnx` 和 `tokens.txt` 四个文件
|
||||
2. `models/keywords.txt` 不用自己准备，首次启动会用配置里的唤醒词自动生成
|
||||
1. 准备 `models/` 目录下的模型文件和 `tokens.txt`
|
||||
2. 确认 `models/keywords.txt` 存在
|
||||
3. 在 `digital-human` 目录运行 `python start.py`
|
||||
4. 浏览器打开 `http://127.0.0.1:8006/index.html`
|
||||
5. 进入设置页检查“唤醒词”配置
|
||||
|
||||
模型没放到位就启动，只会打印「警告: 唤醒词服务启动失败」：页面能打开，麦克风的输入流也已经打开，但没有唤醒词检测在消费这些数据，语音唤不醒。
|
||||
|
||||
### 修改唤醒词
|
||||
|
||||
1. 打开数字人页面设置
|
||||
|
||||
@@ -91,8 +91,6 @@ TTS:
|
||||
```py
|
||||
python app.py
|
||||
```
|
||||
启动`main/digital-human`下的`python start.py`前，先按[唤醒词模型说明](./digital-human-wakeword.md)把模型下载并配置到`main/digital-human/wakeword_runtime/models/`，默认配置下缺模型只会打印「警告: 唤醒词服务启动失败」。页面能打开，麦克风的输入流也已经打开，但没有唤醒词检测在消费这些数据，语音唤不醒。
|
||||
|
||||
启动`main/digital-human`下的`python start.py`后，打开`http://127.0.0.1:8006/index.html`，测试连接和发送消息时paddlespeech端是否有输出日志
|
||||
|
||||
输出日志参考：
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# 语音识别、流式语音识别、大语言模型、非流式语音合成、流式语音合成、视觉模型的性能测试工具使用指南
|
||||
# 语音识别、大语言模型、非流式语音合成、流式语音合成、视觉模型的性能测试工具使用指南
|
||||
|
||||
1.在main/xiaozhi-server目录下创建data目录
|
||||
2.在data目录下创建.config.yaml文件
|
||||
|
||||
+27
-30
@@ -22,7 +22,6 @@ Unterstützt MQTT+UDP-Protokoll, Websocket-Protokoll, MCP-Endpunkte und Stimmabd
|
||||
<a href="./README_vi.md"><img alt="Tiếng Việt" src="https://img.shields.io/badge/Tiếng Việt-DFE0E5"></a>
|
||||
<a href="./README_de.md"><img alt="Deutsch" src="https://img.shields.io/badge/Deutsch-DBEDFA"></a>
|
||||
<a href="./README_pt_BR.md"><img alt="Português (Brasil)" src="https://img.shields.io/badge/Português (Brasil)-DFE0E5"></a>
|
||||
<a href="./README_es.md"><img alt="Español" src="https://img.shields.io/badge/Español-DFE0E5"></a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">
|
||||
<img alt="GitHub Contributors" src="https://img.shields.io/github/v/release/xinnan-tech/xiaozhi-esp32-server?logo=docker" />
|
||||
</a>
|
||||
@@ -55,31 +54,31 @@ Möchten Sie die Nutzungseffekte sehen? Klicken Sie auf die Videos unten 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
|
||||
<picture>
|
||||
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
|
||||
<img alt="响应速度感受" src="docs/images/demo9.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
|
||||
<picture>
|
||||
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
|
||||
<img alt="速度优化秘诀" src="docs/images/demo6.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
|
||||
<picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="docs/images/demo8.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
|
||||
<picture>
|
||||
<img alt="设备呼叫设备，打电话" src="../images/demo0.png" /></picture>
|
||||
<img alt="设备呼叫设备，打电话" src="docs/images/demo0.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
|
||||
<picture>
|
||||
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
|
||||
<img alt="复杂医疗场景" src="docs/images/demo1.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -87,31 +86,31 @@ Möchten Sie die Nutzungseffekte sehen? Klicken Sie auf die Videos unten 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
|
||||
<picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="docs/images/demo7.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
|
||||
<picture>
|
||||
<img alt="实时打断" src="../images/demo10.png" /></picture>
|
||||
<img alt="实时打断" src="docs/images/demo10.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
|
||||
<picture>
|
||||
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
|
||||
<img alt="拍照识物品" src="docs/images/demo12.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
|
||||
<picture>
|
||||
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
|
||||
<img alt="控制家电开关" src="docs/images/demo5.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
|
||||
<picture>
|
||||
<img alt="多指令任务" src="../images/demo11.png" /></picture>
|
||||
<img alt="多指令任务" src="docs/images/demo11.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -119,31 +118,31 @@ Möchten Sie die Nutzungseffekte sehen? Klicken Sie auf die Videos unten 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
|
||||
<picture>
|
||||
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
|
||||
<img alt="MCP接入点" src="docs/images/demo13.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
|
||||
<picture>
|
||||
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
|
||||
<img alt="MQTT指令下发" src="docs/images/demo4.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
|
||||
<picture>
|
||||
<img alt="声纹识别" src="../images/demo14.png" /></picture>
|
||||
<img alt="声纹识别" src="docs/images/demo14.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
|
||||
<picture>
|
||||
<img alt="自定义音色" src="../images/demo2.png" /></picture>
|
||||
<img alt="自定义音色" src="docs/images/demo2.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
|
||||
<picture>
|
||||
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
|
||||
<img alt="使用粤语交流" src="docs/images/demo3.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -168,8 +167,8 @@ Dieses Projekt bietet zwei Bereitstellungsmethoden. Bitte wählen Sie basierend
|
||||
#### 🚀 Auswahl der Bereitstellungsmethode
|
||||
| Bereitstellungsmethode | Funktionen | Anwendungsszenarien | Deployment-Dokumente | Konfigurationsanforderungen | Video-Tutorials |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Vereinfachte Installation** | Intelligenter Dialog, Einzel-Agenten-Verwaltung | Umgebungen mit geringer Konfiguration, Daten in Konfigurationsdateien gespeichert, keine Datenbank erforderlich | [①Docker-Version](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Quellcode-Deployment](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| Lokales `FunASR` benötigt mehr Arbeitsspeicher; bei durchgängiger API-Nutzung ist der Ressourcenverbrauch gering | - |
|
||||
| **Vollständige Modulinstallation** | Intelligenter Dialog, Mehrbenutzerverwaltung, Mehr-Agenten-Verwaltung, Intelligente Steuerkonsole-Bedienung | Vollständige Funktionserfahrung, Daten in Datenbank gespeichert |[①Docker-Version](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Quellcode-Deployment](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Quellcode-Deployment Auto-Update-Tutorial](../dev-ops-integration.md) | Lokales `FunASR` benötigt mehr Arbeitsspeicher, zusätzlich werden Datenbankressourcen benötigt; bei durchgängiger API-Nutzung ist der Ressourcenverbrauch gering| [Video-Tutorial für lokalen Quellcode-Start](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **Vereinfachte Installation** | Intelligenter Dialog, Einzel-Agenten-Verwaltung | Umgebungen mit geringer Konfiguration, Daten in Konfigurationsdateien gespeichert, keine Datenbank erforderlich | [①Docker-Version](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Quellcode-Deployment](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 Kerne 4GB bei Verwendung von `FunASR`, 2 Kerne 2GB bei allen APIs | - |
|
||||
| **Vollständige Modulinstallation** | Intelligenter Dialog, Mehrbenutzerverwaltung, Mehr-Agenten-Verwaltung, Intelligente Steuerkonsole-Bedienung | Vollständige Funktionserfahrung, Daten in Datenbank gespeichert |[①Docker-Version](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Quellcode-Deployment](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Quellcode-Deployment Auto-Update-Tutorial](../dev-ops-integration.md) | 4 Kerne 8GB bei Verwendung von `FunASR`, 2 Kerne 4GB bei allen APIs| [Video-Tutorial für lokalen Quellcode-Start](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
Häufige Fragen und entsprechende Tutorials finden Sie unter [diesem Link](../FAQ.md)
|
||||
|
||||
@@ -211,7 +210,7 @@ Dieses Projekt bietet die folgenden Testwerkzeuge, um Ihnen bei der Überprüfun
|
||||
| Werkzeugname | Standort | Verwendungsmethode | Funktionsbeschreibung |
|
||||
|:---:|:---|:---:|:---:|
|
||||
| Audio-Interaktionstesttool | main》digital-human》index.html | In `main/digital-human` `python start.py` ausführen und dann `http://127.0.0.1:8006/index.html` öffnen | Testet Audio-Wiedergabe- und Empfangsfunktionen, überprüft, ob die Python-seitige Audioverarbeitung normal ist |
|
||||
| Modell-Reaktionstesttool | main》xiaozhi-server》performance_tester.py | Ausführen `python performance_tester.py` | Testet die Reaktionsgeschwindigkeit von Kernmodulen wie ASR (Spracherkennung), LLM (großes Modell), VLLM (Vision-Modell), TTS (Sprachsynthese) |
|
||||
| Modell-Reaktionstesttool | main》xiaozhi-server》performance_tester.py | Ausführen `python performance_tester.py` | Testet die Reaktionsgeschwindigkeit von ASR (Spracherkennung), LLM (großes Modell), VLLM (Vision-Modell), TTS (Sprachsynthese) drei Kernmodulen |
|
||||
|
||||
> 💡 Hinweis: Beim Testen der Modellgeschwindigkeit werden nur Modelle mit konfigurierten Schlüsseln getestet.
|
||||
|
||||
@@ -231,10 +230,10 @@ Dieses Projekt bietet die folgenden Testwerkzeuge, um Ihnen bei der Überprüfun
|
||||
| Wissensdatenbank | Unterstützt RAGFlow-Wissensdatenbank, ermöglicht großem Modell die Bewertung, ob Wissensdatenbank benötigt wird, bevor geantwortet wird |
|
||||
| Werkzeugaufruf | Unterstützt Client-IOT-Protokoll, Client-MCP-Protokoll, Server-MCP-Protokoll, MCP-Endpunktprotokoll, benutzerdefinierte Werkzeugfunktionen |
|
||||
| Befehlsübermittlung | Basierend auf MQTT-Protokoll, unterstützt die Übermittlung von MCP-Befehlen von der intelligenten Steuerkonsole an ESP32-Geräte |
|
||||
| Verwaltungs-Backend | Bietet Web-Verwaltungsoberfläche, unterstützt Benutzerverwaltung, Systemkonfiguration und Geräteverwaltung; Oberfläche unterstützt vereinfachtes Chinesisch, traditionelles Chinesisch, Englisch, Deutsch, Vietnamesisch und brasilianisches Portugiesisch an |
|
||||
| Testwerkzeuge | Bietet Leistungstestwerkzeuge für Modellantworten und Audio-Interaktionstestwerkzeuge |
|
||||
| Verwaltungs-Backend | Bietet Web-Verwaltungsoberfläche, unterstützt Benutzerverwaltung, Systemkonfiguration und Geräteverwaltung; Oberfläche unterstützt vereinfachtes Chinesisch, traditionelles Chinesisch und englische Anzeige |
|
||||
| Testwerkzeuge | Bietet Leistungstestwerkzeuge, Vision-Modell-Testwerkzeuge und Audio-Interaktionstestwerkzeuge |
|
||||
| Deployment-Unterstützung | Unterstützt Docker-Deployment und lokales Deployment, bietet vollständige Konfigurationsdateiverwaltung |
|
||||
| Plugin-System | Unterstützt funktionale Plugin-Erweiterungen und benutzerdefinierte Plugin-Entwicklung; neu hinzugefügte Plugin-Dateien werden bei der nächsten Geräteverbindung geladen, Änderungen an bereits geladenen Plugins erfordern einen Neustart des Dienstes |
|
||||
| Plugin-System | Unterstützt funktionale Plugin-Erweiterungen, benutzerdefinierte Plugin-Entwicklung und Plugin-Hot-Loading |
|
||||
|
||||
### In Entwicklung 🚧
|
||||
|
||||
@@ -254,7 +253,7 @@ Xiaozhi ist ein Ökosystem. Wenn Sie dieses Produkt verwenden, können Sie sich
|
||||
|
||||
| Verwendungsmethode | Unterstützte Plattformen | Kostenlose Plattformen |
|
||||
|:---:|:---:|:---:|
|
||||
| OpenAI-Schnittstellenaufrufe | Alibaba Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini, iFLYTEK | Zhipu ChatGLM, Gemini |
|
||||
| OpenAI-Schnittstellenaufrufe | Alibaba Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
|
||||
| Ollama-Schnittstellenaufrufe | Ollama | - |
|
||||
| Dify-Schnittstellenaufrufe | Dify | - |
|
||||
| FastGPT-Schnittstellenaufrufe | FastGPT | - |
|
||||
@@ -280,8 +279,8 @@ Tatsächlich kann jedes VLLM, das OpenAI-Schnittstellenaufrufe unterstützt, int
|
||||
|
||||
| Verwendungsmethode | Unterstützte Plattformen | Kostenlose Plattformen |
|
||||
|:---:|:---:|:---:|
|
||||
| Schnittstellenaufrufe | EdgeTTS, iFLYTEK, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, AliYun Stream TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS, Volcano Dual-Stream TTS | EdgeTTS, CosyVoiceSiliconflow (teilweise) |
|
||||
| Lokale Dienste | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, IndexStreamTTS, PaddleSpeechTTS | IndexStreamTTS, PaddleSpeechTTS, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
| Schnittstellenaufrufe | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, AliYun Stream TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi Streaming TTS, MinimaxTTS, Volcano Dual-Stream TTS | Lingxi Streaming TTS, EdgeTTS, CosyVoiceSiliconflow (teilweise) |
|
||||
| Lokale Dienste | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, Index-TTS, PaddleSpeech | Index-TTS, PaddleSpeech, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
|
||||
---
|
||||
|
||||
@@ -297,8 +296,8 @@ Tatsächlich kann jedes VLLM, das OpenAI-Schnittstellenaufrufe unterstützt, int
|
||||
|
||||
| Verwendungsmethode | Unterstützte Plattformen | Kostenlose Plattformen |
|
||||
|:---:|:---:|:---:|
|
||||
| Lokale Verwendung | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
|
||||
| Schnittstellenaufrufe | DoubaoASR, Doubao Streaming ASR, FunASRServer, iFLYTEK, TencentASR, AliyunASR, Aliyun Streaming ASR, Baidu ASR, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
|
||||
| Lokale Verwendung | FunASR, SherpaASR | FunASR, SherpaASR |
|
||||
| Schnittstellenaufrufe | DoubaoASR, Doubao Streaming ASR, FunASRServer, TencentASR, AliyunASR, Aliyun Streaming ASR, Baidu ASR, OpenAI ASR | FunASRServer |
|
||||
|
||||
---
|
||||
|
||||
@@ -306,9 +305,7 @@ Tatsächlich kann jedes VLLM, das OpenAI-Schnittstellenaufrufe unterstützt, int
|
||||
|
||||
| Verwendungsmethode | Unterstützte Plattformen | Kostenlose Plattformen |
|
||||
|:---:|:---:|:---:|
|
||||
| Schnittstellenaufrufe | Voiceprint-Dienst (voiceprint-api) | - |
|
||||
|
||||
Die Stimmabdruckerkennung erfordert einen separat bereitgestellten [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api)-Dienst (auf Basis von 3D-Speaker) sowie MySQL. Siehe [Anleitung zur Stimmabdruckerkennung](../voiceprint-integration.md).
|
||||
| Lokale Verwendung | 3D-Speaker | 3D-Speaker |
|
||||
|
||||
---
|
||||
|
||||
|
||||
+26
-29
@@ -22,7 +22,6 @@ Support for MQTT+UDP protocol, Websocket protocol, MCP access point, voiceprint
|
||||
<a href="./README_vi.md"><img alt="Tiếng Việt" src="https://img.shields.io/badge/Tiếng Việt-DFE0E5"></a>
|
||||
<a href="./README_de.md"><img alt="Deutsch" src="https://img.shields.io/badge/Deutsch-DFE0E5"></a>
|
||||
<a href="./README_pt_BR.md"><img alt="Português (Brasil)" src="https://img.shields.io/badge/Português (Brasil)-DFE0E5"></a>
|
||||
<a href="./README_es.md"><img alt="Español" src="https://img.shields.io/badge/Español-DFE0E5"></a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">
|
||||
<img alt="GitHub Contributors" src="https://img.shields.io/github/v/release/xinnan-tech/xiaozhi-esp32-server?logo=docker" />
|
||||
</a>
|
||||
@@ -55,31 +54,31 @@ Want to see the usage effects? Click the videos below 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
|
||||
<picture>
|
||||
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
|
||||
<img alt="响应速度感受" src="docs/images/demo9.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
|
||||
<picture>
|
||||
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
|
||||
<img alt="速度优化秘诀" src="docs/images/demo6.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
|
||||
<picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="docs/images/demo8.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
|
||||
<picture>
|
||||
<img alt="设备呼叫设备，打电话" src="../images/demo0.png" /></picture>
|
||||
<img alt="设备呼叫设备，打电话" src="docs/images/demo0.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
|
||||
<picture>
|
||||
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
|
||||
<img alt="复杂医疗场景" src="docs/images/demo1.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -87,31 +86,31 @@ Want to see the usage effects? Click the videos below 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
|
||||
<picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="docs/images/demo7.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
|
||||
<picture>
|
||||
<img alt="实时打断" src="../images/demo10.png" /></picture>
|
||||
<img alt="实时打断" src="docs/images/demo10.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
|
||||
<picture>
|
||||
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
|
||||
<img alt="拍照识物品" src="docs/images/demo12.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
|
||||
<picture>
|
||||
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
|
||||
<img alt="控制家电开关" src="docs/images/demo5.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
|
||||
<picture>
|
||||
<img alt="多指令任务" src="../images/demo11.png" /></picture>
|
||||
<img alt="多指令任务" src="docs/images/demo11.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -119,31 +118,31 @@ Want to see the usage effects? Click the videos below 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
|
||||
<picture>
|
||||
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
|
||||
<img alt="MCP接入点" src="docs/images/demo13.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
|
||||
<picture>
|
||||
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
|
||||
<img alt="MQTT指令下发" src="docs/images/demo4.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
|
||||
<picture>
|
||||
<img alt="声纹识别" src="../images/demo14.png" /></picture>
|
||||
<img alt="声纹识别" src="docs/images/demo14.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
|
||||
<picture>
|
||||
<img alt="自定义音色" src="../images/demo2.png" /></picture>
|
||||
<img alt="自定义音色" src="docs/images/demo2.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
|
||||
<picture>
|
||||
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
|
||||
<img alt="使用粤语交流" src="docs/images/demo3.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -168,8 +167,8 @@ This project provides two deployment methods. Please choose based on your specif
|
||||
#### 🚀 Deployment Method Selection
|
||||
| Deployment Method | Features | Applicable Scenarios | Deployment Docs | Configuration Requirements | Video Tutorials |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Simplified Installation** | Intelligent dialogue, single agent management | Low-configuration environments, data stored in config files, no database required | [①Docker Version](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| Local `FunASR` needs more memory; using APIs for everything keeps resource usage low | - |
|
||||
| **Full Module Installation** | Intelligent dialogue, multi-user management, multi-agent management, intelligent console interface operation | Complete functionality experience, data stored in database |[①Docker Version](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](../dev-ops-integration.md) | Local `FunASR` needs more memory plus database resources; using APIs for everything keeps resource usage low| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **Simplified Installation** | Intelligent dialogue, single agent management | Low-configuration environments, data stored in config files, no database required | [①Docker Version](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 cores 4GB if using `FunASR`, 2 cores 2GB if all APIs | - |
|
||||
| **Full Module Installation** | Intelligent dialogue, multi-user management, multi-agent management, intelligent console interface operation | Complete functionality experience, data stored in database |[①Docker Version](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](../dev-ops-integration.md) | 4 cores 8GB if using `FunASR`, 2 cores 4GB if all APIs| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
For frequently asked questions and related tutorials, please refer to [this link](../FAQ.md)
|
||||
|
||||
@@ -211,7 +210,7 @@ This project provides the following testing tools to help you verify the system
|
||||
| Tool Name | Location | Usage Method | Function Description |
|
||||
|:---:|:---|:---:|:---:|
|
||||
| Audio Interaction Test Tool | main》digital-human》index.html | Run `python start.py` in `main/digital-human`, then open `http://127.0.0.1:8006/index.html` | Tests audio playback and reception functions, verifies if Python-side audio processing is normal |
|
||||
| Model Response Test Tool | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Tests response speed of core modules such as ASR(speech recognition), LLM(large model), VLLM(vision model), TTS(speech synthesis) |
|
||||
| Model Response Test Tool | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Tests response speed of three core modules: ASR(speech recognition), LLM(large model), VLLM(vision model), TTS(speech synthesis) |
|
||||
|
||||
> 💡 Note: When testing model speed, only models with configured keys will be tested.
|
||||
|
||||
@@ -231,10 +230,10 @@ This project provides the following testing tools to help you verify the system
|
||||
| Knowledge Base | Supports RAGFlow knowledge base, enabling LLM to judge whether to schedule the knowledge base after receiving the user's question, and then answer the question |
|
||||
| Tool Calling | Supports client IOT protocol, client MCP protocol, server MCP protocol, MCP endpoint protocol, custom tool functions |
|
||||
| Command Delivery | Supports MCP command delivery to ESP32 devices via MQTT protocol from Smart Console |
|
||||
| Management Backend | Provides Web management interface, supports user management, system configuration and device management; Supports Simplified Chinese, Traditional Chinese, English, German, Vietnamese and Brazilian Portuguese display |
|
||||
| Testing Tools | Provides model response performance testing tools and audio interaction testing tools |
|
||||
| Management Backend | Provides Web management interface, supports user management, system configuration and device management; Supports Simplified Chinese, Traditional Chinese and English display |
|
||||
| Testing Tools | Provides performance testing tools, vision model testing tools, and audio interaction testing tools |
|
||||
| Deployment Support | Supports Docker deployment and local deployment, provides complete configuration file management |
|
||||
| Plugin System | Supports functional plugin extensions and custom plugin development; newly added plugin files are loaded on the next device connection, code changes to already loaded plugins require a service restart |
|
||||
| Plugin System | Supports functional plugin extensions, custom plugin development, and plugin hot-loading |
|
||||
|
||||
### Under Development 🚧
|
||||
|
||||
@@ -280,8 +279,8 @@ In fact, any VLLM that supports OpenAI interface calls can be integrated and use
|
||||
|
||||
| Usage Method | Supported Platforms | Free Platforms |
|
||||
|:---:|:---:|:---:|
|
||||
| Interface calls | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud and Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
|
||||
| Local services | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, IndexStreamTTS, PaddleSpeechTTS | IndexStreamTTS, PaddleSpeechTTS, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
| Interface calls | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud and Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi Streaming TTS, MinimaxTTS | Lingxi Streaming TTS, EdgeTTS, CosyVoiceSiliconflow(partial) |
|
||||
| Local services | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, Index-TTS, PaddleSpeech | Index-TTS, PaddleSpeech, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
|
||||
---
|
||||
|
||||
@@ -297,8 +296,8 @@ In fact, any VLLM that supports OpenAI interface calls can be integrated and use
|
||||
|
||||
| Usage Method | Supported Platforms | Free Platforms |
|
||||
|:---:|:---:|:---:|
|
||||
| Local use | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
|
||||
| Interface calls | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
|
||||
| Local use | FunASR, SherpaASR | FunASR, SherpaASR |
|
||||
| Interface calls | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR | FunASRServer |
|
||||
|
||||
---
|
||||
|
||||
@@ -306,9 +305,7 @@ In fact, any VLLM that supports OpenAI interface calls can be integrated and use
|
||||
|
||||
| Usage Method | Supported Platforms | Free Platforms |
|
||||
|:---:|:---:|:---:|
|
||||
| Interface calls | Voiceprint service (voiceprint-api) | - |
|
||||
|
||||
Voiceprint recognition requires a separately deployed [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) service (based on 3D-Speaker) and MySQL. See the [voiceprint integration guide](../voiceprint-integration.md).
|
||||
| Local use | 3D-Speaker | 3D-Speaker |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,361 +0,0 @@
|
||||
[](https://github.com/xinnan-tech/xiaozhi-esp32-server)
|
||||
|
||||
<h1 align="center">Servicio backend Xiaozhi xiaozhi-esp32-server</h1>
|
||||
|
||||
<p align="center">
|
||||
Este proyecto desarrolla sistemas de hardware y software para terminales inteligentes a partir de la teoría y la tecnología de la inteligencia simbiótica humano-máquina<br/>y ofrece el servicio backend del proyecto de hardware inteligente de código abierto
|
||||
<a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
|
||||
Implementado en Python, Java y Vue según el <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">protocolo de comunicación Xiaozhi</a><br/>
|
||||
Compatible con el protocolo MQTT+UDP, el protocolo WebSocket, puntos de acceso MCP, reconocimiento de huella de voz y bases de conocimiento
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../FAQ.md">Preguntas frecuentes</a>
|
||||
· <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Informar de problemas</a>
|
||||
· <a href="../../README.md#%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3">Documentación de despliegue</a>
|
||||
· <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Notas de la versión</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-DFE0E5"></a>
|
||||
<a href="./README_en.md"><img alt="README in English" src="https://img.shields.io/badge/English-DFE0E5"></a>
|
||||
<a href="./README_vi.md"><img alt="Tiếng Việt" src="https://img.shields.io/badge/Tiếng Việt-DFE0E5"></a>
|
||||
<a href="./README_de.md"><img alt="Deutsch" src="https://img.shields.io/badge/Deutsch-DFE0E5"></a>
|
||||
<a href="./README_pt_BR.md"><img alt="Português (Brasil)" src="https://img.shields.io/badge/Português (Brasil)-DFE0E5"></a>
|
||||
<a href="./README_es.md"><img alt="Español" src="https://img.shields.io/badge/Español-DBEDFA"></a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">
|
||||
<img alt="GitHub Contributors" src="https://img.shields.io/github/v/release/xinnan-tech/xiaozhi-esp32-server?logo=docker" />
|
||||
</a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/LICENSE">
|
||||
<img alt="GitHub pull requests" src="https://img.shields.io/badge/license-MIT-white?labelColor=black" />
|
||||
</a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server">
|
||||
<img alt="stars" src="https://img.shields.io/github/stars/xinnan-tech/xiaozhi-esp32-server?color=ffcb47&labelColor=black" />
|
||||
</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
Liderado por el equipo del profesor Siyuan Liu (Universidad Tecnológica del Sur de China)
|
||||
</br>
|
||||
刘思源教授团队主导研发（华南理工大学）
|
||||
</br>
|
||||
<img src="../images/hnlg.jpg" alt="Universidad Tecnológica del Sur de China (华南理工大学)" width="50%">
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
## A quién va dirigido 👥
|
||||
|
||||
Este proyecto debe usarse junto con hardware ESP32. Si ya ha comprado hardware relacionado con ESP32, se ha conectado con éxito al servicio backend desplegado por Brother Xia (虾哥) y quiere montar por su cuenta su propio servicio backend `xiaozhi-esp32`, este proyecto encaja con lo que busca.
|
||||
|
||||
¿Quiere ver cómo se usa? Pulse los vídeos 🎥
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
|
||||
<picture>
|
||||
<img alt="Sensación de la velocidad de respuesta" src="../images/demo9.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
|
||||
<picture>
|
||||
<img alt="El secreto para optimizar la velocidad" src="../images/demo6.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
|
||||
<picture>
|
||||
<img alt="Humano digital Xiaozhi con activación por voz" src="../images/demo8.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
|
||||
<picture>
|
||||
<img alt="Llamadas de dispositivo a dispositivo" src="../images/demo0.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
|
||||
<picture>
|
||||
<img alt="Escenario médico complejo" src="../images/demo1.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
|
||||
<picture>
|
||||
<img alt="Reproducir música, consultar el tiempo y leer noticias" src="../images/demo7.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
|
||||
<picture>
|
||||
<img alt="Interrupción en tiempo real" src="../images/demo10.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
|
||||
<picture>
|
||||
<img alt="Reconocer objetos con una foto" src="../images/demo12.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
|
||||
<picture>
|
||||
<img alt="Controlar el encendido de electrodomésticos" src="../images/demo5.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
|
||||
<picture>
|
||||
<img alt="Tareas con varias instrucciones" src="../images/demo11.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
|
||||
<picture>
|
||||
<img alt="Punto de acceso MCP" src="../images/demo13.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
|
||||
<picture>
|
||||
<img alt="Envío de instrucciones MQTT" src="../images/demo4.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
|
||||
<picture>
|
||||
<img alt="Reconocimiento de huella de voz" src="../images/demo14.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
|
||||
<picture>
|
||||
<img alt="Tono de voz personalizado" src="../images/demo2.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
|
||||
<picture>
|
||||
<img alt="Conversación en cantonés" src="../images/demo3.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
## Advertencias ⚠️
|
||||
|
||||
1. Este proyecto es software de código abierto. Este software no mantiene relación comercial alguna con los proveedores de API de terceros a los que se conecta (incluidos, entre otros, plataformas de reconocimiento de voz, modelos de lenguaje y síntesis de voz), y no ofrece ninguna garantía sobre la calidad del servicio de dichos proveedores ni sobre la seguridad de sus fondos. Se recomienda elegir de forma preferente proveedores que cuenten con las licencias de actividad correspondientes y leer con atención sus condiciones de servicio y su política de privacidad. Este software no custodia ninguna clave de cuenta, no interviene en el movimiento de fondos y no asume el riesgo de pérdida del saldo recargado.
|
||||
|
||||
2. Las funciones de este proyecto no están completas y no ha superado una evaluación de seguridad de red. No lo utilice en entornos de producción. Si despliega este proyecto en una red pública con fines de aprendizaje, aplique las protecciones necesarias.
|
||||
|
||||
---
|
||||
|
||||
## Documentación de despliegue
|
||||
|
||||
|
||||
|
||||
Este proyecto ofrece dos formas de despliegue. Elija según lo que necesite:
|
||||
|
||||
#### 🚀 Elección del método de despliegue
|
||||
| Método de despliegue | Características | Escenarios | Documentación de despliegue | Requisitos | Tutoriales en vídeo |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Instalación simplificada** | Diálogo inteligente, gestión de un solo agente | Entornos con pocos recursos; los datos se guardan en archivos de configuración y no hace falta base de datos | [①Versión Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Despliegue desde el código fuente](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 núcleos y 4 GB si usa `FunASR`; 2 núcleos y 2 GB si todo va por API | - |
|
||||
| **Instalación de todos los módulos** | Diálogo inteligente, gestión multiusuario, gestión de varios agentes, uso de la interfaz de la consola inteligente | Experiencia con todas las funciones; los datos se guardan en una base de datos |[①Versión Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Despliegue desde el código fuente](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Tutorial de actualización automática desde el código fuente](../dev-ops-integration.md) | 4 núcleos y 8 GB si usa `FunASR`; 2 núcleos y 4 GB si todo va por API | [Tutorial en vídeo para arrancar el código fuente en local](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
Las preguntas frecuentes y los tutoriales relacionados están en [este enlace](../FAQ.md)
|
||||
|
||||
> 💡 Nota: a continuación hay una plataforma de prueba desplegada con el código más reciente. Si lo necesita, puede grabar el firmware y probarla. La concurrencia es de 6 usuarios y los datos se borran cada día.
|
||||
|
||||
```
|
||||
Dirección de la consola inteligente: https://2662r3426b.vicp.fun
|
||||
Consola inteligente (versión H5): https://2662r3426b.vicp.fun/h5/index.html
|
||||
|
||||
Herramienta de prueba del servicio: https://2662r3426b.vicp.fun/test/
|
||||
Dirección de la interfaz OTA: https://2662r3426b.vicp.fun/xiaozhi/ota/
|
||||
Dirección de la interfaz WebSocket: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
```
|
||||
|
||||
#### 🚩 Descripción y recomendaciones de configuración
|
||||
> [!Note]
|
||||
> Este proyecto ofrece dos esquemas de configuración:
|
||||
>
|
||||
> 1. `Configuración básica y totalmente gratuita`: adecuada para uso personal y doméstico. Todos los componentes usan opciones gratuitas y no hay que pagar nada más.
|
||||
>
|
||||
> 2. `Configuración en streaming`: adecuada para demostraciones, formación y escenarios con más de 2 usuarios concurrentes. Usa procesamiento en streaming, responde antes y se siente mejor.
|
||||
>
|
||||
> Desde la versión `0.5.2`, el proyecto admite la configuración en streaming. Frente a las versiones anteriores, el tiempo de respuesta mejora unos `2,5 segundos` y la experiencia de uso mejora de forma notable.
|
||||
|
||||
| Módulo | Configuración básica y totalmente gratuita | Configuración en streaming |
|
||||
|:---:|:---:|:---:|
|
||||
| ASR (reconocimiento de voz) | FunASR (local) | 👍XunfeiStreamASR (iFLYTEK en streaming) |
|
||||
| LLM (modelo de lenguaje) | glm-4-flash (Zhipu) | 👍qwen-flash (Alibaba Bailian) |
|
||||
| VLLM (modelo de visión) | glm-4v-flash (Zhipu) | 👍qwen3.5-flash (Alibaba Bailian) |
|
||||
| TTS (síntesis de voz) | EdgeTTS (Microsoft) | 👍HuoshanDoubleStreamTTS (Volcano en streaming) |
|
||||
| Intent (reconocimiento de intención) | function_call (llamada a funciones) | function_call (llamada a funciones) |
|
||||
| Memory (memoria) | mem_local_short (memoria local a corto plazo) | mem_local_short (memoria local a corto plazo) |
|
||||
|
||||
Si le importa el tiempo que tarda cada componente, consulte el [informe de rendimiento de los componentes de Xiaozhi](https://github.com/xinnan-tech/xiaozhi-performance-research) y pruebe en su propio entorno con el método que describe el informe.
|
||||
|
||||
#### 🔧 Herramientas de prueba
|
||||
Este proyecto incluye las siguientes herramientas para comprobar el sistema y elegir un modelo adecuado:
|
||||
|
||||
| Herramienta | Ubicación | Uso | Qué hace |
|
||||
|:---:|:---|:---:|:---:|
|
||||
| Prueba de interacción de audio | main》digital-human》index.html | En `main/digital-human`, ejecute `python start.py` y abra `http://127.0.0.1:8006/index.html` | Prueba la reproducción y la recepción de audio, y comprueba que el procesamiento de audio en Python funciona |
|
||||
| Prueba de respuesta de los modelos | main》xiaozhi-server》performance_tester.py | Ejecute `python performance_tester.py` | Mide la velocidad de respuesta de los cuatro módulos principales: ASR (reconocimiento de voz), LLM (modelo de lenguaje), VLLM (modelo de visión) y TTS (síntesis de voz) |
|
||||
|
||||
> 💡 Nota: al medir la velocidad de los modelos, solo se prueban los modelos que tienen una clave configurada.
|
||||
|
||||
---
|
||||
## Lista de funciones ✨
|
||||
### Implementado ✅
|
||||
|
||||
| Módulo | Descripción |
|
||||
|:---:|:---|
|
||||
| Arquitectura principal | Basada en la [pasarela MQTT+UDP](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mqtt-gateway-integration.md) y en servidores WebSocket y HTTP; ofrece un sistema completo de gestión de consola y de autenticación |
|
||||
| Interacción por voz | Admite ASR en streaming (reconocimiento de voz), TTS en streaming (síntesis de voz) y VAD (detección de actividad de voz), además de reconocimiento multilingüe y procesamiento de voz |
|
||||
| Huella de voz | Admite el registro, la gestión y el reconocimiento de la huella de voz de varios usuarios; se procesa en paralelo con el ASR, identifica en tiempo real quién habla y se lo pasa al LLM para una respuesta personalizada |
|
||||
| Diálogo inteligente | Admite varios LLM (modelos de lenguaje) para mantener un diálogo inteligente |
|
||||
| Percepción visual | Admite varios VLLM (modelos de visión) para una interacción multimodal |
|
||||
| Reconocimiento de intención | Admite el reconocimiento de intención con el modelo de lenguaje (LLM) y la llamada autónoma a funciones del modelo, y ofrece un mecanismo de procesamiento de intenciones mediante plugins |
|
||||
| Sistema de memoria | Admite memoria local a corto plazo, memoria por la interfaz mem0ai y memoria inteligente PowerMem, con resumen de la memoria |
|
||||
| Base de conocimiento | Admite la base de conocimiento RAGFlow, para que el modelo decida si debe consultarla antes de responder |
|
||||
| Llamada a herramientas | Admite el protocolo IOT del cliente, el protocolo MCP del cliente, el protocolo MCP del servidor, el protocolo de punto de acceso MCP y funciones de herramientas personalizadas |
|
||||
| Envío de instrucciones | Apoyado en el protocolo MQTT, permite enviar instrucciones MCP desde la consola inteligente a los dispositivos ESP32 |
|
||||
| Backend de administración | Ofrece una interfaz web de administración, con gestión de usuarios, configuración del sistema y gestión de dispositivos. La interfaz se muestra en chino simplificado, chino tradicional e inglés |
|
||||
| Herramientas de prueba | Incluye herramientas de prueba de rendimiento, de modelos de visión y de interacción de audio |
|
||||
| Soporte de despliegue | Admite el despliegue con Docker y el despliegue local, y ofrece una gestión completa de los archivos de configuración |
|
||||
| Sistema de plugins | Admite extensiones de plugins funcionales, el desarrollo de plugins propios y la carga en caliente de plugins |
|
||||
|
||||
### En desarrollo 🚧
|
||||
|
||||
Para ver el avance concreto del plan de desarrollo, [pulse aquí](https://github.com/users/xinnan-tech/projects/3). Las preguntas frecuentes y los tutoriales relacionados están en [este enlace](../FAQ.md)
|
||||
|
||||
Si desarrolla software, aquí tiene una [carta abierta a quienes desarrollan](../contributor_open_letter.md). La invitación está hecha.
|
||||
|
||||
---
|
||||
|
||||
## Ecosistema del producto 👬
|
||||
Xiaozhi es un ecosistema. Al usar este producto, también puede ver otros [proyectos destacados](https://github.com/78/xiaozhi-esp32/blob/main/README_zh.md#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) de este ecosistema
|
||||
|
||||
---
|
||||
|
||||
## Plataformas y componentes compatibles 📋
|
||||
### LLM, modelos de lenguaje
|
||||
|
||||
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Llamadas por la interfaz OpenAI | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK | Zhipu, Gemini |
|
||||
| Llamadas por la interfaz Ollama | Ollama | - |
|
||||
| Llamadas por la interfaz Dify | Dify | - |
|
||||
| Llamadas por la interfaz FastGPT | FastGPT | - |
|
||||
| Llamadas por la interfaz Coze | Coze | - |
|
||||
| Llamadas por la interfaz Xinference | Xinference | - |
|
||||
| Llamadas por la interfaz HomeAssistant | HomeAssistant | - |
|
||||
|
||||
En la práctica, se puede integrar cualquier LLM que admita llamadas por la interfaz OpenAI.
|
||||
|
||||
---
|
||||
|
||||
### VLLM, modelos de visión
|
||||
|
||||
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Llamadas por la interfaz OpenAI | Alibaba Bailian, Zhipu ChatGLMVLLM | Zhipu ChatGLMVLLM |
|
||||
|
||||
En la práctica, se puede integrar cualquier VLLM que admita llamadas por la interfaz OpenAI.
|
||||
|
||||
---
|
||||
|
||||
### TTS, síntesis de voz
|
||||
|
||||
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Llamadas por interfaz | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud y Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi Streaming TTS, MinimaxTTS | Lingxi Streaming TTS, EdgeTTS, CosyVoiceSiliconflow (parcial) |
|
||||
| Servicio local | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, Index-TTS, PaddleSpeech | Index-TTS, PaddleSpeech, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
|
||||
---
|
||||
|
||||
### VAD, detección de actividad de voz
|
||||
|
||||
| Tipo | Plataforma | Forma de uso | Precio | Notas |
|
||||
|:---:|:---------:|:----:|:----:|:--:|
|
||||
| VAD | SileroVAD | Uso local | Gratuito | |
|
||||
|
||||
---
|
||||
|
||||
### ASR, reconocimiento de voz
|
||||
|
||||
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Uso local | FunASR, SherpaASR | FunASR, SherpaASR |
|
||||
| Llamadas por interfaz | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR | FunASRServer |
|
||||
|
||||
---
|
||||
|
||||
### Voiceprint, huella de voz
|
||||
|
||||
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Uso local | 3D-Speaker | 3D-Speaker |
|
||||
|
||||
---
|
||||
|
||||
### Memory, almacenamiento de memoria
|
||||
|
||||
| Tipo | Plataforma | Forma de uso | Precio | Notas |
|
||||
|:------:|:---------------:|:----:|:---------:|:--:|
|
||||
| Memory | mem0ai | Llamadas por interfaz | Cuota de 1000 veces al mes | |
|
||||
| Memory | [powermem](../powermem-integration.md) | Resumen local | Depende del LLM y de la base de datos | OceanBase de código abierto; admite búsqueda inteligente |
|
||||
| Memory | mem_local_short | Resumen local | Gratuito | |
|
||||
| Memory | nomem | Modo sin memoria | Gratuito | |
|
||||
|
||||
---
|
||||
|
||||
### Intent, reconocimiento de intención
|
||||
|
||||
| Tipo | Plataforma | Forma de uso | Precio | Notas |
|
||||
|:------:|:-------------:|:----:|:-------:|:---------------------:|
|
||||
| Intent | intent_llm | Llamadas por interfaz | Según el precio del LLM | Reconoce la intención mediante el modelo de lenguaje (LLM); gran capacidad de generalización |
|
||||
| Intent | function_call | Llamadas por interfaz | Según el precio del LLM | Completa la intención con llamadas a funciones del modelo; rápido y con buen resultado |
|
||||
| Intent | nointent | Modo sin intención | Gratuito | No reconoce la intención y devuelve directamente el resultado del diálogo |
|
||||
|
||||
---
|
||||
|
||||
### Rag, generación aumentada por recuperación
|
||||
|
||||
| Tipo | Plataforma | Forma de uso | Precio | Notas |
|
||||
|:------:|:-------------:|:----:|:-------:|:---------------------:|
|
||||
| Rag | ragflow | Llamadas por interfaz | Cobrado según los tokens consumidos en el troceado y la segmentación | Usa la generación aumentada por recuperación de RagFlow para dar respuestas más precisas |
|
||||
|
||||
---
|
||||
|
||||
## Agradecimientos 🙏
|
||||
|
||||
| Logo | Proyecto o empresa | Descripción |
|
||||
|:---:|:---:|:---|
|
||||
| <img src="../images/logo_bailing.png" width="160"> | [Robot de diálogo por voz Bailing (百聆语音对话机器人)](https://github.com/wwbin2017/bailing) | Este proyecto parte del [robot de diálogo por voz Bailing](https://github.com/wwbin2017/bailing) y se construye sobre esa base |
|
||||
| <img src="../images/logo_tenclass.png" width="160"> | [Tenclass (十方融海)](https://www.tenclass.com/) | Gracias a [Tenclass](https://www.tenclass.com/) por definir el protocolo de comunicación estándar del ecosistema Xiaozhi, las soluciones de compatibilidad con varios dispositivos y los ejemplos prácticos de alta concurrencia, y por el apoyo de documentación técnica de todo el recorrido de este proyecto |
|
||||
| <img src="../images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology (玄凤科技)](https://github.com/Eric0308) | Gracias a [Xuanfeng Technology](https://github.com/Eric0308) por el código del marco de llamada a funciones, del protocolo de comunicación MCP y del mecanismo de llamadas basado en plugins. Con un sistema estandarizado de despacho de instrucciones y capacidad de ampliación dinámica, mejora de forma clara la eficiencia de interacción y la extensibilidad de los dispositivos de frontend (IoT) |
|
||||
| <img src="../images/logo_junsen.png" width="160"> | [huangjunsen](https://github.com/huangjunsen0406) | Gracias a [huangjunsen](https://github.com/huangjunsen0406) por el módulo `consola inteligente móvil`, que permite controlar con eficacia dispositivos móviles de varias plataformas e interactuar en tiempo real, y mejora mucho la comodidad de uso y la eficiencia de gestión en escenarios móviles |
|
||||
| <img src="../images/logo_huiyuan.png" width="160"> | [Huiyuan Design (汇远设计)](http://ui.kwd988.net/) | Gracias a [Huiyuan Design](http://ui.kwd988.net/) por la solución visual de este proyecto, y por poner al servicio de la experiencia de uso la práctica de diseño acumulada con más de mil empresas |
|
||||
| <img src="../images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology (西安勤人信息科技)](https://www.029app.com/) | Gracias a [Xi'an Qinren Information Technology](https://www.029app.com/) por profundizar el sistema visual de este proyecto y mantener un estilo coherente y ampliable en distintos escenarios |
|
||||
| <img src="../images/logo_contributors.png" width="160"> | [Quienes contribuyen código](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors) | Gracias a [todas las personas que contribuyen código](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors). Su trabajo hace el proyecto más sólido y más capaz. |
|
||||
|
||||
|
||||
<a href="https://www.star-history.com/?repos=xinnan-tech%2Fxiaozhi-esp32-server&type=date&legend=top-left">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&theme=dark&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
|
||||
</picture>
|
||||
</a>
|
||||
+26
-29
@@ -22,7 +22,6 @@ Suporte ao protocolo MQTT+UDP, protocolo WebSocket, ponto de acesso MCP, reconhe
|
||||
<a href="./README_vi.md"><img alt="Tiếng Việt" src="https://img.shields.io/badge/Tiếng Việt-DFE0E5"></a>
|
||||
<a href="./README_de.md"><img alt="Deutsch" src="https://img.shields.io/badge/Deutsch-DFE0E5"></a>
|
||||
<a href="./README_pt_BR.md"><img alt="Português (Brasil)" src="https://img.shields.io/badge/Português (Brasil)-DBEDFA"></a>
|
||||
<a href="./README_es.md"><img alt="Español" src="https://img.shields.io/badge/Español-DFE0E5"></a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">
|
||||
<img alt="GitHub Contributors" src="https://img.shields.io/github/v/release/xinnan-tech/xiaozhi-esp32-server?logo=docker" />
|
||||
</a>
|
||||
@@ -55,31 +54,31 @@ Quer ver os efeitos de uso? Clique nos vídeos abaixo 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
|
||||
<picture>
|
||||
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
|
||||
<img alt="响应速度感受" src="docs/images/demo9.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
|
||||
<picture>
|
||||
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
|
||||
<img alt="速度优化秘诀" src="docs/images/demo6.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
|
||||
<picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="docs/images/demo8.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
|
||||
<picture>
|
||||
<img alt="设备呼叫设备，打电话" src="../images/demo0.png" /></picture>
|
||||
<img alt="设备呼叫设备，打电话" src="docs/images/demo0.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
|
||||
<picture>
|
||||
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
|
||||
<img alt="复杂医疗场景" src="docs/images/demo1.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -87,31 +86,31 @@ Quer ver os efeitos de uso? Clique nos vídeos abaixo 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
|
||||
<picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="docs/images/demo7.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
|
||||
<picture>
|
||||
<img alt="实时打断" src="../images/demo10.png" /></picture>
|
||||
<img alt="实时打断" src="docs/images/demo10.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
|
||||
<picture>
|
||||
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
|
||||
<img alt="拍照识物品" src="docs/images/demo12.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
|
||||
<picture>
|
||||
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
|
||||
<img alt="控制家电开关" src="docs/images/demo5.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
|
||||
<picture>
|
||||
<img alt="多指令任务" src="../images/demo11.png" /></picture>
|
||||
<img alt="多指令任务" src="docs/images/demo11.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -119,31 +118,31 @@ Quer ver os efeitos de uso? Clique nos vídeos abaixo 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
|
||||
<picture>
|
||||
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
|
||||
<img alt="MCP接入点" src="docs/images/demo13.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
|
||||
<picture>
|
||||
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
|
||||
<img alt="MQTT指令下发" src="docs/images/demo4.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
|
||||
<picture>
|
||||
<img alt="声纹识别" src="../images/demo14.png" /></picture>
|
||||
<img alt="声纹识别" src="docs/images/demo14.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
|
||||
<picture>
|
||||
<img alt="自定义音色" src="../images/demo2.png" /></picture>
|
||||
<img alt="自定义音色" src="docs/images/demo2.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
|
||||
<picture>
|
||||
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
|
||||
<img alt="使用粤语交流" src="docs/images/demo3.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -168,8 +167,8 @@ Este projeto oferece dois métodos de implantação. Por favor, escolha de acord
|
||||
#### 🚀 Seleção do Método de Implantação
|
||||
| Método de Implantação | Funcionalidades | Cenários Aplicáveis | Documentação de Implantação | Requisitos de Configuração | Tutoriais em Vídeo |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Instalação Simplificada** | Diálogo inteligente, gerenciamento de agente único | Ambientes de baixa configuração, dados armazenados em arquivos de configuração, sem necessidade de banco de dados | [①Versão Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Implantação via Código-Fonte](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| `FunASR` local exige mais memória; usar apenas APIs mantém o consumo de recursos baixo | - |
|
||||
| **Instalação de Módulo Completo** | Diálogo inteligente, gerenciamento multiusuário, gerenciamento de múltiplos agentes, operação de interface do console inteligente | Experiência com funcionalidade completa, dados armazenados em banco de dados |[①Versão Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Implantação via Código-Fonte](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Tutorial de Atualização Automática via Código-Fonte](../dev-ops-integration.md) | `FunASR` local exige mais memória e recursos de banco de dados; usar apenas APIs mantém o consumo de recursos baixo| [Tutorial em Vídeo de Inicialização via Código-Fonte Local](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **Instalação Simplificada** | Diálogo inteligente, gerenciamento de agente único | Ambientes de baixa configuração, dados armazenados em arquivos de configuração, sem necessidade de banco de dados | [①Versão Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Implantação via Código-Fonte](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 núcleos 4GB se usar `FunASR`, 2 núcleos 2GB se todas APIs | - |
|
||||
| **Instalação de Módulo Completo** | Diálogo inteligente, gerenciamento multiusuário, gerenciamento de múltiplos agentes, operação de interface do console inteligente | Experiência com funcionalidade completa, dados armazenados em banco de dados |[①Versão Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Implantação via Código-Fonte](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Tutorial de Atualização Automática via Código-Fonte](../dev-ops-integration.md) | 4 núcleos 8GB se usar `FunASR`, 2 núcleos 4GB se todas APIs| [Tutorial em Vídeo de Inicialização via Código-Fonte Local](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
Perguntas frequentes e tutoriais relacionados podem ser consultados [neste link](../FAQ.md)
|
||||
|
||||
@@ -211,7 +210,7 @@ Este projeto fornece as seguintes ferramentas de teste para ajudá-lo a verifica
|
||||
| Nome da Ferramenta | Localização | Método de Uso | Descrição da Função |
|
||||
|:---:|:---|:---:|:---:|
|
||||
| Ferramenta de Teste de Interação por Áudio | main》digital-human》index.html | Execute `python start.py` em `main/digital-human` e depois abra `http://127.0.0.1:8006/index.html` | Testa as funções de reprodução e recepção de áudio, verifica se o processamento de áudio no lado Python está normal |
|
||||
| Ferramenta de Teste de Resposta de Modelo | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Testa a velocidade de resposta de módulos principais como ASR(reconhecimento de fala), LLM(modelo de linguagem), VLLM(modelo de visão), TTS(síntese de voz) |
|
||||
| Ferramenta de Teste de Resposta de Modelo | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Testa a velocidade de resposta dos três módulos principais: ASR(reconhecimento de fala), LLM(modelo de linguagem), VLLM(modelo de visão), TTS(síntese de voz) |
|
||||
|
||||
> 💡 Nota: Ao testar a velocidade dos modelos, apenas os modelos com chaves configuradas serão testados.
|
||||
|
||||
@@ -231,10 +230,10 @@ Este projeto fornece as seguintes ferramentas de teste para ajudá-lo a verifica
|
||||
| Base de Conhecimento | Suporta base de conhecimento RAGFlow, permitindo que o LLM julgue se deve acionar a base de conhecimento após receber a pergunta do usuário, e então responda à pergunta |
|
||||
| Chamada de Ferramentas | Suporta protocolo IOT do cliente, protocolo MCP do cliente, protocolo MCP do servidor, protocolo de endpoint MCP, funções de ferramentas personalizadas |
|
||||
| Envio de Comandos | Suporta envio de comandos MCP para dispositivos ESP32 via protocolo MQTT a partir do Console Inteligente |
|
||||
| Backend de Gerenciamento | Fornece interface de gerenciamento Web, suporta gerenciamento de usuários, configuração do sistema e gerenciamento de dispositivos; Suporta exibição em Chinês Simplificado, Chinês Tradicional, Inglês, Alemão, Vietnamita e Português do Brasil |
|
||||
| Ferramentas de Teste | Fornece ferramentas de teste de desempenho de resposta de modelo e ferramentas de teste de interação por áudio |
|
||||
| Backend de Gerenciamento | Fornece interface de gerenciamento Web, suporta gerenciamento de usuários, configuração do sistema e gerenciamento de dispositivos; Suporta exibição em Chinês Simplificado, Chinês Tradicional e Inglês |
|
||||
| Ferramentas de Teste | Fornece ferramentas de teste de desempenho, ferramentas de teste de modelo de visão e ferramentas de teste de interação por áudio |
|
||||
| Suporte à Implantação | Suporta implantação via Docker e implantação local, fornece gerenciamento completo de arquivos de configuração |
|
||||
| Sistema de Plugins | Suporta extensões de plugins funcionais e desenvolvimento de plugins personalizados; novos arquivos de plugin são carregados na próxima conexão do dispositivo e alterações em plugins já carregados exigem reiniciar o serviço |
|
||||
| Sistema de Plugins | Suporta extensões de plugins funcionais, desenvolvimento de plugins personalizados e carregamento dinâmico de plugins |
|
||||
|
||||
### Em Desenvolvimento 🚧
|
||||
|
||||
@@ -280,8 +279,8 @@ Na verdade, qualquer VLLM que suporte chamadas via interface OpenAI pode ser int
|
||||
|
||||
| Método de Uso | Plataformas Suportadas | Plataformas Gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Chamadas via interface | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud e Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS | EdgeTTS, CosyVoiceSiliconflow(parcial) |
|
||||
| Serviços locais | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, IndexStreamTTS, PaddleSpeechTTS | IndexStreamTTS, PaddleSpeechTTS, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
| Chamadas via interface | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud e Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi Streaming TTS, MinimaxTTS | Lingxi Streaming TTS, EdgeTTS, CosyVoiceSiliconflow(parcial) |
|
||||
| Serviços locais | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, Index-TTS, PaddleSpeech | Index-TTS, PaddleSpeech, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
|
||||
---
|
||||
|
||||
@@ -297,8 +296,8 @@ Na verdade, qualquer VLLM que suporte chamadas via interface OpenAI pode ser int
|
||||
|
||||
| Método de Uso | Plataformas Suportadas | Plataformas Gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Uso local | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
|
||||
| Chamadas via interface | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
|
||||
| Uso local | FunASR, SherpaASR | FunASR, SherpaASR |
|
||||
| Chamadas via interface | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR | FunASRServer |
|
||||
|
||||
---
|
||||
|
||||
@@ -306,9 +305,7 @@ Na verdade, qualquer VLLM que suporte chamadas via interface OpenAI pode ser int
|
||||
|
||||
| Método de Uso | Plataformas Suportadas | Plataformas Gratuitas |
|
||||
|:---:|:---:|:---:|
|
||||
| Chamadas via interface | Serviço de voiceprint (voiceprint-api) | - |
|
||||
|
||||
O reconhecimento de voz por impressão digital exige um serviço [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) implantado separadamente (baseado em 3D-Speaker) e MySQL. Veja o [guia de integração de voiceprint](../voiceprint-integration.md).
|
||||
| Uso local | 3D-Speaker | 3D-Speaker |
|
||||
|
||||
---
|
||||
|
||||
|
||||
+26
-29
@@ -22,7 +22,6 @@ Hỗ trợ giao thức MQTT+UDP, giao thức Websocket, điểm truy cập MCP,
|
||||
<a href="./README_vi.md"><img alt="Tiếng Việt" src="https://img.shields.io/badge/Tiếng Việt-DBEDFA"></a>
|
||||
<a href="./README_de.md"><img alt="Deutsch" src="https://img.shields.io/badge/Deutsch-DFE0E5"></a>
|
||||
<a href="./README_pt_BR.md"><img alt="Português (Brasil)" src="https://img.shields.io/badge/Português (Brasil)-DFE0E5"></a>
|
||||
<a href="./README_es.md"><img alt="Español" src="https://img.shields.io/badge/Español-DFE0E5"></a>
|
||||
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">
|
||||
<img alt="GitHub Contributors" src="https://img.shields.io/github/v/release/xinnan-tech/xiaozhi-esp32-server?logo=docker" />
|
||||
</a>
|
||||
@@ -55,31 +54,31 @@ Muốn xem hiệu quả sử dụng? Hãy xem video 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
|
||||
<picture>
|
||||
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
|
||||
<img alt="响应速度感受" src="docs/images/demo9.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
|
||||
<picture>
|
||||
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
|
||||
<img alt="速度优化秘诀" src="docs/images/demo6.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
|
||||
<picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
|
||||
<img alt="小智数字人 支持语音唤醒" src="docs/images/demo8.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
|
||||
<picture>
|
||||
<img alt="设备呼叫设备，打电话" src="../images/demo0.png" /></picture>
|
||||
<img alt="设备呼叫设备，打电话" src="docs/images/demo0.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
|
||||
<picture>
|
||||
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
|
||||
<img alt="复杂医疗场景" src="docs/images/demo1.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -87,31 +86,31 @@ Muốn xem hiệu quả sử dụng? Hãy xem video 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
|
||||
<picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
|
||||
<img alt="播放音乐查询天气播报新闻" src="docs/images/demo7.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
|
||||
<picture>
|
||||
<img alt="实时打断" src="../images/demo10.png" /></picture>
|
||||
<img alt="实时打断" src="docs/images/demo10.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
|
||||
<picture>
|
||||
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
|
||||
<img alt="拍照识物品" src="docs/images/demo12.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
|
||||
<picture>
|
||||
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
|
||||
<img alt="控制家电开关" src="docs/images/demo5.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
|
||||
<picture>
|
||||
<img alt="多指令任务" src="../images/demo11.png" /></picture>
|
||||
<img alt="多指令任务" src="docs/images/demo11.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -119,31 +118,31 @@ Muốn xem hiệu quả sử dụng? Hãy xem video 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
|
||||
<picture>
|
||||
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
|
||||
<img alt="MCP接入点" src="docs/images/demo13.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
|
||||
<picture>
|
||||
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
|
||||
<img alt="MQTT指令下发" src="docs/images/demo4.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
|
||||
<picture>
|
||||
<img alt="声纹识别" src="../images/demo14.png" /></picture>
|
||||
<img alt="声纹识别" src="docs/images/demo14.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
|
||||
<picture>
|
||||
<img alt="自定义音色" src="../images/demo2.png" /></picture>
|
||||
<img alt="自定义音色" src="docs/images/demo2.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
|
||||
<picture>
|
||||
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
|
||||
<img alt="使用粤语交流" src="docs/images/demo3.png" /></picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -169,8 +168,8 @@ Dự án này cung cấp hai phương pháp triển khai, vui lòng chọn theo
|
||||
#### 🚀 Lựa chọn phương pháp triển khai
|
||||
| Phương pháp triển khai | Đặc điểm | Tình huống áp dụng | Tài liệu triển khai | Yêu cầu cấu hình | Video hướng dẫn |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Cài đặt tối giản** | Đối thoại thông minh, quản lý đơn tác nhân | Môi trường cấu hình thấp, dữ liệu lưu trong tệp cấu hình, không cần cơ sở dữ liệu | [①Phiên bản Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Triển khai mã nguồn](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| `FunASR` cục bộ cần nhiều bộ nhớ; dùng toàn API thì mức dùng tài nguyên thấp | - |
|
||||
| **Cài đặt toàn bộ module** | Đối thoại thông minh, quản lý đa người dùng, quản lý đa tác nhân, bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](../dev-ops-integration.md) | `FunASR` cục bộ cần nhiều bộ nhớ, thêm tài nguyên cơ sở dữ liệu; dùng toàn API thì mức dùng tài nguyên thấp| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **Cài đặt tối giản** | Đối thoại thông minh, quản lý đơn tác nhân | Môi trường cấu hình thấp, dữ liệu lưu trong tệp cấu hình, không cần cơ sở dữ liệu | [①Phiên bản Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Triển khai mã nguồn](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 nhân 4GB nếu dùng `FunASR`, 2 nhân 2GB nếu toàn API | - |
|
||||
| **Cài đặt toàn bộ module** | Đối thoại thông minh, quản lý đa người dùng, quản lý đa tác nhân, bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](../dev-ops-integration.md) | 4 nhân 8GB nếu dùng `FunASR`, 2 nhân 4GB nếu toàn API| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
Câu hỏi thường gặp và hướng dẫn liên quan, vui lòng tham khảo [liên kết này](../FAQ.md)
|
||||
|
||||
@@ -212,7 +211,7 @@ Dự án này cung cấp các công cụ kiểm tra sau để giúp bạn xác m
|
||||
| Tên công cụ | Vị trí | Phương pháp sử dụng | Mô tả chức năng |
|
||||
|:---:|:---|:---:|:---:|
|
||||
| Công cụ kiểm tra tương tác âm thanh | main》digital-human》index.html | Chạy `python start.py` trong `main/digital-human`, sau đó mở `http://127.0.0.1:8006/index.html` | Kiểm tra chức năng phát và nhận âm thanh, xác minh xử lý âm thanh phía Python có bình thường không |
|
||||
| Công cụ kiểm tra phản hồi mô hình | main》xiaozhi-server》performance_tester.py | Thực hiện `python performance_tester.py` | Kiểm tra tốc độ phản hồi của các module cốt lõi như ASR(Nhận dạng giọng nói), LLM(Mô hình lớn), VLLM(Mô hình thị giác), TTS(Tổng hợp giọng nói) |
|
||||
| Công cụ kiểm tra phản hồi mô hình | main》xiaozhi-server》performance_tester.py | Thực hiện `python performance_tester.py` | Kiểm tra tốc độ phản hồi của ba module cốt lõi ASR(Nhận dạng giọng nói), LLM(Mô hình lớn), VLLM(Mô hình thị giác), TTS(Tổng hợp giọng nói) |
|
||||
|
||||
> 💡 Gợi ý: Khi kiểm tra tốc độ mô hình, chỉ kiểm tra các mô hình đã cấu hình khóa.
|
||||
|
||||
@@ -232,10 +231,10 @@ Dự án này cung cấp các công cụ kiểm tra sau để giúp bạn xác m
|
||||
| Kho tri thức | Hỗ trợ kho tri thức RAGFlow, cho phép mô hình lớn đánh giá cần gọi kho tri thức trước khi trả lời |
|
||||
| Gọi công cụ | Hỗ trợ giao thức IOT phía client, giao thức MCP phía client, giao thức MCP phía server, giao thức điểm truy cập MCP, hàm công cụ tùy chỉnh |
|
||||
| Gửi lệnh | Dựa vào giao thức MQTT, hỗ trợ gửi lệnh MCP từ bảng điều khiển thông minh xuống thiết bị ESP32 |
|
||||
| Backend quản lý | Cung cấp giao diện quản lý Web, hỗ trợ quản lý người dùng, cấu hình hệ thống và quản lý thiết bị; giao diện hỗ trợ hiển thị tiếng Trung giản thể, tiếng Trung phồn thể, tiếng Anh, tiếng Đức, tiếng Việt và tiếng Bồ Đào Nha Brazil |
|
||||
| Công cụ kiểm tra | Cung cấp công cụ kiểm tra hiệu suất phản hồi mô hình và công cụ kiểm tra tương tác âm thanh |
|
||||
| Backend quản lý | Cung cấp giao diện quản lý Web, hỗ trợ quản lý người dùng, cấu hình hệ thống và quản lý thiết bị; giao diện hỗ trợ hiển thị tiếng Trung giản thể, tiếng Trung phồn thể, tiếng Anh |
|
||||
| Công cụ kiểm tra | Cung cấp công cụ kiểm tra hiệu suất, công cụ kiểm tra mô hình thị giác và công cụ kiểm tra tương tác âm thanh |
|
||||
| Hỗ trợ triển khai | Hỗ trợ triển khai Docker và triển khai cục bộ, cung cấp quản lý tệp cấu hình hoàn chỉnh |
|
||||
| Hệ thống plugin | Hỗ trợ mở rộng plugin chức năng và phát triển plugin tùy chỉnh; file plugin mới được tải ở lần kết nối thiết bị tiếp theo, thay đổi mã của plugin đã tải cần khởi động lại dịch vụ |
|
||||
| Hệ thống plugin | Hỗ trợ mở rộng plugin chức năng, phát triển plugin tùy chỉnh và hot loading plugin |
|
||||
|
||||
### Đang phát triển 🚧
|
||||
|
||||
@@ -281,8 +280,8 @@ Trên thực tế, bất kỳ VLLM nào hỗ trợ gọi giao diện openai đ
|
||||
|
||||
| Phương pháp sử dụng | Nền tảng hỗ trợ | Nền tảng miễn phí |
|
||||
|:---:|:---:|:---:|
|
||||
| Gọi giao diện | EdgeTTS, iFlytek, Volcano Engine, Tencent Cloud, Alibaba Cloud và Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS | EdgeTTS, CosyVoiceSiliconflow(một phần) |
|
||||
| Dịch vụ cục bộ | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, IndexStreamTTS, PaddleSpeechTTS | IndexStreamTTS, PaddleSpeechTTS, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
| Gọi giao diện | EdgeTTS, iFlytek, Volcano Engine, Tencent Cloud, Alibaba Cloud và Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi streaming TTS, MinimaxTTS | Lingxi streaming TTS, EdgeTTS, CosyVoiceSiliconflow(một phần) |
|
||||
| Dịch vụ cục bộ | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, Index-TTS, PaddleSpeech | Index-TTS, PaddleSpeech, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
|
||||
|
||||
---
|
||||
|
||||
@@ -298,8 +297,8 @@ Trên thực tế, bất kỳ VLLM nào hỗ trợ gọi giao diện openai đ
|
||||
|
||||
| Phương pháp sử dụng | Nền tảng hỗ trợ | Nền tảng miễn phí |
|
||||
|:---:|:---:|:---:|
|
||||
| Sử dụng cục bộ | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
|
||||
| Gọi giao diện | FunASRServer, Volcano Engine, iFlytek, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
|
||||
| Sử dụng cục bộ | FunASR, SherpaASR | FunASR, SherpaASR |
|
||||
| Gọi giao diện | FunASRServer, Volcano Engine, iFlytek, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR | FunASRServer |
|
||||
|
||||
---
|
||||
|
||||
@@ -307,9 +306,7 @@ Trên thực tế, bất kỳ VLLM nào hỗ trợ gọi giao diện openai đ
|
||||
|
||||
| Phương pháp sử dụng | Nền tảng hỗ trợ | Nền tảng miễn phí |
|
||||
|:---:|:---:|:---:|
|
||||
| Gọi giao diện | Dịch vụ nhận dạng giọng nói (voiceprint-api) | - |
|
||||
|
||||
Nhận dạng giọng nói theo dấu vân tay cần dịch vụ [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) được triển khai riêng (dựa trên 3D-Speaker) và MySQL. Xem [hướng dẫn tích hợp voiceprint](../voiceprint-integration.md).
|
||||
| Sử dụng cục bộ | 3D-Speaker | 3D-Speaker |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -46,6 +46,10 @@
|
||||
8. 将复制过来的第二个关键配置`API Host`,填入到`开发者 API Host`里
|
||||
9. 保存配置，再保存智能体配置
|
||||
|
||||
智控台这边插件参数里的「API 密钥」和「API Host」两栏默认是空的：仓库早期版本内置过一把已经失效的和风天气凭据，数据库变更脚本把它清掉了，得照着本文件的「API Key 申请指南」申请自己的一套再填。字段名后面括号里写了申请路径。
|
||||
|
||||
脚本只清仓库内置的那一把，你自己填的凭据原样保留。以后要换凭据，在智控台「角色配置 → 编辑功能 → 天气查询」的参数表单里改完保存就行。清理的范围见文末那节。
|
||||
|
||||
### 方式2. 如果你只是单模块xiaozhi-server部署
|
||||
|
||||
在 `data/.config.yaml` 中配置：
|
||||
@@ -62,3 +66,13 @@ plugins:
|
||||
default_location: "你的默认查询城市"
|
||||
```
|
||||
|
||||
## 智控台那边内置的凭据清到了什么程度
|
||||
|
||||
早期版本把一把和风天气的 API Key 和一个 API Host 写在模型提供方「天气插件」那一行的字段默认值里，角色配置的表单拿这个默认值预填，不填就等于沿用仓库里那一份。那把 key 已经失效，仓库也不该带着凭据分发。数据库变更脚本做两件事：
|
||||
|
||||
- 把那一行的默认值清空，同时把申请路径写进字段名，用户在界面上能直接看到去哪儿申请
|
||||
- 把已有智能体的参数里还等于那把内置 key 的项清空
|
||||
|
||||
清不掉的有两类。一是管理员自己填的值，脚本不动，误清就是误删人家的凭据。二是这一种：管理员在界面上改过那一行的默认值之后，仓库内置的那把失效 key 就被覆盖了，再也认不出来，早于管理员改动建立、参数里还存着那把旧内置 key 的智能体不会被自动清掉，只能由管理员自己在界面上改。
|
||||
|
||||
这两条清理跟着另外两个改动走：一个是插件在没有凭据时给出提示而不是报错，另一个是 CI 真的会跑测试，这条变更脚本才在 CI 里被验证过。两者都要先落地。
|
||||
@@ -16,8 +16,6 @@ digital-human 是独立的数字人测试模块，负责提供本地测试页面
|
||||
pip install -r wakeword_runtime/requirements.txt
|
||||
```
|
||||
|
||||
唤醒词模型要另外下载：仓库里不带模型文件，默认配置（`wakeword_runtime/config.json` 里 `wakeword.enabled` 为 true）下，`wakeword_runtime/models/` 里没有模型文件时启动会打印「警告: 唤醒词服务启动失败」。页面能打开，麦克风的输入流也已经打开，但没有唤醒词检测在消费这些数据，语音唤不醒，页面上的其它功能不受影响。模型下载地址和摆放位置见[唤醒词专题文档](../../docs/digital-human-wakeword.md)。
|
||||
|
||||
启动模块：
|
||||
|
||||
```bash
|
||||
|
||||
@@ -1,6 +1,3 @@
|
||||
sherpa-onnx==1.12.29
|
||||
sounddevice>=0.4.4
|
||||
pypinyin==0.55.0
|
||||
# 唤醒词运行时在 core/detector.py、core/microphone.py 里直接用 numpy，
|
||||
# 版本跟 main/xiaozhi-server/requirements.txt 一致，两个模块共用一个环境时不会冲突
|
||||
numpy==1.26.4
|
||||
@@ -0,0 +1,132 @@
|
||||
-- 天气插件不再内置和风天气凭据
|
||||
--
|
||||
-- 这条变更记录在 db.changelog-master.yaml 里标了 runOnChange：脚本内容在这个
|
||||
-- PR 里改过好几轮，跑过旧版的库启动时直接重跑新版，不会报校验不一致。
|
||||
-- 那不是手滑，别删掉这个标记
|
||||
--
|
||||
-- 早期版本把一把和风天气的 API Key 和一个 API Host 写在模型提供方
|
||||
-- SYSTEM_PLUGIN_WEATHER 的字段默认值里。智控台的角色配置表单拿这个默认值预填，
|
||||
-- 用户不填就等于沿用仓库里那一份。那把 key 已经失效，仓库里也不该带着凭据分发，
|
||||
-- 用的人也不知道它从哪来、该不该用。
|
||||
--
|
||||
-- 这里清两处：
|
||||
-- 1. 字段默认值置空，同时把申请路径写进字段名，用户在界面上就能看到去哪儿申请
|
||||
-- 2. 已有智能体的参数里还等于内置值的那些，值清空。用户自己填的 key 不动
|
||||
--
|
||||
-- 关键前提：模型提供方那一行的「默认值」管理员能在界面上改，不能直接拿它当
|
||||
-- 「内置值」的基准——那样会把管理员自己填的凭据清掉。判断这一行有没有被人动过，
|
||||
-- 看两列：种子数据插进来的时候 updater 是 0、update_date 跟 create_date 一样；
|
||||
-- 界面上保存过一次之后，updater 变成操作人、update_date 变成改的时间。
|
||||
-- 只有从没被改过的那一行，它里面的默认值才是仓库内置的那一份，才拿来当基准。
|
||||
-- 被改过的一律不动——那是管理员自己的值，误清就是误删人家的凭据。
|
||||
--
|
||||
-- 有一处清不干净，说明白：管理员动过这一行之后，仓库内置的那把失效 key 就再也
|
||||
-- 认不出来了（它被管理员的值覆盖了），早期建立、参数里还存着那把失效 key 的智能体
|
||||
-- 只能由管理员自己在界面上改。默认配置那边同理，只能整行不动。
|
||||
--
|
||||
-- 内置值从模型提供方那一行现读，不写死在脚本里，免得仓库里再留一份凭据
|
||||
|
||||
-- 基准行有没有被人编辑过
|
||||
SET @weather_provider_untouched = (
|
||||
SELECT COUNT(*)
|
||||
FROM `ai_model_provider`
|
||||
WHERE `id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND `updater` = 0
|
||||
AND `update_date` = `create_date`
|
||||
);
|
||||
|
||||
-- 内置值。基准行被改过就没有内置值可比，取不到
|
||||
SET @weather_api_key = (
|
||||
SELECT jt.`field_default`
|
||||
FROM `ai_model_provider` p,
|
||||
JSON_TABLE(p.`fields`, '$[*]' COLUMNS (
|
||||
`field_key` VARCHAR(64) PATH '$.key',
|
||||
`field_default` VARCHAR(255) PATH '$.default'
|
||||
)) jt
|
||||
WHERE p.`id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_provider_untouched > 0
|
||||
AND jt.`field_key` = 'api_key'
|
||||
LIMIT 1
|
||||
);
|
||||
|
||||
SET @weather_api_host = (
|
||||
SELECT jt.`field_default`
|
||||
FROM `ai_model_provider` p,
|
||||
JSON_TABLE(p.`fields`, '$[*]' COLUMNS (
|
||||
`field_key` VARCHAR(64) PATH '$.key',
|
||||
`field_default` VARCHAR(255) PATH '$.default'
|
||||
)) jt
|
||||
WHERE p.`id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_provider_untouched > 0
|
||||
AND jt.`field_key` = 'api_host'
|
||||
LIMIT 1
|
||||
);
|
||||
|
||||
-- 字段在数组里的位置。数组下标从 0 起，JSON_TABLE 的序号从 1 起，两者差一
|
||||
SET @weather_api_key_pos = (
|
||||
SELECT jt.`field_ord`
|
||||
FROM `ai_model_provider` p,
|
||||
JSON_TABLE(p.`fields`, '$[*]' COLUMNS (
|
||||
`field_ord` FOR ORDINALITY,
|
||||
`field_key` VARCHAR(64) PATH '$.key'
|
||||
)) jt
|
||||
WHERE p.`id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_provider_untouched > 0
|
||||
AND jt.`field_key` = 'api_key'
|
||||
LIMIT 1
|
||||
);
|
||||
|
||||
SET @weather_api_host_pos = (
|
||||
SELECT jt.`field_ord`
|
||||
FROM `ai_model_provider` p,
|
||||
JSON_TABLE(p.`fields`, '$[*]' COLUMNS (
|
||||
`field_ord` FOR ORDINALITY,
|
||||
`field_key` VARCHAR(64) PATH '$.key'
|
||||
)) jt
|
||||
WHERE p.`id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_provider_untouched > 0
|
||||
AND jt.`field_key` = 'api_host'
|
||||
LIMIT 1
|
||||
);
|
||||
|
||||
-- 智能体参数里还等于内置值的项，清空。基准行被改过就取不到内置值，这时一律不动：
|
||||
-- 管理员自己填的值不能碰，早于管理员编辑建立的智能体里存的旧内置值也认不出来了
|
||||
UPDATE `ai_agent_plugin_mapping`
|
||||
SET `param_info` = JSON_SET(`param_info`, '$.api_key', '')
|
||||
WHERE `plugin_id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_api_key IS NOT NULL
|
||||
AND @weather_api_key <> ''
|
||||
AND JSON_UNQUOTE(JSON_EXTRACT(`param_info`, '$.api_key')) = @weather_api_key;
|
||||
|
||||
UPDATE `ai_agent_plugin_mapping`
|
||||
SET `param_info` = JSON_SET(`param_info`, '$.api_host', '')
|
||||
WHERE `plugin_id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_api_host IS NOT NULL
|
||||
AND @weather_api_host <> ''
|
||||
AND JSON_UNQUOTE(JSON_EXTRACT(`param_info`, '$.api_host')) = @weather_api_host;
|
||||
|
||||
-- 字段默认值置空，字段名里写清申请路径。默认值那一栏在界面上还带着提示，
|
||||
-- 光把值清空没用，填表的人根本不知道该去哪儿拿
|
||||
-- 两项各自一条更新，各守卫各的字段位置：管理员把其中一项的字段删了，
|
||||
-- 另一项的默认值照样要清掉，不能因为少一个位置就整条都跳过
|
||||
UPDATE `ai_model_provider`
|
||||
SET `fields` = JSON_SET(
|
||||
`fields`,
|
||||
CONCAT('$[', @weather_api_key_pos - 1, '].default'), '',
|
||||
CONCAT('$[', @weather_api_key_pos - 1, '].label'),
|
||||
'天气插件 API 密钥（到和风天气控制台 https://console.qweather.com/project?lang=zh 申请，创建应用时勾选 GeoAPI 城市查询、实时天气、7 天天气预报三项权限；仓库里的 docs/weather-integration.md 有申请步骤）'
|
||||
)
|
||||
WHERE `id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_provider_untouched > 0
|
||||
AND @weather_api_key_pos IS NOT NULL;
|
||||
|
||||
UPDATE `ai_model_provider`
|
||||
SET `fields` = JSON_SET(
|
||||
`fields`,
|
||||
CONCAT('$[', @weather_api_host_pos - 1, '].default'), '',
|
||||
CONCAT('$[', @weather_api_host_pos - 1, '].label'),
|
||||
'开发者 API Host（和风天气控制台「设置」https://console.qweather.com/setting?lang=zh 里查看，跟 API 密钥配套使用；仓库里的 docs/weather-integration.md 有申请步骤）'
|
||||
)
|
||||
WHERE `id` = 'SYSTEM_PLUGIN_WEATHER'
|
||||
AND @weather_provider_untouched > 0
|
||||
AND @weather_api_host_pos IS NOT NULL;
|
||||
@@ -0,0 +1,15 @@
|
||||
-- 兜底：早期版本把天气插件的凭据放在系统参数里，那两条记录后来被 202505292203.sql
|
||||
-- 删掉了，新装的库里本来就没有这两行。这里再置一次空，防止有环境绕过了那步删除。
|
||||
-- 正常情况下影响零行
|
||||
--
|
||||
-- 这条变更记录在 db.changelog-master.yaml 里标了 runOnChange：脚本内容在这个
|
||||
-- PR 里改过，跑过旧版的库启动时直接重跑新版，不会报校验不一致。那不是手滑
|
||||
--
|
||||
-- 只清从没被人改过的行：系统参数界面里这两行也能改，管理员填进去的凭据不能被抹掉。
|
||||
-- 种子数据插进来时没写 updater 和 update_date，这两列是空的；界面上保存过一次之后
|
||||
-- 就会写上操作人和时间
|
||||
UPDATE `sys_params`
|
||||
SET `param_value` = ''
|
||||
WHERE `param_code` IN ('plugins.get_weather.api_key', 'plugins.get_weather.api_host')
|
||||
AND `updater` IS NULL
|
||||
AND `update_date` IS NULL;
|
||||
@@ -753,3 +753,21 @@ databaseChangeLog:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202609172100.sql
|
||||
# 下面两条是清理天气插件内置凭据用的，脚本内容在本 PR 里改过好几次，
|
||||
# 标成 runOnChange：跑过旧版的库启动时直接重跑新版，不会报校验不一致
|
||||
- changeSet:
|
||||
id: 202610241030
|
||||
author: hrz
|
||||
runOnChange: true
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202610241030.sql
|
||||
- changeSet:
|
||||
id: 202610241130
|
||||
author: hrz
|
||||
runOnChange: true
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202610241130.sql
|
||||
@@ -17,11 +17,7 @@ class MCPClient:
|
||||
self.name_mapping = {}
|
||||
self.ready = False
|
||||
self.call_results = {} # To store Futures for tool call responses
|
||||
# Handshake hardcodes id=1 (initialize) and id=2 (tools/list) in mcp_handler.
|
||||
# JSON-RPC 2.0 forbids reusing request ids on the same connection, and
|
||||
# compliant device MCP servers reject tools/call with id 1/2 (see #3461).
|
||||
# Official protocol sequence expects the first tools/call to use id=3.
|
||||
self.next_id = 3
|
||||
self.next_id = 1
|
||||
self.lock = asyncio.Lock()
|
||||
self._cached_available_tools = None # Cache for get_available_tools
|
||||
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
import json
|
||||
import asyncio
|
||||
import re
|
||||
from core.utils.util import get_vision_url
|
||||
from concurrent.futures import Future
|
||||
from core.utils.util import get_vision_url, sanitize_tool_name
|
||||
from core.utils.auth import AuthToken
|
||||
from config.logger import setup_logging
|
||||
from .mcp_client import MCPClient
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -15,6 +15,91 @@ if TYPE_CHECKING:
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
|
||||
|
||||
class MCPClient:
|
||||
"""设备端MCP客户端，用于管理MCP状态和工具"""
|
||||
|
||||
def __init__(self):
|
||||
self.tools = {} # sanitized_name -> tool_data
|
||||
self.name_mapping = {}
|
||||
self.ready = False
|
||||
self.call_results = {} # To store Futures for tool call responses
|
||||
self.next_id = 1
|
||||
self.lock = asyncio.Lock()
|
||||
self._cached_available_tools = None # Cache for get_available_tools
|
||||
|
||||
def has_tool(self, name: str) -> bool:
|
||||
return name in self.tools
|
||||
|
||||
def get_available_tools(self) -> list:
|
||||
# Check if the cache is valid
|
||||
if self._cached_available_tools is not None:
|
||||
return self._cached_available_tools
|
||||
|
||||
# If cache is not valid, regenerate the list
|
||||
result = []
|
||||
for tool_name, tool_data in self.tools.items():
|
||||
function_def = {
|
||||
"name": tool_name,
|
||||
"description": tool_data["description"],
|
||||
"parameters": {
|
||||
"type": tool_data["inputSchema"].get("type", "object"),
|
||||
"properties": tool_data["inputSchema"].get("properties", {}),
|
||||
"required": tool_data["inputSchema"].get("required", []),
|
||||
},
|
||||
}
|
||||
result.append({"type": "function", "function": function_def})
|
||||
|
||||
self._cached_available_tools = result # Store the generated list in cache
|
||||
return result
|
||||
|
||||
async def is_ready(self) -> bool:
|
||||
async with self.lock:
|
||||
return self.ready
|
||||
|
||||
async def set_ready(self, status: bool):
|
||||
async with self.lock:
|
||||
self.ready = status
|
||||
|
||||
async def add_tool(self, tool_data: dict):
|
||||
async with self.lock:
|
||||
sanitized_name = sanitize_tool_name(tool_data["name"])
|
||||
self.tools[sanitized_name] = tool_data
|
||||
self.name_mapping[sanitized_name] = tool_data["name"]
|
||||
self._cached_available_tools = (
|
||||
None # Invalidate the cache when a tool is added
|
||||
)
|
||||
|
||||
async def get_next_id(self) -> int:
|
||||
async with self.lock:
|
||||
current_id = self.next_id
|
||||
self.next_id += 1
|
||||
return current_id
|
||||
|
||||
async def register_call_result_future(self, id: int, future: Future):
|
||||
async with self.lock:
|
||||
self.call_results[id] = future
|
||||
|
||||
async def resolve_call_result(self, id: int, result: any):
|
||||
async with self.lock:
|
||||
if id in self.call_results:
|
||||
future = self.call_results.pop(id)
|
||||
if not future.done():
|
||||
future.set_result(result)
|
||||
|
||||
async def reject_call_result(self, id: int, exception: Exception):
|
||||
async with self.lock:
|
||||
if id in self.call_results:
|
||||
future = self.call_results.pop(id)
|
||||
if not future.done():
|
||||
future.set_exception(exception)
|
||||
|
||||
async def cleanup_call_result(self, id: int):
|
||||
async with self.lock:
|
||||
if id in self.call_results:
|
||||
self.call_results.pop(id)
|
||||
|
||||
|
||||
async def send_mcp_message(conn: "ConnectionHandler", payload: dict):
|
||||
"""Helper to send MCP messages, encapsulating common logic."""
|
||||
if not conn.features.get("mcp"):
|
||||
|
||||
@@ -18,9 +18,7 @@ class MCPEndpointClient:
|
||||
self.name_mapping = {}
|
||||
self.ready = False
|
||||
self.call_results = {} # To store Futures for tool call responses
|
||||
# Handshake hardcodes id=1 (initialize) and id=2 (tools/list) in
|
||||
# mcp_endpoint_handler; tools/call must start at 3 to avoid id reuse.
|
||||
self.next_id = 3
|
||||
self.next_id = 1
|
||||
self.lock = asyncio.Lock()
|
||||
self._cached_available_tools = None # Cache for get_available_tools
|
||||
self.websocket = None # WebSocket连接
|
||||
|
||||
@@ -2,7 +2,7 @@ import os
|
||||
import importlib.util
|
||||
import asyncio
|
||||
|
||||
print("使用前请根据docs/performance_tester.md的说明准备配置。")
|
||||
print("使用前请根据doc/performance_testerer.md的说明准备配置。")
|
||||
|
||||
|
||||
def list_performance_tester_modules():
|
||||
|
||||
Reference in new issue
Block a user