``` ├── .dockerignore ├── .github/ ├── ISSUE_TEMPLATE/ ├── bug_report.md ├── feature-or-enhancement-.md ├── question.md ├── .gitignore ├── .husky/ ├── commit-msg ├── .npmrc ├── .prettierrc.js ├── .windsurfrules ├── ARCHITECTURE.md ├── Dockerfile ├── README.md ├── README.zh-CN.md ├── app/ ├── api/ ├── check-update/ ├── route.js ├── llm/ ├── ollama/ ├── models/ ├── route.js ├── projects/ ├── [projectId]/ ├── chunks/ ├── [chunkId]/ ├── questions/ ├── route.js ├── route.js ├── config/ ├── route.js ├── datasets/ ├── optimize/ ├── route.js ├── route.js ├── files/ ├── route.js ├── generate-questions/ ├── route.js ├── llamaFactory/ ├── checkConfig/ ├── route.js ├── generate/ ├── route.js ├── models/ ├── [modelId]/ ├── route.js ├── route.js ├── pdf/ ├── route.js ├── playground/ ├── chat/ ├── route.js ├── stream/ ├── route.js ├── preview/ ├── [fileName]/ ├── route.js ├── questions/ ├── [questionId]/ ├── route.js ├── batch-delete/ ├── route.js ├── route.js ├── route.js ├── split/ ├── route.js ├── tags/ ├── route.js ├── tasks/ ├── route.js ├── text-split/ ├── route.js ├── route.js ├── update/ ├── route.js ├── dataset-square/ ├── page.js ├── globals.css ├── layout.js ├── page.js ├── projects/ ├── [projectId]/ ├── datasets/ ├── [datasetId]/ ├── page.js ├── page.js ├── layout.js ├── page.js ├── playground/ ├── page.js ├── questions/ ├── components/ ├── QuestionEditDialog.js ├── hooks/ ├── useQuestionEdit.js ├── page.js ├── settings/ ├── components/ ├── PromptSettings.js ├── page.js ├── text-split/ ├── page.js ├── commitlint.config.mjs ├── components/ ├── ExportDatasetDialog.js ``` ## /.dockerignore ```dockerignore path="/.dockerignore" node_modules .next .git .github README.md README.zh-CN.md .gitignore .env.local .env.development.local .env.test.local .env.production.local /test /local-db /video ``` ## /.github/ISSUE_TEMPLATE/bug_report.md --- name: Bug report about: Create a report to help us improve title: "[Bug]" labels: bug assignees: '' --- **问题描述** 清晰、简洁地描述该问题的具体情况。 **复现步骤** 重现该问题的操作步骤： 1. 进入“……”页面。 2. 点击“……”。 3. 向下滚动到“……”。 4. 这时会看到错误提示。 **预期结果** 清晰、简洁地描述你原本期望出现的情况。 **截图** 如果有必要，请附上截图，以便更好地说明你的问题。 **桌面设备（请完善以下信息）** - 操作系统：[例如：、Window、MAC] - 浏览器：[例如：谷歌浏览器（Chrome），苹果浏览器（Safari）] - Easy Dataset 版本：[例如：1.2.2] **其他相关信息** 在此处添加关于该问题的其他任何相关背景信息。 ## /.github/ISSUE_TEMPLATE/feature-or-enhancement-.md --- name: 'Feature or enhancement ' about: Suggest an idea for this project title: "[Feature]" labels: enhancement assignees: '' --- **你的功能请求是否与某个问题相关？请描述。** 清晰、简洁地描述一下存在的问题是什么。例如：当我[具体情况]时，我总是感到很沮丧。 **描述你期望的解决方案** 清晰、简洁地描述你希望实现的情况。 **描述你考虑过的替代方案** 清晰、简洁地描述你所考虑过的任何其他解决方案或功能。 **其他相关信息** 在此处添加与该功能请求相关的其他任何背景信息或截图。 ## /.github/ISSUE_TEMPLATE/question.md --- name: Question about: Ask questions you want to know title: "[Question]" labels: question assignees: '' --- ## /.gitignore ```gitignore path="/.gitignore" node_modules build .vscode website-local.json ai-local.json .next .DS_Store tsconfig.tsbuildinfo mock-login-callback.ts .env.local /src/test/crawler /src/test/mock /local-db /test /dist ``` ## /.husky/commit-msg ```husky/commit-msg path="/.husky/commit-msg" #!/usr/bin/env sh npx commitlint --edit "$1" ``` ## /.npmrc ```npmrc path="/.npmrc" registry=https://registry.npmjs.org ``` ## /.prettierrc.js ```js path="/.prettierrc.js" module.exports = { semi: true, trailingComma: 'none', singleQuote: true, tabWidth: 2, useTabs: false, bracketSpacing: true, arrowParens: 'avoid', proseWrap: 'preserve', jsxBracketSameLine: true, printWidth: 120, endOfLine: 'auto' }; ``` ## /.windsurfrules ```windsurfrules path="/.windsurfrules" # Easy DataSet 项目架构设计 ## 项目概述 Easy DataSet 是一个用于创建大模型微调数据集的应用程序。用户可以上传文本文件，系统会自动分割文本并生成问题，最终生成用于微调的数据集。 ## 技术栈 - **前端框架**: Next.js 14 (App Router) - **UI 框架**: Material-UI (MUI) - **数据存储**: fs 文件系统模拟数据库 - **开发语言**: JavaScript - **依赖管理**: pnpm ## 目录结构 \`\`\` easy-dataset/ ├── app/ # Next.js 应用目录 │ ├── api/ # API 路由 │ │ └── projects/ # 项目相关 API │ ├── projects/ # 项目相关页面 │ │ ├── [projectId]/ # 项目详情页面 │ └── page.js # 主页 ├── components/ # React 组件 │ ├── home/ # 主页相关组件 │ │ ├── HeroSection.js │ │ ├── ProjectList.js │ │ └── StatsCard.js │ ├── Navbar.js # 导航栏组件 │ └── CreateProjectDialog.js ├── lib/ # 工具库 │ └── db/ # 数据库模块 │ ├── base.js # 基础工具函数 │ ├── projects.js # 项目管理 │ ├── texts.js # 文本处理 │ ├── datasets.js # 数据集管理 │ └── index.js # 模块导出 ├── styles/ # 样式文件 │ └── home.js # 主页样式 └── local-db/ # 本地数据库目录 \`\`\` ## 核心模块设计 ### 1. 数据库模块 (`lib/db/`) #### base.js - 提供基础的文件操作功能 - 确保数据库目录存在 - 读写 JSON 文件的工具函数 #### projects.js - 项目的 CRUD 操作 - 项目配置管理 - 项目目录结构维护 #### texts.js - 文献处理功能 - 文本片段存储和检索 - 文件上传处理 #### datasets.js - 数据集生成和管理 - 问题列表管理 - 标签树管理 ### 2. 前端组件 (`components/`) #### Navbar.js - 顶部导航栏 - 项目切换 - 模型选择 - 主题切换 #### home/ 目录组件 - HeroSection.js: 主页顶部展示区 - ProjectList.js: 项目列表展示 - StatsCard.js: 数据统计展示 - CreateProjectDialog.js: 创建项目的对话框 ### 3. 页面路由 (`app/`) #### 主页 (`page.js`) - 项目列表展示 - 创建项目入口 - 数据统计展示 #### 项目详情页 (`projects/[projectId]/`) - text-split/: 文献处理页面 - questions/: 问题列表页面 - datasets/: 数据集页面 - settings/: 项目设置页面 #### API 路由 (`api/`) - projects/: 项目管理 API - texts/: 文本处理 API - questions/: 问题生成 API - datasets/: 数据集管理 API ## 数据流设计 ### 项目创建流程 1. 用户通过主页或导航栏创建新项目 2. 填写项目基本信息（名称、描述） 3. 系统创建项目目录和初始配置文件 4. 重定向到项目详情页 ### 文献处理流程 1. 用户上传 Markdown 文件 2. 系统保存原始文件到项目目录 3. 调用文本分割服务，生成片段和目录结构 4. 展示分割结果和提取的目录 ### 问题生成流程 1. 用户选择需要生成问题的文本片段 2. 系统调用大模型API生成问题 3. 保存问题到问题列表和标签树 ### 数据集生成流程 1. 用户选择需要生成答案的问题 2. 系统调用大模型API生成答案 3. 保存数据集结果 4. 提供导出功能 ``` ## /ARCHITECTURE.md # Easy DataSet 项目架构设计 ## 项目概述 Easy DataSet 是一个用于创建大模型微调数据集的应用程序。用户可以上传文本文件，系统会自动分割文本并生成问题，最终生成用于微调的数据集。 ## 技术栈 - **前端框架**: Next.js 14 (App Router) - **UI 框架**: Material-UI (MUI) - **数据存储**: fs 文件系统模拟数据库 - **开发语言**: JavaScript ## 目录结构 ``` easy-dataset/ ├── app/ # Next.js 应用目录 │ ├── api/ # API 路由 │ │ └── projects/ # 项目相关 API │ ├── projects/ # 项目相关页面 │ │ ├── [projectId]/ # 项目详情页面 │ └── page.js # 主页 ├── components/ # React 组件 │ ├── home/ # 主页相关组件 │ │ ├── HeroSection.js │ │ ├── ProjectList.js │ │ └── StatsCard.js │ ├── Navbar.js # 导航栏组件 │ └── CreateProjectDialog.js ├── lib/ # 工具库 │ └── db/ # 数据库模块 │ ├── base.js # 基础工具函数 │ ├── projects.js # 项目管理 │ ├── texts.js # 文本处理 │ ├── datasets.js # 数据集管理 │ └── index.js # 模块导出 ├── styles/ # 样式文件 │ └── home.js # 主页样式 └── local-db/ # 本地数据库目录 ``` ## 核心模块设计 ### 1. 数据库模块 (`lib/db/`) #### base.js - 提供基础的文件操作功能 - 确保数据库目录存在 - 读写 JSON 文件的工具函数 #### projects.js - 项目的 CRUD 操作 - 项目配置管理 - 项目目录结构维护 #### texts.js - 文献处理功能 - 文本片段存储和检索 - 文件上传处理 #### datasets.js - 数据集生成和管理 - 问题列表管理 - 标签树管理 ### 2. 前端组件 (`components/`) #### Navbar.js - 顶部导航栏 - 项目切换 - 模型选择 - 主题切换 #### home/ 目录组件 - HeroSection.js: 主页顶部展示区 - ProjectList.js: 项目列表展示 - StatsCard.js: 数据统计展示 - CreateProjectDialog.js: 创建项目的对话框 ### 3. 页面路由 (`app/`) #### 主页 (`page.js`) - 项目列表展示 - 创建项目入口 - 数据统计展示 #### 项目详情页 (`projects/[projectId]/`) - text-split/: 文献处理页面 - questions/: 问题列表页面 - datasets/: 数据集页面 - settings/: 项目设置页面 #### API 路由 (`api/`) - projects/: 项目管理 API - texts/: 文本处理 API - questions/: 问题生成 API - datasets/: 数据集管理 API ## 数据流设计 ### 项目创建流程 1. 用户通过主页或导航栏创建新项目 2. 填写项目基本信息（名称、描述） 3. 系统创建项目目录和初始配置文件 4. 重定向到项目详情页 ### 文献处理流程 1. 用户上传 Markdown 文件 2. 系统保存原始文件到项目目录 3. 调用文本分割服务，生成片段和目录结构 4. 展示分割结果和提取的目录 ### 问题生成流程 1. 用户选择需要生成问题的文本片段 2. 系统调用大模型API生成问题 3. 保存问题到问题列表和标签树 ### 数据集生成流程 1. 用户选择需要生成答案的问题 2. 系统调用大模型API生成答案 3. 保存数据集结果 4. 提供导出功能 ## 模型配置 支持多种大模型提供商配置： - Ollama - OpenAI - 硅基流动 - 深度求索 - 智谱AI 每个提供商支持配置： - API 地址 - API 密钥 - 模型名称 ## 未来扩展方向 1. 支持更多文件格式（PDF、DOC等） 2. 增加数据集质量评估功能 3. 添加数据集版本管理 4. 实现团队协作功能 5. 增加更多数据集导出格式 ## 国际化处理 ### 技术选型 - **国际化库**: i18next + react-i18next - **语言检测**: i18next-browser-languagedetector - **支持语言**: 英文(en)、简体中文(zh-CN) ### 目录结构 ``` easy-dataset/ ├── locales/ # 国际化资源目录 │ ├── en/ # 英文翻译 │ │ └── translation.json │ └── zh-CN/ # 中文翻译 │ └── translation.json ├── lib/ │ └── i18n.js # i18next 配置 ``` ## /Dockerfile ``` path="/Dockerfile" # 使用Node.js 18作为基础镜像 FROM docker.1ms.run/library/node:18 # 设置工作目录 WORKDIR /app RUN apt-get update && apt-get install -y \ build-essential \ libcairo2-dev \ libpango1.0-dev \ libjpeg-dev \ libgif-dev \ librsvg2-dev \ && rm -rf /var/lib/apt/lists/* # 复制package.json和package-lock.json COPY package.json package-lock.json* ./ # 安装依赖 RUN npm install # 复制所有文件 COPY . . # 构建应用 RUN npm run build # 暴露端口 EXPOSE 1717 # 启动应用 CMD ["npm", "start"] ``` ## /README.md
![](./public/imgs/bg2.png) version 1.2.3 Apache 2.0 License Next.js 14.1.0 React 18.2.0 Material UI 5.15.7 **A powerful tool for creating fine-tuning datasets for Large Language Models** [简体中文](./README.zh-CN.md) | [English](./README.md) [Features](#features) • [Getting Started](#getting-started) • [Usage](#usage) • [Documentation](https://rncg5jvpme.feishu.cn/docx/IRuad1eUIo8qLoxxwAGcZvqJnDb) • [Contributing](#contributing) • [License](#license)
If you like this project, please leave a Star ⭐️ for it. Or you can buy the author a cup of coffee => [Support the author](./public/imgs/aw.jpg) ❤️! ## Overview Easy Dataset is a specialized application designed to streamline the creation of fine-tuning datasets for Large Language Models (LLMs). It offers an intuitive interface for uploading domain-specific files, intelligently splitting content, generating questions, and producing high-quality training data for model fine-tuning. With Easy Dataset, you can transform your domain knowledge into structured datasets compatible with all OpenAI-format compatible LLM APIs, making the fine-tuning process accessible and efficient. ![](./public/imgs/en-arc.png) ## Features - **Intelligent Document Processing**: Upload Markdown files and automatically split them into meaningful segments - **Smart Question Generation**: Extract relevant questions from each text segment - **Answer Generation**: Generate comprehensive answers for each question using LLM APIs - **Flexible Editing**: Edit questions, answers, and datasets at any stage of the process - **Multiple Export Formats**: Export datasets in various formats (Alpaca, ShareGPT) and file types (JSON, JSONL) - **Wide Model Support**: Compatible with all LLM APIs that follow the OpenAI format - **User-Friendly Interface**: Intuitive UI designed for both technical and non-technical users - **Customizable System Prompts**: Add custom system prompts to guide model responses ## Getting Started ### Download Client
Windows MacOS Linux

Setup.exe
Intel
M
AppImage
### Using npm - Node.js 18.x or higher - pnpm (recommended) or npm 1. Clone the repository: ```bash git clone https://github.com/ConardLi/easy-dataset.git cd easy-dataset ``` 2. Install dependencies: ```bash npm install ``` 3. Start the development server: ```bash npm run build npm run start ``` ### Build with Local Dockerfile If you want to build the image yourself, you can use the Dockerfile in the project root directory: 1. Clone the repository: ```bash git clone https://github.com/ConardLi/easy-dataset.git cd easy-dataset ``` 2. Build the Docker image: ```bash docker build -t easy-dataset . ``` 3. Run the container: ```bash docker run -d -p 1717:1717 -v {YOUR_LOCAL_DB_PATH}:/app/local-db --name easy-dataset easy-dataset ``` **Note:** Replace `{YOUR_LOCAL_DB_PATH}` with the actual path where you want to store the local database. 4. Open your browser and navigate to `http://localhost:1717` ## Usage ### Creating a Project
1. Click the "Create Project" button on the home page 2. Enter a project name and description 3. Configure your preferred LLM API settings ### Processing Documents
1. Upload your Markdown files in the "Text Split" section 2. Review the automatically split text segments 3. Adjust the segmentation if needed ### Generating Questions
1. Navigate to the "Questions" section 2. Select text segments to generate questions from 3. Review and edit the generated questions 4. Organize questions using the tag tree ### Creating Datasets
1. Go to the "Datasets" section 2. Select questions to include in your dataset 3. Generate answers using your configured LLM 4. Review and edit the generated answers ### Exporting Datasets
1. Click the "Export" button in the Datasets section 2. Select your preferred format (Alpaca or ShareGPT) 3. Choose file format (JSON or JSONL) 4. Add custom system prompts if needed 5. Export your dataset ## Project Structure ``` easy-dataset/ ├── app/ # Next.js application directory │ ├── api/ # API routes │ │ ├── llm/ # LLM API integration │ │ │ ├── ollama/ # Ollama API integration │ │ │ └── openai/ # OpenAI API integration │ │ ├── projects/ # Project management APIs │ │ │ ├── [projectId]/ # Project-specific operations │ │ │ │ ├── chunks/ # Text chunk operations │ │ │ │ ├── datasets/ # Dataset generation and management │ │ │ │ │ └── optimize/ # Dataset optimization API │ │ │ │ ├── generate-questions/ # Batch question generation │ │ │ │ ├── questions/ # Question management │ │ │ │ └── split/ # Text splitting operations │ │ │ └── user/ # User-specific project operations │ ├── projects/ # Front-end project pages │ │ └── [projectId]/ # Project-specific pages │ │ ├── datasets/ # Dataset management UI │ │ ├── questions/ # Question management UI │ │ ├── settings/ # Project settings UI │ │ └── text-split/ # Text processing UI │ └── page.js # Home page ├── components/ # React components │ ├── datasets/ # Dataset-related components │ ├── home/ # Home page components │ ├── projects/ # Project management components │ ├── questions/ # Question management components │ └── text-split/ # Text processing components ├── lib/ # Core libraries and utilities │ ├── db/ # Database operations │ ├── i18n/ # Internationalization │ ├── llm/ # LLM integration │ │ ├── common/ # Common LLM utilities │ │ ├── core/ # Core LLM client │ │ └── prompts/ # Prompt templates │ │ ├── answer.js # Answer generation prompts (Chinese) │ │ ├── answerEn.js # Answer generation prompts (English) │ │ ├── question.js # Question generation prompts (Chinese) │ │ ├── questionEn.js # Question generation prompts (English) │ │ └── ... other prompts │ └── text-splitter/ # Text splitting utilities ├── locales/ # Internationalization resources │ ├── en/ # English translations │ └── zh-CN/ # Chinese translations ├── public/ # Static assets │ └── imgs/ # Image resources └── local-db/ # Local file-based database └── projects/ # Project data storage ``` ## Documentation For detailed documentation on all features and APIs, please visit our [Documentation Site](https://rncg5jvpme.feishu.cn/docx/IRuad1eUIo8qLoxxwAGcZvqJnDb?302from=wiki). ## Contributing We welcome contributions from the community! If you'd like to contribute to Easy Dataset, please follow these steps: 1. Fork the repository 2. Create a new branch (`git checkout -b feature/amazing-feature`) 3. Make your changes 4. Commit your changes (`git commit -m 'Add some amazing feature'`) 5. Push to the branch (`git push origin feature/amazing-feature`) 6. Open a Pull Request Please make sure to update tests as appropriate and adhere to the existing coding style. ## License This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details. ## Star History [![Star History Chart](https://api.star-history.com/svg?repos=ConardLi/easy-dataset&type=Date)](https://www.star-history.com/#ConardLi/easy-dataset&Date)
Built with ❤️ by ConardLi • Follow me：WeChat｜Bilibili｜Juijin｜Zhihu
## /README.zh-CN.md
![](./public//imgs/bg2.png) 版本 1.2.3 Apache 2.0 许可证 Next.js 14.1.0 React 18.2.0 Material UI 5.15.7 **一个强大的大型语言模型微调数据集创建工具** [简体中文](./README.zh-CN.md) | [English](./README.md) [功能特点](#功能特点) • [快速开始](#本地运行) • [使用文档](https://rncg5jvpme.feishu.cn/wiki/NT7aw7rBfi8HwukHaUTcvrQIn6f) • [贡献](#贡献) • [许可证](#许可证)
如果喜欢本项目，请给本项目留下 Star⭐️，或者请作者喝杯咖啡呀 => [打赏作者](./public/imgs/aw.jpg) ❤️！ ## 概述 Easy Dataset 是一个专为创建大型语言模型（LLM）微调数据集而设计的应用程序。它提供了直观的界面，用于上传特定领域的文件，智能分割内容，生成问题，并为模型微调生成高质量的训练数据。 通过 Easy Dataset，您可以将领域知识转化为结构化数据集，兼容所有遵循 OpenAI 格式的 LLM API，使微调过程变得简单高效。 ![](./public/imgs/cn-arc.png) ## 功能特点 * **智能文档处理**：上传 Markdown 文件并自动将其分割为有意义的片段 * **智能问题生成**：从每个文本片段中提取相关问题 * **答案生成**：使用 LLM API 为每个问题生成全面的答案 * **灵活编辑**：在流程的任何阶段编辑问题、答案和数据集 * **多种导出格式**：以各种格式（Alpaca、ShareGPT）和文件类型（JSON、JSONL）导出数据集 * **广泛的模型支持**：兼容所有遵循 OpenAI 格式的 LLM API * **用户友好界面**：为技术和非技术用户设计的直观 UI * **自定义系统提示**：添加自定义系统提示以引导模型响应 ## 本地运行 ### 下载客户端
Windows MacOS Linux

Setup.exe
Intel
M
AppImage
### 使用 NPM 安装 1. 克隆仓库： ```bash git clone https://github.com/ConardLi/easy-dataset.git cd easy-dataset ``` 2. 安装依赖： ```bash npm install ``` 3. 启动开发服务器： ```bash npm run build npm run start ``` 4. 打开浏览器并访问 `http://localhost:1717` ### 使用本地 Dockerfile 构建 如果你想自行构建镜像，可以使用项目根目录中的 Dockerfile： 1. 克隆仓库： ```bash git clone https://github.com/ConardLi/easy-dataset.git cd easy-dataset ``` 2. 构建 Docker 镜像： ```bash docker build -t easy-dataset . ``` 3. 运行容器： ```bash docker run -d -p 1717:1717 -v {YOUR_LOCAL_DB_PATH}:/app/local-db --name easy-dataset easy-dataset ``` **注意：** 请将 `{YOUR_LOCAL_DB_PATH}` 替换为你希望存储本地数据库的实际路径。 4. 打开浏览器，访问 `http://localhost:1717` ## 使用方法 ### 创建项目
1. 在首页点击"创建项目"按钮； 2. 输入项目名称和描述； 3. 配置您首选的 LLM API 设置 ### 处理文档
1. 在"文本分割"部分上传您的 Markdown 文件； 2. 查看自动分割的文本片段； 3. 根据需要调整分段 ### 生成问题
1. 导航到"问题"部分； 2. 选择要从中生成问题的文本片段； 3. 查看并编辑生成的问题； 4. 使用标签树组织问题 ### 创建数据集
1. 转到"数据集"部分； 2. 选择要包含在数据集中的问题； 3. 使用配置的 LLM 生成答案； 4. 查看并编辑生成的答案 ### 导出数据集
1. 在数据集部分点击"导出"按钮； 2. 选择您喜欢的格式（Alpaca 或 ShareGPT）； 3. 选择文件格式（JSON 或 JSONL）； 4. 根据需要添加自定义系统提示；5. 导出您的数据集 ## 项目结构 ``` easy-dataset/ ├── app/ # Next.js 应用目录 │ ├── api/ # API 路由 │ │ ├── llm/ # LLM API 集成 │ │ │ ├── ollama/ # Ollama API 集成 │ │ │ └── openai/ # OpenAI API 集成 │ │ ├── projects/ # 项目管理 API │ │ │ ├── [projectId]/ # 项目特定操作 │ │ │ │ ├── chunks/ # 文本块操作 │ │ │ │ ├── datasets/ # 数据集生成和管理 │ │ │ │ ├── generate-questions/ # 批量问题生成 │ │ │ │ ├── questions/ # 问题管理 │ │ │ │ └── split/ # 文本分割操作 │ │ │ └── user/ # 用户特定项目操作 │ ├── projects/ # 前端项目页面 │ │ └── [projectId]/ # 项目特定页面 │ │ ├── datasets/ # 数据集管理 UI │ │ ├── questions/ # 问题管理 UI │ │ ├── settings/ # 项目设置 UI │ │ └── text-split/ # 文本处理 UI │ └── page.js # 主页 ├── components/ # React 组件 │ ├── datasets/ # 数据集相关组件 │ ├── home/ # 主页组件 │ ├── projects/ # 项目管理组件 │ ├── questions/ # 问题管理组件 │ └── text-split/ # 文本处理组件 ├── lib/ # 核心库和工具 │ ├── db/ # 数据库操作 │ ├── i18n/ # 国际化 │ ├── llm/ # LLM 集成 │ │ ├── common/ # 通用 LLM 工具 │ │ ├── core/ # 核心 LLM 客户端 │ │ └── prompts/ # 提示词模板 │ │ ├── answer.js # 答案生成提示词（中文） │ │ ├── answerEn.js # 答案生成提示词（英文） │ │ ├── question.js # 问题生成提示词（中文） │ │ ├── questionEn.js # 问题生成提示词（英文） │ │ └── ... 其他提示词 │ └── text-splitter/ # 文本分割工具 ├── locales/ # 国际化资源 │ ├── en/ # 英文翻译 │ └── zh-CN/ # 中文翻译 ├── public/ # 静态资源 │ └── imgs/ # 图片资源 └── local-db/ # 本地文件数据库 └── projects/ # 项目数据存储 ``` ## 文档 - 查看本项目的演示视频：[Easy Dataset 演示视频](https://www.bilibili.com/video/BV1y8QpYGE57/) - 有关所有功能和 API 的详细文档，请访问我们的[文档站点](https://rncg5jvpme.feishu.cn/wiki/NT7aw7rBfi8HwukHaUTcvrQIn6f)。 ## 贡献 我们欢迎社区的贡献！如果您想为 Easy Dataset 做出贡献，请按照以下步骤操作： 1. Fork 仓库 2. 创建新分支（`git checkout -b feature/amazing-feature`） 3. 进行更改 4. 提交更改（`git commit -m '添加一些惊人的功能'`） 5. 推送到分支（`git push origin feature/amazing-feature`） 6. 打开 Pull Request 请确保适当更新测试并遵守现有的编码风格。 ## 许可证 本项目采用 Apache License 2.0 许可证 - 有关详细信息，请参阅 [LICENSE](LICENSE) 文件。 ## Star History [![Star History Chart](https://api.star-history.com/svg?repos=ConardLi/easy-dataset&type=Date)](https://www.star-history.com/#ConardLi/easy-dataset&Date)
由 ConardLi 用 ❤️ 构建 • 关注我：公众号｜B站｜掘金｜知乎
## /app/api/check-update/route.js ```js path="/app/api/check-update/route.js" import { NextResponse } from 'next/server'; import path from 'path'; import fs from 'fs'; // 获取当前版本 function getCurrentVersion() { try { const packageJsonPath = path.join(process.cwd(), 'package.json'); const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8')); return packageJson.version; } catch (error) { console.error('读取版本信息失败:', error); return '1.0.0'; } } // 从 GitHub 获取最新版本 async function getLatestVersion() { try { const owner = 'ConardLi'; const repo = 'easy-dataset'; const response = await fetch(`https://api.github.com/repos/${owner}/${repo}/releases/latest`); if (!response.ok) { throw new Error(`GitHub API 请求失败: ${response.status}`); } const data = await response.json(); return data.tag_name.replace('v', ''); } catch (error) { console.error('获取最新版本失败:', error); return null; } } // 检查是否有更新 export async function GET() { try { const currentVersion = getCurrentVersion(); const latestVersion = await getLatestVersion(); if (!latestVersion) { return NextResponse.json({ hasUpdate: false, currentVersion, latestVersion: null, error: '获取最新版本失败' }); } // 简单的版本比较 const hasUpdate = compareVersions(latestVersion, currentVersion) > 0; return NextResponse.json({ hasUpdate, currentVersion, latestVersion, releaseUrl: hasUpdate ? `https://github.com/ConardLi/easy-dataset/releases/tag/v${latestVersion}` : null }); } catch (error) { console.error('检查更新失败:', error); } } // 简单的版本比较函数 function compareVersions(a, b) { const partsA = a.split('.').map(Number); const partsB = b.split('.').map(Number); for (let i = 0; i < Math.max(partsA.length, partsB.length); i++) { const numA = i < partsA.length ? partsA[i] : 0; const numB = i < partsB.length ? partsB[i] : 0; if (numA > numB) return 1; if (numA < numB) return -1; } return 0; } ``` ## /app/api/llm/ollama/models/route.js ```js path="/app/api/llm/ollama/models/route.js" import {NextResponse} from 'next/server'; const OllamaClient = require('@/lib/llm/core/providers/ollama'); // 设置为强制动态路由，防止静态生成 export const dynamic = 'force-dynamic'; export async function GET(request) { try { // 从查询参数中获取 host 和 port const {searchParams} = new URL(request.url); const host = searchParams.get('host') || '127.0.0.1'; const port = searchParams.get('port') || '11434'; // 创建 Ollama API 实例 const ollama = new OllamaClient({ endpoint: `http://${host}:${port}/api` }); // 获取模型列表 const models = await ollama.getModels(); return NextResponse.json(models); } catch (error) { // console.error('fetch Ollama models error:', error); return NextResponse.json({error: 'fetch Models failed'}, {status: 500}); } } ``` ## /app/api/projects/[projectId]/chunks/[chunkId]/questions/route.js ```js path="/app/api/projects/[projectId]/chunks/[chunkId]/questions/route.js" import { NextResponse } from 'next/server'; import { getTextChunk } from '@/lib/db/texts'; import LLMClient from '@/lib/llm/core/index'; import getQuestionPrompt from '@/lib/llm/prompts/question'; import getQuestionEnPrompt from '@/lib/llm/prompts/questionEn'; import getAddLabelPrompt from '@/lib/llm/prompts/addLabel'; import getAddLabelEnPrompt from '@/lib/llm/prompts/addLabelEn'; import { addQuestionsForChunk, getQuestionsForChunk } from '@/lib/db/questions'; import { extractJsonFromLLMOutput } from '@/lib/llm/common/util'; import { getTaskConfig, getProject } from '@/lib/db/projects'; import { getTags } from '@/lib/db/tags'; import logger from '@/lib/util/logger'; // 为指定文本块生成问题 export async function POST(request, { params }) { try { const { projectId, chunkId: c } = params; // 验证项目ID和文本块ID if (!projectId || !c) { return NextResponse.json({ error: 'Project ID or text block ID cannot be empty' }, { status: 400 }); } const chunkId = decodeURIComponent(c); // 获取请求体 const { model, language = '中文', number } = await request.json(); if (!model) { return NextResponse.json({ error: 'Model cannot be empty' }, { status: 400 }); } // 获取文本块内容 const chunk = await getTextChunk(projectId, chunkId); if (!chunk) { return NextResponse.json({ error: 'Text block does not exist' }, { status: 404 }); } // 获取项目 task-config 信息 const taskConfig = await getTaskConfig(projectId); const config = await getProject(projectId); const { questionGenerationLength } = taskConfig; const { globalPrompt, questionPrompt } = config; // 创建LLM客户端 const llmClient = new LLMClient({ provider: model.provider, endpoint: model.endpoint, apiKey: model.apiKey, model: model.name, temperature: model.temperature, maxTokens: model.maxTokens }); // 生成问题的数量，如果未指定，则根据文本长度自动计算 const questionNumber = number || Math.floor(chunk.content.length / questionGenerationLength); // 根据语言选择相应的提示词函数 const promptFunc = language === 'en' ? getQuestionEnPrompt : getQuestionPrompt; // 生成问题 const prompt = promptFunc({ text: chunk.content, number: questionNumber, language, globalPrompt, questionPrompt }); const response = await llmClient.getResponse(prompt); // 从LLM输出中提取JSON格式的问题列表 const questions = extractJsonFromLLMOutput(response); console.log(projectId, chunkId, 'Questions：', questions); if (!questions || !Array.isArray(questions)) { return NextResponse.json({ error: 'Failed to generate questions' }, { status: 500 }); } // 打标签 const tags = await getTags(projectId); // 根据语言选择相应的标签提示词函数 const labelPromptFunc = language === 'en' ? getAddLabelEnPrompt : getAddLabelPrompt; const labelPrompt = labelPromptFunc(JSON.stringify(tags), JSON.stringify(questions)); const labelResponse = await llmClient.getResponse(labelPrompt); // 从LLM输出中提取JSON格式的问题列表 const labelQuestions = extractJsonFromLLMOutput(labelResponse); console.log(projectId, chunkId, 'Label Questions：', labelQuestions); // 保存问题到数据库 await addQuestionsForChunk(projectId, chunkId, labelQuestions); // 返回生成的问题 return NextResponse.json({ chunkId, labelQuestions, total: labelQuestions.length }); } catch (error) { logger.error('Error generating questions:', error); return NextResponse.json({ error: error.message || 'Error generating questions' }, { status: 500 }); } } // 获取指定文本块的问题 export async function GET(request, { params }) { try { const { projectId, chunkId } = params; // 验证项目ID和文本块ID if (!projectId || !chunkId) { return NextResponse.json({ error: 'The item ID or text block ID cannot be empty' }, { status: 400 }); } // 获取文本块的问题 const questions = await getQuestionsForChunk(projectId, chunkId); // 返回问题列表 return NextResponse.json({ chunkId, questions, total: questions.length }); } catch (error) { console.error('Error getting questions:', error); return NextResponse.json({ error: error.message || 'Error getting questions' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/chunks/[chunkId]/route.js ```js path="/app/api/projects/[projectId]/chunks/[chunkId]/route.js" import { NextResponse } from 'next/server'; import { getChunkContent } from '@/lib/text-splitter'; import fs from 'fs/promises'; import path from 'path'; import { getProjectRoot } from '@/lib/db/base'; // 获取文本块内容 export async function GET(request, { params }) { try { const { projectId, chunkId: c } = params; const chunkId = decodeURIComponent(c); // 验证参数 if (!projectId) { return NextResponse.json({ error: 'Project ID cannot be empty' }, { status: 400 }); } if (!chunkId) { return NextResponse.json({ error: 'Text block ID cannot be empty' }, { status: 400 }); } // 获取文本块内容 const chunk = await getChunkContent(projectId, chunkId); return NextResponse.json(chunk); } catch (error) { console.error('Failed to get text block content:', error); return NextResponse.json({ error: error.message || 'Failed to get text block content' }, { status: 500 }); } } // 删除文本块 export async function DELETE(request, { params }) { try { const { projectId, chunkId: c } = params; const chunkId = decodeURIComponent(c); // 验证参数 if (!projectId) { return NextResponse.json({ error: 'Project ID cannot be empty' }, { status: 400 }); } if (!chunkId) { return NextResponse.json({ error: 'Text block ID cannot be empty' }, { status: 400 }); } // 获取文本块路径 const projectRoot = await getProjectRoot(); const chunkPath = path.join(projectRoot, projectId, 'chunks', `${chunkId}.txt`); // 检查文件是否存在 try { await fs.access(chunkPath); } catch (error) { return NextResponse.json({ error: 'Text block does not exist' }, { status: 404 }); } // 删除文件 await fs.unlink(chunkPath); return NextResponse.json({ message: 'Text block deleted successfully' }); } catch (error) { console.error('Failed to delete text block:', error); return NextResponse.json({ error: error.message || 'Failed to delete text block' }, { status: 500 }); } } // 编辑文本块内容 export async function PATCH(request, { params }) { try { const { projectId, chunkId: c } = params; const chunkId = decodeURIComponent(c); // 验证参数 if (!projectId) { return NextResponse.json({ error: '项目ID不能为空' }, { status: 400 }); } if (!chunkId) { return NextResponse.json({ error: '文本块ID不能为空' }, { status: 400 }); } // 解析请求体获取新内容 const requestData = await request.json(); const { content } = requestData; if (!content) { return NextResponse.json({ error: '内容不能为空' }, { status: 400 }); } // 获取文本块路径 const projectRoot = await getProjectRoot(); const chunkPath = path.join(projectRoot, projectId, 'chunks', `${chunkId}.txt`); // 检查文件是否存在 try { await fs.access(chunkPath); } catch (error) { return NextResponse.json({ error: '文本块不存在' }, { status: 404 }); } // 更新文件内容 await fs.writeFile(chunkPath, content, 'utf-8'); // 获取更新后的文本块内容 const updatedChunk = await getChunkContent(projectId, chunkId); return NextResponse.json(updatedChunk); } catch (error) { console.error('编辑文本块失败:', error); return NextResponse.json({ error: error.message || '编辑文本块失败' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/config/route.js ```js path="/app/api/projects/[projectId]/config/route.js" import { NextResponse } from 'next/server'; import { getProject, updateProject } from '@/lib/db/projects'; // 获取项目配置 export async function GET(request, { params }) { try { const projectId = params.projectId; const config = await getProject(projectId); return NextResponse.json(config); } catch (error) { console.error('获取项目配置失败:', error); return NextResponse.json({ error: error.message }, { status: 500 }); } } // 更新项目配置 export async function PUT(request, { params }) { try { const projectId = params.projectId; const newConfig = await request.json(); const currentConfig = await getProject(projectId); // 只更新 prompts 部分 const updatedConfig = { ...currentConfig, ...newConfig.prompts }; const config = await updateProject(projectId, updatedConfig); return NextResponse.json(config); } catch (error) { console.error('更新项目配置失败:', error); return NextResponse.json({ error: error.message }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/datasets/optimize/route.js ```js path="/app/api/projects/[projectId]/datasets/optimize/route.js" import { NextResponse } from 'next/server'; import { getDataset, updateDataset } from '@/lib/db/datasets'; import LLMClient from '@/lib/llm/core/index'; import getNewAnswerPrompt from '@/lib/llm/prompts/newAnswer'; import getNewAnswerEnPrompt from '@/lib/llm/prompts/newAnswerEn'; import { extractJsonFromLLMOutput } from '@/lib/llm/common/util'; // 优化数据集答案 export async function POST(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'Project ID cannot be empty' }, { status: 400 }); } // 获取请求体 const { datasetId, model, advice, language } = await request.json(); if (!datasetId) { return NextResponse.json({ error: 'Dataset ID cannot be empty' }, { status: 400 }); } if (!model) { return NextResponse.json({ error: 'Model cannot be empty' }, { status: 400 }); } if (!advice) { return NextResponse.json({ error: 'Please provide optimization suggestions' }, { status: 400 }); } // 获取数据集内容 const dataset = await getDataset(projectId, datasetId); if (!dataset) { return NextResponse.json({ error: 'Dataset does not exist' }, { status: 404 }); } // 创建LLM客户端 const llmClient = new LLMClient({ provider: model.provider, endpoint: model.endpoint, apiKey: model.apiKey, model: model.name, temperature: model.temperature, maxTokens: model.maxTokens }); // 生成优化后的答案和思维链 const prompt = language === 'en' ? getNewAnswerEnPrompt(dataset.question, dataset.answer || '', dataset.cot || '', advice) : getNewAnswerPrompt(dataset.question, dataset.answer || '', dataset.cot || '', advice); const response = await llmClient.getResponse(prompt); // 从LLM输出中提取JSON格式的优化结果 const optimizedResult = extractJsonFromLLMOutput(response); if (!optimizedResult || !optimizedResult.answer) { return NextResponse.json({ error: 'Failed to optimize answer, please try again' }, { status: 500 }); } // 更新数据集 const updatedDataset = { ...dataset, answer: optimizedResult.answer, cot: optimizedResult.cot || dataset.cot }; await updateDataset(projectId, datasetId, updatedDataset); // 返回优化后的数据集 return NextResponse.json({ success: true, dataset: updatedDataset }); } catch (error) { console.error('Failed to optimize answer:', error); return NextResponse.json({ error: error.message || 'Failed to optimize answer' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/datasets/route.js ```js path="/app/api/projects/[projectId]/datasets/route.js" import { NextResponse } from 'next/server'; import { getTextChunk } from '@/lib/db/texts'; import { getQuestionsForChunk } from '@/lib/db/questions'; import { getDatasets, saveDatasets, updateDataset } from '@/lib/db/datasets'; import { getProject } from '@/lib/db/projects'; import getAnswerPrompt from '@/lib/llm/prompts/answer'; import getAnswerEnPrompt from '@/lib/llm/prompts/answerEn'; import getOptimizeCotPrompt from '@/lib/llm/prompts/optimizeCot'; import getOptimizeCotEnPrompt from '@/lib/llm/prompts/optimizeCotEn'; const LLMClient = require('@/lib/llm/core'); async function optimizeCot(originalQuestion, answer, originalCot, language, llmClient, id, projectId) { const prompt = language === 'en' ? getOptimizeCotEnPrompt(originalQuestion, answer, originalCot) : getOptimizeCotPrompt(originalQuestion, answer, originalCot); const { answer: optimizedAnswer } = await llmClient.getResponseWithCOT(prompt); await updateDataset(projectId, id, { cot: optimizedAnswer.replace('优化后的思维链', '') }); console.log(originalQuestion, id, 'Successfully optimized thought process'); } /** * 生成数据集（为单个问题生成答案） */ export async function POST(request, { params }) { try { const { projectId } = params; const { questionId, chunkId, model, language } = await request.json(); // 验证参数 if (!projectId || !questionId || !chunkId || !model) { return NextResponse.json( { error: '缺少必要参数' }, { status: 400 } ); } // 获取文本块内容 const chunk = await getTextChunk(projectId, chunkId); if (!chunk) { return NextResponse.json( { error: 'Text block does not exist' }, { status: 404 } ); } // 获取问题 const questions = await getQuestionsForChunk(projectId, chunkId); const question = questions.find(q => q.question === questionId); if (!question) { return NextResponse.json( { error: 'Question not found' }, { status: 404 } ); } // 获取项目配置 const project = await getProject(projectId); const { globalPrompt, answerPrompt } = project; // 创建LLM客户端 const llmClient = new LLMClient({ provider: model.provider, endpoint: model.endpoint, apiKey: model.apiKey, model: model.name, temperature: model.temperature, maxTokens: model.maxTokens }); const promptFuc = language === 'en' ? getAnswerEnPrompt : getAnswerPrompt; // 生成答案的提示词 const prompt = promptFuc({ text: chunk.content, question: question.question, globalPrompt, answerPrompt }); // 调用大模型生成答案 const { answer, cot } = await llmClient.getResponseWithCOT(prompt); // 获取现有数据集 const datasets = await getDatasets(projectId); const datasetId = `${Date.now()}-${Math.random().toString(36).substring(2, 9)}`; // 创建新的数据集项 const datasetItem = { id: datasetId, question: question.question, answer: answer, chunkId: chunkId, model: model.name, createdAt: new Date().toISOString(), questionLabel: question.label || null }; if (cot) { // 为了性能考虑，这里异步优化 optimizeCot(question.question, answer, cot, language, llmClient, datasetId, projectId); } // 添加到数据集 datasets.push(datasetItem); await saveDatasets(projectId, datasets); console.log(datasets.length, 'Successfully generated dataset', question.question); return NextResponse.json({ success: true, dataset: datasetItem }); } catch (error) { console.error('Failed to generate dataset:', error); return NextResponse.json( { error: error.message || 'Failed to generate dataset' }, { status: 500 } ); } } /** * 获取项目的所有数据集 */ export async function GET(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json( { error: '项目ID不能为空' }, { status: 400 } ); } // 获取数据集 const datasets = await getDatasets(projectId); return NextResponse.json(datasets); } catch (error) { console.error('获取数据集失败:', error); return NextResponse.json( { error: error.message || '获取数据集失败' }, { status: 500 } ); } } /** * 删除数据集 */ export async function DELETE(request, { params }) { try { const { projectId } = params; const { searchParams } = new URL(request.url); const datasetId = searchParams.get('id'); // 验证参数 if (!projectId) { return NextResponse.json( { error: 'Project ID cannot be empty' }, { status: 400 } ); } if (!datasetId) { return NextResponse.json( { error: 'Dataset ID cannot be empty' }, { status: 400 } ); } // 获取所有数据集 const datasets = await getDatasets(projectId); // 找到要删除的数据集索引 const datasetIndex = datasets.findIndex(dataset => dataset.id === datasetId); if (datasetIndex === -1) { return NextResponse.json( { error: 'Dataset does not exist' }, { status: 404 } ); } // 删除数据集 datasets.splice(datasetIndex, 1); // 保存更新后的数据集列表 await saveDatasets(projectId, datasets); return NextResponse.json({ success: true, message: 'Dataset deleted successfully' }); } catch (error) { console.error('Failed to delete dataset:', error); return NextResponse.json( { error: error.message || 'Failed to delete dataset' }, { status: 500 } ); } } /** * 编辑数据集 */ export async function PATCH(request, { params }) { try { const { projectId } = params; const { searchParams } = new URL(request.url); const datasetId = searchParams.get('id'); const { answer, cot, confirmed } = await request.json(); // 验证参数 if (!projectId) { return NextResponse.json( { error: 'Project ID cannot be empty' }, { status: 400 } ); } if (!datasetId) { return NextResponse.json( { error: 'Dataset ID cannot be empty' }, { status: 400 } ); } // 获取所有数据集 const datasets = await getDatasets(projectId); // 找到要编辑的数据集 const datasetIndex = datasets.findIndex(dataset => dataset.id === datasetId); if (datasetIndex === -1) { return NextResponse.json( { error: 'Dataset does not exist' }, { status: 404 } ); } // 更新数据集 const dataset = datasets[datasetIndex]; if (answer !== undefined) dataset.answer = answer; if (cot !== undefined) dataset.cot = cot; if (confirmed !== undefined) dataset.confirmed = confirmed; // 保存更新后的数据集列表 await saveDatasets(projectId, datasets); return NextResponse.json({ success: true, message: 'Dataset updated successfully', dataset: dataset }); } catch (error) { console.error('Failed to update dataset:', error); return NextResponse.json( { error: error.message || 'Failed to update dataset' }, { status: 500 } ); } } ``` ## /app/api/projects/[projectId]/files/route.js ```js path="/app/api/projects/[projectId]/files/route.js" import { NextResponse } from 'next/server'; import { getFiles, deleteFile } from '@/lib/db/texts'; import { getProject, updateProject } from '@/lib/db/projects'; import path from 'path'; import { getProjectRoot, ensureDir } from '@/lib/db/base'; import { promises as fs } from 'fs'; // Replace the deprecated config export with the new export syntax export const dynamic = 'force-dynamic'; // This tells Next.js not to parse the request body automatically export const bodyParser = false; // 获取项目文件列表 export async function GET(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取文件列表 const files = await getFiles(projectId); return NextResponse.json({ files }); } catch (error) { console.error('Error obtaining file list:', error); return NextResponse.json({ error: error.message || 'Error obtaining file list' }, { status: 500 }); } } // 删除文件 export async function DELETE(request, { params }) { try { const { projectId } = params; const { searchParams } = new URL(request.url); const fileName = searchParams.get('fileName'); // 验证项目ID和文件名 if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } if (!fileName) { return NextResponse.json({ error: 'The file name cannot be empty' }, { status: 400 }); } // 获取项目信息 const project = await getProject(projectId); if (!project) { return NextResponse.json({ error: 'The project does not exist' }, { status: 404 }); } // 删除文件及相关数据 const result = await deleteFile(projectId, fileName); // 更新项目配置，移除已删除的文件 const uploadedFiles = project.uploadedFiles || []; const updatedFiles = uploadedFiles.filter(f => f !== fileName); await updateProject(projectId, { ...project, uploadedFiles: updatedFiles }); return NextResponse.json({ message: 'File deleted successfully', fileName }); } catch (error) { console.error('Error deleting file:', error); return NextResponse.json({ error: error.message || 'Error deleting file' }, { status: 500 }); } } // 上传文件 export async function POST(request, { params }) { console.log('File upload request processing, parameters:', params); const { projectId } = params; // 验证项目ID if (!projectId) { console.log('The project ID cannot be empty, returning 400 error'); return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取项目信息 const project = await getProject(projectId); if (!project) { console.log('The project does not exist, returning 404 error'); return NextResponse.json({ error: 'The project does not exist' }, { status: 404 }); } console.log('Project information retrieved successfully:', project.name || project.id); try { console.log('Try using alternate methods for file upload...'); // 检查请求头中是否包含文件名 const encodedFileName = request.headers.get('x-file-name'); const fileName = encodedFileName ? decodeURIComponent(encodedFileName) : null; console.log('Get file name from request header:', fileName); if (!fileName) { console.log('The request header does not contain a file name'); return NextResponse.json( { error: 'The request header does not contain a file name (x-file-name)' }, { status: 400 } ); } // 检查文件类型 if (!fileName.endsWith('.md')&&!fileName.endsWith('.pdf')) { return NextResponse.json({ error: 'Only Markdown files are supported' }, { status: 400 }); } // 直接从请求体中读取二进制数据 const fileBuffer = Buffer.from(await request.arrayBuffer()); // 保存文件 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); const filesDir = path.join(projectPath, 'files'); await ensureDir(filesDir); const filePath = path.join(filesDir, fileName); await fs.writeFile(filePath, fileBuffer); // 更新项目配置，添加上传的文件记录 const uploadedFiles = project.uploadedFiles || []; if (!uploadedFiles.includes(fileName)) { uploadedFiles.push(fileName); // 更新项目配置 await updateProject(projectId, { ...project, uploadedFiles }); } console.log('The file upload process is complete, and a successful response is returned'); return NextResponse.json({ message: 'File uploaded successfully', fileName, uploadedFiles, filePath }); } catch (error) { console.error('Error processing file upload:', error); console.error('Error stack:', error.stack); return NextResponse.json( { error: 'File upload failed: ' + (error.message || 'Unknown error') }, { status: 500 } ); } } ``` ## /app/api/projects/[projectId]/generate-questions/route.js ```js path="/app/api/projects/[projectId]/generate-questions/route.js" import { NextResponse } from 'next/server'; import { getProjectChunks } from '@/lib/text-splitter'; import { getTextChunk } from '@/lib/db/texts'; import LLMClient from '@/lib/llm/core/index'; import getQuestionPrompt from '@/lib/llm/prompts/question'; import getQuestionEnPrompt from '@/lib/llm/prompts/questionEn'; import { addQuestionsForChunk } from '@/lib/db/questions'; import { getTaskConfig } from '@/lib/db/projects'; const { extractJsonFromLLMOutput } = require('@/lib/llm/common/util'); // 批量生成问题 export async function POST(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取请求体 const { model, chunkIds, language = '中文' } = await request.json(); if (!model) { return NextResponse.json({ error: 'The model cannot be empty' }, { status: 400 }); } // 如果没有指定文本块ID，则获取所有文本块 let chunks = []; if (!chunkIds || chunkIds.length === 0) { const result = await getProjectChunks(projectId); chunks = result.chunks || []; } else { // 获取指定的文本块 chunks = await Promise.all( chunkIds.map(async chunkId => { const chunk = await getTextChunk(projectId, chunkId); if (chunk) { return { id: chunk.id, content: chunk.content, length: chunk.content.length }; } return null; }) ); chunks = chunks.filter(Boolean); // 过滤掉不存在的文本块 } if (chunks.length === 0) { return NextResponse.json({ error: 'No valid text blocks found' }, { status: 404 }); } const llmClient = new LLMClient({ provider: model.provider, endpoint: model.endpoint, apiKey: model.apiKey, model: model.name, temperature: model.temperature, maxTokens: model.maxTokens }); const results = []; const errors = []; // 获取项目 task-config 信息 const taskConfig = await getTaskConfig(projectId); const { questionGenerationLength } = taskConfig; for (const chunk of chunks) { try { // 根据文本长度自动计算问题数量 const questionNumber = Math.floor(chunk.length / questionGenerationLength); // 根据语言选择相应的提示词函数 const promptFunc = language === 'en' ? getQuestionEnPrompt : getQuestionPrompt; // 生成问题 const prompt = promptFunc(chunk.content, questionNumber, language); const response = await llmClient.getResponse(prompt); // 从LLM输出中提取JSON格式的问题列表 const questions = extractJsonFromLLMOutput(response); if (questions && Array.isArray(questions)) { // 保存问题到数据库 await addQuestionsForChunk(projectId, chunk.id, questions); results.push({ chunkId: chunk.id, success: true, questions, total: questions.length }); } else { errors.push({ chunkId: chunk.id, error: 'Failed to parse questions' }); } } catch (error) { console.error(`Failed to generate questions for text block ${chunk.id}:`, error); errors.push({ chunkId: chunk.id, error: error.message || 'Failed to generate questions' }); } } // 返回生成结果 return NextResponse.json({ results, errors, totalSuccess: results.length, totalErrors: errors.length, totalChunks: chunks.length }); } catch (error) { console.error('Failed to generate questions:', error); return NextResponse.json({ error: error.message || 'Failed to generate questions' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/llamaFactory/checkConfig/route.js ```js path="/app/api/projects/[projectId]/llamaFactory/checkConfig/route.js" import { NextResponse } from 'next/server'; import path from 'path'; import fs from 'fs'; import { getProjectRoot } from '@/lib/db/base'; export async function GET(request, { params }) { try { const { projectId } = params; if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); const configPath = path.join(projectPath, 'dataset_info.json'); const exists = fs.existsSync(configPath); return NextResponse.json({ exists, configPath: exists ? configPath : null }); } catch (error) { console.error('Error checking Llama Factory config:', error); return NextResponse.json({ error: error.message }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/llamaFactory/generate/route.js ```js path="/app/api/projects/[projectId]/llamaFactory/generate/route.js" import { NextResponse } from 'next/server'; import path from 'path'; import fs from 'fs'; import { getProjectRoot } from '@/lib/db/base'; import { getDatasets } from '@/lib/db/datasets'; export async function POST(request, { params }) { try { const { projectId } = params; const { formatType, systemPrompt, confirmedOnly, includeCOT } = await request.json(); if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); const configPath = path.join(projectPath, 'dataset_info.json'); const alpacaPath = path.join(projectPath, 'alpaca.json'); const sharegptPath = path.join(projectPath, 'sharegpt.json'); // 获取数据集 let datasets = await getDatasets(projectId); // 如果只导出已确认的数据集 if (confirmedOnly) { datasets = datasets.filter(dataset => dataset.confirmed); } // 创建 dataset_info.json 配置 const config = { [`[Easy Dataset] [${projectId}] Alpaca`]: { file_name: 'alpaca.json', columns: { prompt: 'instruction', query: 'input', response: 'output', system: 'system' } }, [`[Easy Dataset] [${projectId}] ShareGPT`]: { file_name: 'sharegpt.json', formatting: 'sharegpt', columns: { messages: 'messages' }, tags: { role_tag: 'role', content_tag: 'content', user_tag: 'user', assistant_tag: 'assistant', system_tag: 'system' } } }; // 生成数据文件 const alpacaData = datasets.map(({ question, answer, cot }) => ({ instruction: question, input: '', output: cot && includeCOT ? `${cot}\n${answer}` : answer, system: systemPrompt || '' })); const sharegptData = datasets.map(({ question, answer, cot }) => { const messages = []; if (systemPrompt) { messages.push({ role: 'system', content: systemPrompt }); } messages.push({ role: 'user', content: question }); messages.push({ role: 'assistant', content: cot && includeCOT ? `${cot}\n${answer}` : answer }); return { messages }; }); // 写入文件 await fs.promises.writeFile(configPath, JSON.stringify(config, null, 2)); await fs.promises.writeFile(alpacaPath, JSON.stringify(alpacaData, null, 2)); await fs.promises.writeFile(sharegptPath, JSON.stringify(sharegptData, null, 2)); return NextResponse.json({ success: true, configPath, files: [ { path: alpacaPath, format: 'alpaca' }, { path: sharegptPath, format: 'sharegpt' } ] }); } catch (error) { console.error('Error generating Llama Factory config:', error); return NextResponse.json({ error: error.message }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/models/[modelId]/route.js ```js path="/app/api/projects/[projectId]/models/[modelId]/route.js" import { NextResponse } from 'next/server'; import { getProjectRoot } from '@/lib/db/base'; import path from 'path'; import fs from 'fs/promises'; export async function GET(request, { params }) { try { const { projectId, modelId } = params; // 验证项目ID和模型ID if (!projectId || !modelId) { return NextResponse.json({ error: 'The project ID and model ID cannot be empty' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 检查项目是否存在 try { await fs.access(projectPath); } catch (error) { return NextResponse.json({ error: 'The project does not exist' }, { status: 404 }); } // 获取模型配置文件路径 const modelConfigPath = path.join(projectPath, 'model-config.json'); // 检查模型配置文件是否存在 try { await fs.access(modelConfigPath); } catch (error) { return NextResponse.json({ error: 'The model configuration does not exist' }, { status: 404 }); } // 读取模型配置文件 const modelConfigData = await fs.readFile(modelConfigPath, 'utf-8'); const modelConfig = JSON.parse(modelConfigData); // 查找指定ID的模型 const model = modelConfig.find(model => model.id === modelId); if (!model) { return NextResponse.json({ error: 'The model does not exist' }, { status: 404 }); } return NextResponse.json(model); } catch (error) { console.error('Error getting model:', error); return NextResponse.json({ error: 'Failed to get model' }, { status: 500 }); } } export async function PUT(request, { params }) { try { const { projectId, modelId } = params; // 验证项目ID和模型ID if (!projectId || !modelId) { return NextResponse.json({ error: 'The project ID and model ID cannot be empty' }, { status: 400 }); } // 获取请求体 const modelData = await request.json(); // 验证请求体 if (!modelData || !modelData.provider || !modelData.name) { return NextResponse.json({ error: 'The model data is incomplete' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 检查项目是否存在 try { await fs.access(projectPath); } catch (error) { return NextResponse.json({ error: 'The project does not exist' }, { status: 404 }); } // 获取模型配置文件路径 const modelConfigPath = path.join(projectPath, 'model-config.json'); // 读取模型配置文件 let modelConfig = []; try { const modelConfigData = await fs.readFile(modelConfigPath, 'utf-8'); modelConfig = JSON.parse(modelConfigData); } catch (error) { // 如果文件不存在，创建一个空数组 } // 更新模型数据 const modelIndex = modelConfig.findIndex(model => model.id === modelId); if (modelIndex >= 0) { // 更新现有模型 modelConfig[modelIndex] = { ...modelConfig[modelIndex], ...modelData, id: modelId // 确保ID不变 }; } else { // 添加新模型 modelConfig.push({ ...modelData, id: modelId }); } // 写入模型配置文件 await fs.writeFile(modelConfigPath, JSON.stringify(modelConfig, null, 2), 'utf-8'); return NextResponse.json({ message: 'Model configuration updated successfully' }); } catch (error) { console.error('Error updating model configuration:', error); return NextResponse.json({ error: 'Failed to update model configuration' }, { status: 500 }); } } export async function DELETE(request, { params }) { try { const { projectId, modelId } = params; // 验证项目ID和模型ID if (!projectId || !modelId) { return NextResponse.json({ error: 'The project ID and model ID cannot be empty' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 检查项目是否存在 try { await fs.access(projectPath); } catch (error) { return NextResponse.json({ error: 'The project does not exist' }, { status: 404 }); } // 获取模型配置文件路径 const modelConfigPath = path.join(projectPath, 'model-config.json'); // 检查模型配置文件是否存在 try { await fs.access(modelConfigPath); } catch (error) { return NextResponse.json({ error: 'The model configuration does not exist' }, { status: 404 }); } // 读取模型配置文件 const modelConfigData = await fs.readFile(modelConfigPath, 'utf-8'); let modelConfig = JSON.parse(modelConfigData); // 过滤掉要删除的模型 const initialLength = modelConfig.length; modelConfig = modelConfig.filter(model => model.id !== modelId); // 检查是否找到并删除了模型 if (modelConfig.length === initialLength) { return NextResponse.json({ error: 'The model does not exist' }, { status: 404 }); } // 写入模型配置文件 await fs.writeFile(modelConfigPath, JSON.stringify(modelConfig, null, 2), 'utf-8'); return NextResponse.json({ message: 'Model deleted successfully' }); } catch (error) { console.error('Error deleting model:', error); return NextResponse.json({ error: 'Failed to delete model' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/models/route.js ```js path="/app/api/projects/[projectId]/models/route.js" import { NextResponse } from 'next/server'; import path from 'path'; import fs from 'fs/promises'; import { getProjectRoot } from '@/lib/db/base'; // 获取模型配置 export async function GET(request, { params }) { try { const { projectId } = params; // 验证项目 ID if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 检查项目是否存在 try { await fs.access(projectPath); } catch (error) { return NextResponse.json({ error: 'The project does not exist' }, { status: 404 }); } // 获取模型配置文件路径 const modelConfigPath = path.join(projectPath, 'model-config.json'); // 检查模型配置文件是否存在 try { await fs.access(modelConfigPath); } catch (error) { // 如果配置文件不存在，返回默认配置 return NextResponse.json([]); } // 读取模型配置文件 const modelConfigData = await fs.readFile(modelConfigPath, 'utf-8'); const modelConfig = JSON.parse(modelConfigData); return NextResponse.json(modelConfig); } catch (error) { console.error('Error obtaining model configuration:', error); return NextResponse.json({ error: 'Failed to obtain model configuration' }, { status: 500 }); } } // 更新模型配置 export async function PUT(request, { params }) { try { const { projectId } = params; // 验证项目 ID if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取请求体 const modelConfig = await request.json(); // 验证请求体 if (!modelConfig || !Array.isArray(modelConfig)) { return NextResponse.json({ error: 'The model configuration must be an array' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 检查项目是否存在 try { await fs.access(projectPath); } catch (error) { return NextResponse.json({ error: 'The project does not exist' }, { status: 404 }); } // 获取模型配置文件路径 const modelConfigPath = path.join(projectPath, 'model-config.json'); // 写入模型配置文件 await fs.writeFile(modelConfigPath, JSON.stringify(modelConfig, null, 2), 'utf-8'); return NextResponse.json({ message: 'Model configuration updated successfully' }); } catch (error) { console.error('Error updating model configuration:', error); return NextResponse.json({ error: 'Failed to update model configuration' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/pdf/route.js ```js path="/app/api/projects/[projectId]/pdf/route.js" import { NextResponse } from 'next/server'; import { deleteFile } from '@/lib/db/texts'; import PdfProcessor from '@/lib/pdf-processing/core'; import { getProject, updateProject } from '@/lib/db/index'; // Replace the deprecated config export with the new export syntax export const dynamic = 'force-dynamic'; // This tells Next.js not to parse the request body automatically export const bodyParser = false; // 处理PDF文件 export async function GET(request, { params }) { try { const { projectId } = params; const fileName = request.nextUrl.searchParams.get('fileName'); let strategy = request.nextUrl.searchParams.get('strategy'); const currentLanguage = request.nextUrl.searchParams.get('currentLanguage'); const visionModel = request.nextUrl.searchParams.get('modelId'); // 验证项目ID if (!projectId) { return NextResponse.json({ error: '项目ID不能为空' }, { status: 400 }); } if (!fileName) { return NextResponse.json({ error: '文件名不能为空' }, { status: 400 }); } //如果没有正确获取到strategy字段，则使用默认配置 if (!strategy) { strategy = 'default'; } // 获取项目信息 const project = await getProject(projectId); // 创建处理器 const processor = new PdfProcessor(strategy); // 使用当前策略处理 const result = await processor.process(projectId, fileName, { language: currentLanguage, visionModelId: visionModel}); //准换完成后删除pdf文件 deleteFile(projectId, fileName); // 更新项目配置，移除已删除的文件 const uploadedFiles = project.uploadedFiles || []; const updatedFiles = uploadedFiles.filter(f => f !== fileName); await updateProject(projectId, { ...project, uploadedFiles: updatedFiles }); //先检查PDF转换是否成功，再将转换后的文件写入配置 if (!result.success) { throw new Error(result.error); } //将转换后文件加入到配置中 if (!updatedFiles.includes(fileName)) { updatedFiles.push(fileName.replace('.pdf', '.md')); } await updateProject(projectId, { ...project, uploadedFiles: updatedFiles }); return NextResponse.json({ projectId, project, uploadedFiles: updatedFiles, batch_id: result.data }); } catch (error) { console.error('PDF处理流程出错:', error); return NextResponse.json({ error: error.message || 'PDF处理流程' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/playground/chat/route.js ```js path="/app/api/projects/[projectId]/playground/chat/route.js" import { NextResponse } from 'next/server'; import LLMClient from '@/lib/llm/core/index'; export async function POST(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取请求体 const { model, messages } = await request.json(); // 验证请求参数 if (!model) { return NextResponse.json({ error: 'The model parameters cannot be empty' }, { status: 400 }); } if (!Array.isArray(messages) || messages.length === 0) { return NextResponse.json({ error: 'The message list cannot be empty' }, { status: 400 }); } // 使用自定义的LLM客户端 const llmClient = new LLMClient({ provider: model.provider, endpoint: model.endpoint, apiKey: model.apiKey, model: model.name, temperature: model.temperature, maxTokens: model.maxTokens, type: model.type // 添加模型类型，用于区分语言模型和视觉模型 }); // 格式化消息历史 const formattedMessages = messages.map(msg => { // 处理纯文本消息 if (typeof msg.content === 'string') { return { role: msg.role, content: msg.content }; } // 处理包含图片的复合消息（用于视觉模型） else if (Array.isArray(msg.content)) { return { role: msg.role, content: msg.content }; } // 默认情况 return { role: msg.role, content: msg.content }; }); // 调用LLM API let response = ''; try { response = await llmClient.getResponse(formattedMessages); } catch (error) { console.error('Failed to call LLM API:', error); return NextResponse.json( { error: `Failed to call ${model.provider} model: ${error.message}` }, { status: 500 } ); } return NextResponse.json({ response }); } catch (error) { console.error('Failed to process chat request:', error); return NextResponse.json({ error: `Failed to process chat request: ${error.message}` }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/playground/chat/stream/route.js ```js path="/app/api/projects/[projectId]/playground/chat/stream/route.js" import { NextResponse } from 'next/server'; import LLMClient from '@/lib/llm/core/index'; /** * 流式输出的聊天接口 */ export async function POST(request, { params }) { const { projectId } = params; try { const body = await request.json(); const { model, messages } = body; if (!model || !messages) { return NextResponse.json({ error: 'Missing necessary parameters' }, { status: 400 }); } // 创建 LLM 客户端 const llmClient = new LLMClient({ provider: model.provider, endpoint: model.endpoint, apiKey: model.apiKey, model: model.name, temperature: model.temperature, maxTokens: model.maxTokens, type: model.type // 添加模型类型，用于区分语言模型和视觉模型 }); // 格式化消息历史 const formattedMessages = messages.map(msg => { // 处理纯文本消息 if (typeof msg.content === 'string') { return { role: msg.role, content: msg.content }; } // 处理包含图片的复合消息（用于视觉模型） else if (Array.isArray(msg.content)) { return { role: msg.role, content: msg.content }; } // 默认情况 return { role: msg.role, content: msg.content }; }); try { // 调用流式 API const stream = await llmClient.chatStream(formattedMessages); // 返回流式响应 return stream; } catch (error) { console.error('Failed to call LLM API:', error); return NextResponse.json( { error: `Failed to call ${model.provider} model: ${error.message}` }, { status: 500 } ); } } catch (error) { console.error('Failed to process stream chat request:', error); return NextResponse.json({ error: `Failed to process stream chat request: ${error.message}` }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/preview/[fileName]/route.js ```js path="/app/api/projects/[projectId]/preview/[fileName]/route.js" import { NextResponse } from 'next/server'; import fs from 'fs'; import path from 'path'; import { getProjectRoot } from '@/lib/db/base'; // 获取文件内容 export async function GET(request, { params }) { try { const { projectId, fileName: f } = params; const fileName = decodeURIComponent(f); // 验证参数 if (!projectId) { return NextResponse.json({ error: 'Project ID cannot be empty' }, { status: 400 }); } if (!fileName) { return NextResponse.json({ error: 'file Name cannot be empty' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 获取文件路径 const filePath = path.join(projectPath, 'files', fileName); //获取文件 const buffer = fs.readFileSync(filePath); const text = buffer.toString('utf-8'); return NextResponse.json({fileName:fileName,content:text}); } catch (error) { console.error('Failed to get text block content:', error); return NextResponse.json({ error: error.message || 'Failed to get text block content' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/questions/[questionId]/route.js ```js path="/app/api/projects/[projectId]/questions/[questionId]/route.js" import { NextResponse } from 'next/server'; import { deleteQuestion } from '@/lib/db/questions'; // 删除单个问题 export async function DELETE(request, { params }) { try { const { projectId, questionId } = params; // 验证参数 if (!projectId) { return NextResponse.json({ error: 'Project ID is required' }, { status: 400 }); } if (!questionId) { return NextResponse.json({ error: 'Question ID is required' }, { status: 400 }); } // 从请求体中获取 chunkId const { chunkId } = await request.json(); if (!chunkId) { return NextResponse.json({ error: 'Chunk ID is required' }, { status: 400 }); } // 删除问题 await deleteQuestion(projectId, questionId, chunkId); return NextResponse.json({ success: true, message: 'Delete successful' }); } catch (error) { console.error('Delete failed:', error); return NextResponse.json({ error: error.message || 'Delete failed' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/questions/batch-delete/route.js ```js path="/app/api/projects/[projectId]/questions/batch-delete/route.js" import { NextResponse } from 'next/server'; import { batchDeleteQuestions } from '@/lib/db/questions'; // 批量删除问题 export async function DELETE(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'Project ID is required' }, { status: 400 }); } // 从请求体中获取要删除的问题列表 const { questions } = await request.json(); if (!questions || !Array.isArray(questions) || questions.length === 0) { return NextResponse.json({ error: 'Questions list is required' }, { status: 400 }); } // 验证每个问题都有必要的字段 for (const question of questions) { if (!question.questionId || !question.chunkId) { return NextResponse.json( { error: 'Question information is incomplete, must include questionId and chunkId' }, { status: 400 } ); } } // 批量删除问题 await batchDeleteQuestions(projectId, questions); return NextResponse.json({ success: true, message: `Successfully deleted ${questions.length} questions` }); } catch (error) { console.error('Failed to batch delete questions:', error); return NextResponse.json({ error: error.message || 'Failed to batch delete questions' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/questions/route.js ```js path="/app/api/projects/[projectId]/questions/route.js" import { NextResponse } from 'next/server'; import { getQuestions, saveQuestions } from '@/lib/db/questions'; import { getDatasets } from '@/lib/db/datasets'; // 获取项目的所有问题 export async function GET(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'Missing project ID' }, { status: 400 }); } // 获取问题列表 const nestedQuestions = await getQuestions(projectId); // 获取数据集 const datasets = await getDatasets(projectId); // 将嵌套的问题数据结构拍平 const flattenedQuestions = []; nestedQuestions.forEach(item => { const { chunkId, questions } = item; if (questions && Array.isArray(questions)) { questions.forEach(question => { const dataSites = datasets.filter(dataset => dataset.question === question.question); flattenedQuestions.push({ ...question, chunkId, dataSites }); }); } }); return NextResponse.json(flattenedQuestions); } catch (error) { console.error('Failed to get questions:', error); return NextResponse.json({ error: error.message || 'Failed to get questions' }, { status: 500 }); } } // 新增问题 export async function POST(request, { params }) { try { const { projectId } = params; const body = await request.json(); const { question, chunkId, label } = body; // 验证必要参数 if (!projectId || !question || !chunkId) { return NextResponse.json({ error: 'Missing necessary parameters' }, { status: 400 }); } // 获取所有问题 const questionsData = await getQuestions(projectId); // 找到或创建文本块 let chunkIndex = questionsData.findIndex(item => item.chunkId === chunkId); if (chunkIndex === -1) { questionsData.push({ chunkId: chunkId, questions: [] }); chunkIndex = questionsData.length - 1; } // 检查问题是否已存在 const existingQuestion = questionsData[chunkIndex].questions.find(q => q.question === question); if (existingQuestion) { return NextResponse.json({ error: 'Question already exists' }, { status: 400 }); } // 添加新问题 questionsData[chunkIndex].questions.push({ question: question, label: label || 'other' }); // 保存更新后的数据 await saveQuestions(projectId, questionsData); // 返回成功响应 return NextResponse.json({ question, chunkId, label, dataSites: [] // 新问题还没有数据集 }); } catch (error) { console.error('Failed to create question:', error); return NextResponse.json({ error: error.message || 'Failed to create question' }, { status: 500 }); } } // 更新问题 export async function PUT(request, { params }) { try { const { projectId } = params; const body = await request.json(); const { question, oldQuestion, chunkId, label, oldChunkId } = body; // 验证必要参数 if (!projectId || !question || !oldQuestion || !chunkId || !oldChunkId) { return NextResponse.json({ error: 'Missing necessary parameters' }, { status: 400 }); } // 获取所有问题 const questionsData = await getQuestions(projectId); // 找到原问题所在的文本块 const oldChunkIndex = questionsData.findIndex(item => item.chunkId === oldChunkId); if (oldChunkIndex === -1) { return NextResponse.json({ error: 'Original chunk not found' }, { status: 404 }); } // 找到原问题在文本块中的位置 const oldQuestionIndex = questionsData[oldChunkIndex].questions.findIndex(q => q.question === oldQuestion); if (oldQuestionIndex === -1) { return NextResponse.json({ error: 'Original question not found' }, { status: 404 }); } // 如果文本块发生变化 if (chunkId !== oldChunkId) { // 从原文本块中删除问题 questionsData[oldChunkIndex].questions.splice(oldQuestionIndex, 1); // 找到或创建新文本块 let newChunkIndex = questionsData.findIndex(item => item.chunkId === chunkId); if (newChunkIndex === -1) { questionsData.push({ chunkId: chunkId, questions: [] }); newChunkIndex = questionsData.length - 1; } // 添加到新文本块 questionsData[newChunkIndex].questions.push({ question: question, label: label || 'other' }); } else { // 更新问题内容和标签 questionsData[oldChunkIndex].questions[oldQuestionIndex] = { question: question, label: label || 'other' }; } // 保存更新后的数据 await saveQuestions(projectId, questionsData); const datasets = await getDatasets(projectId); const dataSites = datasets.filter(dataset => dataset.question === question); // 返回更新后的问题数据 return NextResponse.json({ question, chunkId, label, dataSites }); } catch (error) { console.error('更新问题失败:', error); return NextResponse.json({ error: error.message || '更新问题失败' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/route.js ```js path="/app/api/projects/[projectId]/route.js" import { getProject, updateProject, deleteProject } from '@/lib/db/index'; // 获取项目详情 export async function GET(request, { params }) { try { const { projectId } = params; const project = await getProject(projectId); if (!project) { return Response.json({ error: '项目不存在' }, { status: 404 }); } return Response.json(project); } catch (error) { console.error('获取项目详情出错:', error); return Response.json({ error: error.message }, { status: 500 }); } } // 更新项目 export async function PUT(request, { params }) { try { const { projectId } = params; const projectData = await request.json(); // 验证必要的字段 if (!projectData.name) { return Response.json({ error: '项目名称不能为空' }, { status: 400 }); } const updatedProject = await updateProject(projectId, projectData); if (!updatedProject) { return Response.json({ error: '项目不存在' }, { status: 404 }); } return Response.json(updatedProject); } catch (error) { console.error('更新项目出错:', error); return Response.json({ error: error.message }, { status: 500 }); } } // 删除项目 export async function DELETE(request, { params }) { try { const { projectId } = params; const success = await deleteProject(projectId); if (!success) { return Response.json({ error: '项目不存在' }, { status: 404 }); } return Response.json({ success: true }); } catch (error) { console.error('删除项目出错:', error); return Response.json({ error: error.message }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/split/route.js ```js path="/app/api/projects/[projectId]/split/route.js" import { NextResponse } from 'next/server'; import { splitProjectFile, getProjectChunks } from '@/lib/text-splitter'; import LLMClient from '@/lib/llm/core/index'; import getLabelPrompt from '@/lib/llm/prompts/label'; import getLabelEnPrompt from '@/lib/llm/prompts/labelEn'; import { deleteFile } from '@/lib/db/texts'; import { getProject, updateProject } from '@/lib/db/projects'; import { saveTags, getTags } from '@/lib/db/tags'; const { extractJsonFromLLMOutput } = require('@/lib/llm/common/util'); // 处理文本分割请求 export async function POST(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: '项目ID不能为空' }, { status: 400 }); } // 获取请求体 const { fileName, model, language } = await request.json(); if (!model) { return NextResponse.json({ error: '请选择模型' }, { status: 400 }); } // 验证文件名 if (!fileName) { return NextResponse.json({ error: '文件名不能为空' }, { status: 400 }); } const project = await getProject(projectId); const { globalPrompt, domainTreePrompt } = project; // 分割文本 const result = await splitProjectFile(projectId, fileName); const { toc } = result; const llmClient = new LLMClient({ provider: model.provider, endpoint: model.endpoint, apiKey: model.apiKey, model: model.name, temperature: model.temperature, maxTokens: model.maxTokens }); // 生成领域树 console.log(projectId, fileName, 'Text split completed, starting to build domain tree'); const promptFunc = language === 'en' ? getLabelEnPrompt : getLabelPrompt; const prompt = promptFunc({ text: toc, globalPrompt, domainTreePrompt }); const response = await llmClient.getResponse(prompt); const tags = extractJsonFromLLMOutput(response); if (!response || !tags) { // 删除前面生成的文件 await deleteFile(projectId, fileName); const uploadedFiles = project.uploadedFiles || []; const updatedFiles = uploadedFiles.filter(f => f !== fileName); await updateProject(projectId, { ...project, uploadedFiles: updatedFiles }); return NextResponse.json( { error: 'AI analysis failed, please check model configuration, delete file and retry!' }, { status: 400 } ); } console.log(projectId, fileName, 'Domain tree built:', tags); await saveTags(projectId, tags); return NextResponse.json({ ...result, tags }); } catch (error) { console.error('Text split error:', error); return NextResponse.json({ error: error.message || 'Text split failed' }, { status: 500 }); } } // 获取项目中的所有文本块 export async function GET(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'The project ID cannot be empty' }, { status: 400 }); } // 获取文本块详细信息 const result = await getProjectChunks(projectId); const tags = await getTags(projectId); // 返回详细的文本块信息和文件结果（单个文件） return NextResponse.json({ chunks: result.chunks, ...result.fileResult, // 单个文件结果，而不是数组 tags }); } catch (error) { console.error('Failed to get text chunks:', error); return NextResponse.json({ error: error.message || 'Failed to get text chunks' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/tags/route.js ```js path="/app/api/projects/[projectId]/tags/route.js" import { NextResponse } from 'next/server'; import { saveTags, getTags } from '@/lib/db/tags'; // 获取项目的标签树 export async function GET(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'Project ID is required' }, { status: 400 }); } // 获取标签树 const tags = await getTags(projectId); return NextResponse.json({ tags }); } catch (error) { console.error('Failed to obtain the label tree:', error); return NextResponse.json({ error: error.message || 'Failed to obtain the label tree' }, { status: 500 }); } } // 更新项目的标签树 export async function PUT(request, { params }) { try { const { projectId } = params; // 验证项目ID if (!projectId) { return NextResponse.json({ error: 'Project ID is required' }, { status: 400 }); } // 获取请求体 const { tags } = await request.json(); // 验证标签数据 if (!tags || !Array.isArray(tags)) { return NextResponse.json({ error: 'Tag data format is incorrect' }, { status: 400 }); } // 保存更新后的标签树 const updatedTags = await saveTags(projectId, tags); return NextResponse.json({ tags: updatedTags }); } catch (error) { console.error('Failed to update tags:', error); return NextResponse.json({ error: error.message || 'Failed to update tags' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/tasks/route.js ```js path="/app/api/projects/[projectId]/tasks/route.js" import { NextResponse } from 'next/server'; import path from 'path'; import fs from 'fs/promises'; import { getProjectRoot } from '@/lib/db/base'; import { getTaskConfig } from '@/lib/db/projects'; // 获取任务配置 export async function GET(request, { params }) { try { const { projectId } = params; // 验证项目 ID if (!projectId) { return NextResponse.json({ error: 'Project ID is required' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 检查项目是否存在 try { await fs.access(projectPath); } catch (error) { return NextResponse.json({ error: 'Project does not exist' }, { status: 404 }); } const taskConfig = await getTaskConfig(projectId); return NextResponse.json(taskConfig); } catch (error) { console.error('Failed to obtain task configuration:', error); return NextResponse.json({ error: 'Failed to obtain task configuration' }, { status: 500 }); } } // 更新任务配置 export async function PUT(request, { params }) { try { const { projectId } = params; // 验证项目 ID if (!projectId) { return NextResponse.json({ error: 'Project ID is required' }, { status: 400 }); } // 获取请求体 const taskConfig = await request.json(); // 验证请求体 if (!taskConfig) { return NextResponse.json({ error: 'Task configuration cannot be empty' }, { status: 400 }); } // 获取项目根目录 const projectRoot = await getProjectRoot(); const projectPath = path.join(projectRoot, projectId); // 检查项目是否存在 try { await fs.access(projectPath); } catch (error) { return NextResponse.json({ error: 'Project does not exist' }, { status: 404 }); } // 获取任务配置文件路径 const taskConfigPath = path.join(projectPath, 'task-config.json'); // 写入任务配置文件 await fs.writeFile(taskConfigPath, JSON.stringify(taskConfig, null, 2), 'utf-8'); return NextResponse.json({ message: 'Task configuration updated successfully' }); } catch (error) { console.error('Failed to update task configuration:', error); return NextResponse.json({ error: 'Failed to update task configuration' }, { status: 500 }); } } ``` ## /app/api/projects/[projectId]/text-split/route.js ```js path="/app/api/projects/[projectId]/text-split/route.js" import { NextResponse } from 'next/server'; import { saveFile, getProject, saveTextChunk } from '@/lib/db/index'; import { v4 as uuidv4 } from 'uuid'; import fs from 'fs'; // 用于处理文本分割的函数 function splitTextContent(text, minChars = 1500, maxChars = 2000) { // 基本的分割算法，按照段落分割，然后合并到合适的长度 const paragraphs = text.split(/\n\s*\n/); const chunks = []; let currentChunk = ''; for (const paragraph of paragraphs) { // 如果当前段落很长，需要进一步分割 if (paragraph.length > maxChars) { // 如果当前chunk不为空，先添加到chunks if (currentChunk.length > 0) { chunks.push(currentChunk); currentChunk = ''; } // 按句子分割长段落 const sentences = paragraph.split(/(?<=[.!?])\s+/); let sentenceChunk = ''; for (const sentence of sentences) { if (sentenceChunk.length + sentence.length <= maxChars) { sentenceChunk += (sentenceChunk ? ' ' : '') + sentence; } else { chunks.push(sentenceChunk); sentenceChunk = sentence; } } if (sentenceChunk.length > 0) { currentChunk = sentenceChunk; } } else { // 尝试将段落添加到当前chunk if (currentChunk.length + paragraph.length + 2 <= maxChars) { currentChunk += (currentChunk ? '\n\n' : '') + paragraph; } else { // 如果当前chunk至少达到了最小长度，添加到chunks if (currentChunk.length >= minChars) { chunks.push(currentChunk); currentChunk = paragraph; } else { // 否则继续添加，即使超过了最大长度 currentChunk += (currentChunk ? '\n\n' : '') + paragraph; } } } } // 添加最后一个chunk if (currentChunk.length > 0) { chunks.push(currentChunk); } return chunks; } // 从Markdown中提取目录结构 function extractDirectoryFromMarkdown(text) { const headings = text.match(/^#{1,6}\s+(.+)$/gm) || []; // 映射标题级别和内容 return headings.map(heading => { const level = heading.match(/^(#{1,6})\s/)[1].length; const content = heading.replace(/^#{1,6}\s+/, ''); return { level, content }; }); } export async function POST(request, { params }) { try { const { projectId } = params; // 获取项目信息 const project = await getProject(projectId); if (!project) { return NextResponse.json({ error: 'Project does not exist' }, { status: 404 }); } // 文本分割设置 const settings = project.settings || {}; const textSplitSettings = settings.textSplit || { minChars: 1500, maxChars: 2000 }; // 获取上传的文件 const formData = await request.formData(); const files = formData.getAll('files'); if (!files || files.length === 0) { return NextResponse.json({ error: 'No files uploaded' }, { status: 400 }); } const results = []; for (const file of files) { // 只处理Markdown文件 if (!file.name.toLowerCase().endsWith('.md')) { continue; } const buffer = Buffer.from(await file.arrayBuffer()); const fileName = file.name; // 保存原始文件 await saveFile(projectId, buffer, fileName); // 读取文件内容并分割 const text = buffer.toString('utf-8'); // 提取目录结构 const directory = extractDirectoryFromMarkdown(text); // 分割文本 const chunks = splitTextContent(text, textSplitSettings.minChars, textSplitSettings.maxChars); // 保存分割后的文本片段 const chunkResults = []; for (let i = 0; i < chunks.length; i++) { const chunkId = uuidv4(); const chunkData = { id: chunkId, title: `${fileName}-片段${i + 1}`, content: chunks[i], wordCount: chunks[i].length, fileName: fileName, hasQuestions: false, createdAt: new Date().toISOString() }; await saveTextChunk(projectId, chunkId, chunkData); chunkResults.push(chunkData); } results.push({ fileName, chunksCount: chunks.length, directory, chunks: chunkResults.map(chunk => ({ id: chunk.id, title: chunk.title, wordCount: chunk.wordCount })) }); } return NextResponse.json(results); } catch (error) { console.error('Failed to split text:', error); return NextResponse.json({ error: error.message }, { status: 500 }); } } // 获取项目中所有文本片段 export async function GET(request, { params }) { try { const { projectId } = params; // 获取项目信息 const project = await getProject(projectId); if (!project) { return NextResponse.json({ error: 'Project does not exist' }, { status: 404 }); } // 获取所有文本片段 const chunkIds = await getTextChunkIds(projectId); const chunks = []; for (const chunkId of chunkIds) { const chunk = await getTextChunk(projectId, chunkId); chunks.push(chunk); } return NextResponse.json(chunks); } catch (error) { console.error('Failed to get text chunks:', error); return NextResponse.json({ error: error.message }, { status: 500 }); } } ``` ## /app/api/projects/route.js ```js path="/app/api/projects/route.js" import { createProject, getProjects, getProjectModelConfig } from '@/lib/db/index'; export async function POST(request) { try { const projectData = await request.json(); // 验证必要的字段 if (!projectData.name) { return Response.json({ error: '项目名称不能为空' }, { status: 400 }); } // 如果指定了要复用的项目配置 if (projectData.reuseConfigFrom) { projectData.modelConfig = await getProjectModelConfig(projectData.reuseConfigFrom); } // 创建项目 const newProject = await createProject(projectData); return Response.json(newProject, { status: 201 }); } catch (error) { console.error('创建项目出错:', error); return Response.json({ error: error.message }, { status: 500 }); } } export async function GET(request) { try { // 获取所有项目 const userProjects = await getProjects(); // 为每个项目添加问题数量和数据集数量 const projectsWithStats = await Promise.all( userProjects.map(async project => { // 获取问题数量 const questions = (await import('@/lib/db/questions').then(module => module.getQuestions(project.id))) || []; const ques = questions.map(q => q.questions).flat(); const questionsCount = ques.length; // 获取数据集数量 const datasets = await import('@/lib/db/datasets').then(module => module.getDatasets(project.id)); const datasetsCount = Array.isArray(datasets) ? datasets.length : 0; // 添加最后更新时间 const lastUpdated = new Date().toLocaleDateString('zh-CN'); return { ...project, questionsCount, datasetsCount, lastUpdated }; }) ); return Response.json(projectsWithStats); } catch (error) { console.error('获取项目列表出错:', error); return Response.json({ error: error.message }, { status: 500 }); } } ``` ## /app/api/update/route.js ```js path="/app/api/update/route.js" import { NextResponse } from 'next/server'; import { exec } from 'child_process'; import path from 'path'; import fs from 'fs'; // 执行更新脚本 export async function POST() { try { // 检查是否在客户端环境中运行 const desktopDir = path.join(process.cwd(), 'desktop'); const updaterPath = path.join(desktopDir, 'scripts', 'updater.js'); if (!fs.existsSync(updaterPath)) { return NextResponse.json( { success: false, message: '更新功能仅在客户端环境中可用' }, { status: 400 } ); } // 执行更新脚本 return new Promise(resolve => { const updaterProcess = exec(`node "${updaterPath}"`, { cwd: process.cwd() }); let output = ''; updaterProcess.stdout.on('data', data => { output += data.toString(); console.log(`Update output: ${data}`); }); updaterProcess.stderr.on('data', data => { output += data.toString(); console.error(`Update error: ${data}`); }); updaterProcess.on('close', code => { console.log(`Update process exit, exit code: ${code}`); if (code === 0) { resolve( NextResponse.json({ success: true, message: 'Update successful, application will restart' }) ); } else { resolve( NextResponse.json( { success: false, message: `Update failed, exit code: ${code}, output: ${output}` }, { status: 500 } ) ); } }); }); } catch (error) { console.error('Failed to execute update:', error); return NextResponse.json( { success: false, message: `Failed to execute update: ${error.message}` }, { status: 500 } ); } } ``` ## /app/dataset-square/page.js ```js path="/app/dataset-square/page.js" 'use client'; import { useState, useEffect } from 'react'; import { Box, Container, Typography, Paper, Divider, useTheme, alpha } from '@mui/material'; import StorageIcon from '@mui/icons-material/Storage'; import Navbar from '@/components/Navbar'; import { DatasetSearchBar } from '@/components/dataset-square/DatasetSearchBar'; import { DatasetSiteList } from '@/components/dataset-square/DatasetSiteList'; import { useTranslation } from 'react-i18next'; export default function DatasetSquarePage() { const [projects, setProjects] = useState([]); const [models, setModels] = useState([]); const theme = useTheme(); const { t } = useTranslation(); // 获取项目列表和模型列表 useEffect(() => { async function fetchData() { try { // 获取用户创建的项目详情 const response = await fetch('/api/projects'); if (response.ok) { const projectsData = await response.json(); setProjects(projectsData); } // 获取模型列表 const modelsResponse = await fetch('/api/models'); if (modelsResponse.ok) { const modelsData = await modelsResponse.json(); setModels(modelsData); } } catch (error) { console.error('获取数据失败:', error); } } fetchData(); }, []); return (
{/* 导航栏 */} {/* 头部区域 */} {/* 背景装饰 */} {t('datasetSquare.title')} {t('datasetSquare.subtitle')} {/* 搜索栏组件 */} {/* 内容区域 */} {/* 数据集网站列表组件 */}
); } ``` ## /app/globals.css ```css path="/app/globals.css" * { box-sizing: border-box; padding: 0; margin: 0; } html, body { max-width: 100vw; overflow-x: hidden; height: 100%; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } a { color: inherit; text-decoration: none; } /* 渐变文本样式 */ .gradient-text { background: linear-gradient(90deg, #2a5caa 0%, #8b5cf6 100%); -webkit-background-clip: text; -webkit-text-fill-color: transparent; background-clip: text; text-fill-color: transparent; } /* 页面容器下间距 */ main { min-height: calc(100vh - 64px); } /* 自定义滚动条 */ ::-webkit-scrollbar { width: 8px; height: 8px; } ::-webkit-scrollbar-track { background: transparent; } ::-webkit-scrollbar-thumb { background-color: rgba(0, 0, 0, 0.2); border-radius: 4px; } ::-webkit-scrollbar-thumb:hover { background-color: rgba(0, 0, 0, 0.3); } /* 暗色模式滚动条 */ [data-theme='dark'] ::-webkit-scrollbar-thumb { background-color: rgba(255, 255, 255, 0.2); } [data-theme='dark'] ::-webkit-scrollbar-thumb:hover { background-color: rgba(255, 255, 255, 0.3); } /* 方便的间距类 */ .mt-1 { margin-top: 8px; } .mt-2 { margin-top: 16px; } .mt-3 { margin-top: 24px; } .mt-4 { margin-top: 32px; } .mb-1 { margin-bottom: 8px; } .mb-2 { margin-bottom: 16px; } .mb-3 { margin-bottom: 24px; } .mb-4 { margin-bottom: 32px; } /* 响应式样式 */ @media (max-width: 600px) { .hide-on-mobile { display: none !important; } } /* 输入框和选择框边框简化 */ .plain-select .MuiOutlinedInput-notchedOutline, .plain-input .MuiOutlinedInput-notchedOutline { border-color: transparent !important; } /* 卡片悬停效果 */ .hover-card { transition: transform 0.2s ease, box-shadow 0.2s ease; } .hover-card:hover { transform: translateY(-4px); box-shadow: 0 12px 20px rgba(0, 0, 0, 0.1); } [data-theme='dark'] .hover-card:hover { box-shadow: 0 12px 20px rgba(0, 0, 0, 0.3); } ``` ## /app/layout.js ```js path="/app/layout.js" import './globals.css'; import ThemeRegistry from '@/components/ThemeRegistry'; import I18nProvider from '@/components/I18nProvider'; export const metadata = { title: 'Easy Dataset', description: '一个强大的 LLM 数据集生成工具', icons: { icon: '/imgs/logo.ico' // 更新为正确的文件名 } }; export default function RootLayout({ children }) { return ( {children} ); } ``` ## /app/page.js ```js path="/app/page.js" 'use client'; import { useState, useEffect } from 'react'; import { Container, Box, Typography, CircularProgress, Stack, useTheme } from '@mui/material'; import ErrorOutlineIcon from '@mui/icons-material/ErrorOutline'; import Navbar from '@/components/Navbar'; import HeroSection from '@/components/home/HeroSection'; import StatsCard from '@/components/home/StatsCard'; import ProjectList from '@/components/home/ProjectList'; import CreateProjectDialog from '@/components/home/CreateProjectDialog'; import { motion } from 'framer-motion'; import { useTranslation } from 'react-i18next'; export default function Home() { const { t } = useTranslation(); const [projects, setProjects] = useState([]); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); const [createDialogOpen, setCreateDialogOpen] = useState(false); useEffect(() => { async function fetchProjects() { try { setLoading(true); // 获取用户创建的项目详情 const response = await fetch(`/api/projects`); if (!response.ok) { throw new Error(t('projects.fetchFailed')); } const data = await response.json(); setProjects(data); } catch (error) { console.error(t('projects.fetchError'), error); setError(error.message); } finally { setLoading(false); } } fetchProjects(); }, []); const theme = useTheme(); return (
setCreateDialogOpen(true)} /> {/* */} {loading && ( {t('projects.loading')} )} {error && !loading && ( {t('projects.fetchFailed')}: {error} )} {!loading && ( setCreateDialogOpen(true)} /> )} setCreateDialogOpen(false)} />
); } ``` ## /app/projects/[projectId]/datasets/[datasetId]/page.js ```js path="/app/projects/[projectId]/datasets/[datasetId]/page.js" 'use client'; import { useState, useEffect } from 'react'; import { Container, Box, Typography, Button, IconButton, TextField, CircularProgress, Alert, Snackbar, Paper, Chip, Divider, alpha, useTheme, Dialog, DialogTitle, DialogContent, DialogActions, Tooltip } from '@mui/material'; import EditIcon from '@mui/icons-material/Edit'; import NavigateBeforeIcon from '@mui/icons-material/NavigateBefore'; import NavigateNextIcon from '@mui/icons-material/NavigateNext'; import DeleteIcon from '@mui/icons-material/Delete'; import AutoFixHighIcon from '@mui/icons-material/AutoFixHigh'; import VisibilityIcon from '@mui/icons-material/Visibility'; import { useRouter } from 'next/navigation'; import { useTranslation } from 'react-i18next'; import i18n from '@/lib/i18n'; import ChunkViewDialog from '@/components/text-split/ChunkViewDialog'; // 编辑区域组件 const EditableField = ({ label, value, multiline = true, editing, onEdit, onChange, onSave, onCancel, onOptimize }) => { const { t } = useTranslation(); const theme = useTheme(); return ( {label} {!editing && ( <> {onOptimize && ( )} )} {editing ? ( ) : ( {value || t('common.noData')} )} ); }; // AI优化对话框组件 const OptimizeDialog = ({ open, onClose, onConfirm, loading }) => { const [advice, setAdvice] = useState(''); const { t } = useTranslation(); const handleConfirm = () => { onConfirm(advice); }; return ( {t('datasets.aiOptimize')} {t('datasets.aiOptimizeAdvice')} setAdvice(e.target.value)} placeholder={t('datasets.aiOptimizeAdvicePlaceholder')} disabled={loading} /> ); }; export default function DatasetDetailsPage({ params }) { const { projectId, datasetId } = params; const router = useRouter(); const [dataset, setDataset] = useState(null); const [loading, setLoading] = useState(true); const [editingAnswer, setEditingAnswer] = useState(false); const [editingCot, setEditingCot] = useState(false); const [answerValue, setAnswerValue] = useState(''); const [cotValue, setCotValue] = useState(''); const [snackbar, setSnackbar] = useState({ open: false, message: '', severity: 'success' }); const [confirming, setConfirming] = useState(false); const [optimizeDialog, setOptimizeDialog] = useState({ open: false, loading: false }); const [viewDialogOpen, setViewDialogOpen] = useState(false); const [viewChunk, setViewChunk] = useState(null); const theme = useTheme(); // 获取数据集列表（用于导航） const [datasets, setDatasets] = useState([]); const { t } = useTranslation(); // 从本地存储获取模型参数 const getModelFromLocalStorage = () => { if (typeof window === 'undefined') return null; try { let model = null; // 尝试从 localStorage 获取完整的模型信息 const modelInfoStr = localStorage.getItem('selectedModelInfo'); if (modelInfoStr) { try { model = JSON.parse(modelInfoStr); } catch (e) { console.error('解析模型信息失败', e); return null; } } return model; } catch (error) { console.error('获取模型配置失败', error); return null; } }; // 获取所有数据集 const fetchDatasets = async () => { try { const response = await fetch(`/api/projects/${projectId}/datasets`); if (!response.ok) throw new Error(t('datasets.fetchFailed')); const data = await response.json(); setDatasets(data); // 找到当前数据集 const currentDataset = data.find(d => d.id === datasetId); if (currentDataset) { setDataset(currentDataset); setAnswerValue(currentDataset.answer); setCotValue(currentDataset.cot || ''); } } catch (error) { setSnackbar({ open: true, message: error.message, severity: 'error' }); } finally { setLoading(false); } }; const handleConfirm = async () => { try { setConfirming(true); const response = await fetch(`/api/projects/${projectId}/datasets?id=${datasetId}`, { method: 'PATCH', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ confirmed: true }) }); if (!response.ok) { throw new Error(t('common.failed')); } setDataset(prev => ({ ...prev, confirmed: true })); setSnackbar({ open: true, message: t('common.success'), severity: 'success' }); // 导航到下一个数据集 handleNavigate('next'); } catch (error) { setSnackbar({ open: true, message: error.message || t('common.failed'), severity: 'error' }); } finally { setConfirming(false); } }; useEffect(() => { fetchDatasets(); }, [projectId, datasetId]); // 导航到其他数据集 const handleNavigate = direction => { const currentIndex = datasets.findIndex(d => d.id === datasetId); if (currentIndex === -1) return; const newIndex = direction === 'prev' ? (currentIndex - 1 + datasets.length) % datasets.length : (currentIndex + 1) % datasets.length; const newDataset = datasets[newIndex]; router.push(`/projects/${projectId}/datasets/${newDataset.id}`); }; // 保存编辑 const handleSave = async (field, value) => { try { const response = await fetch(`/api/projects/${projectId}/datasets?id=${datasetId}`, { method: 'PATCH', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ [field]: value }) }); if (!response.ok) { throw new Error(t('common.failed')); } const data = await response.json(); setDataset(prev => ({ ...prev, [field]: value })); setSnackbar({ open: true, message: t('common.success'), severity: 'success' }); // 重置编辑状态 if (field === 'answer') setEditingAnswer(false); if (field === 'cot') setEditingCot(false); } catch (error) { setSnackbar({ open: true, message: error.message || t('common.failed'), severity: 'error' }); } }; // 删除数据集 const handleDelete = async () => { if (!confirm(t('datasets.confirmDeleteMessage'))) return; try { const response = await fetch(`/api/projects/${projectId}/datasets?id=${datasetId}`, { method: 'DELETE' }); if (!response.ok) { throw new Error(t('common.failed')); } // 找到当前数据集的索引 const currentIndex = datasets.findIndex(d => d.id === datasetId); // 如果这是最后一个数据集，返回列表页 if (datasets.length === 1) { router.push(`/projects/${projectId}/datasets`); return; } // 计算下一个数据集的索引 const nextIndex = (currentIndex + 1) % datasets.length; // 如果是最后一个，就去第一个 const nextDataset = datasets[nextIndex] || datasets[0]; // 导航到下一个数据集 router.push(`/projects/${projectId}/datasets/${nextDataset.id}`); // 更新本地数据集列表 setDatasets(prev => prev.filter(d => d.id !== datasetId)); } catch (error) { setSnackbar({ open: true, message: error.message || t('common.failed'), severity: 'error' }); } }; // 打开优化对话框 const handleOpenOptimizeDialog = () => { setOptimizeDialog({ open: true, loading: false }); }; // 关闭优化对话框 const handleCloseOptimizeDialog = () => { if (optimizeDialog.loading) return; setOptimizeDialog({ open: false, loading: false }); }; // 查看文本块详情 const handleViewChunk = async (chunkId) => { try { setViewDialogOpen(true); setViewChunk(null); const response = await fetch(`/api/projects/${projectId}/chunks/${encodeURIComponent(chunkId)}`); if (!response.ok) { throw new Error(t('textSplit.fetchChunkFailed')); } const data = await response.json(); setViewChunk(data); } catch (error) { console.error(t('textSplit.fetchChunkError'), error); setSnackbar({ open: true, message: error.message, severity: 'error' }); setViewDialogOpen(false); } }; // 关闭文本块详情对话框 const handleCloseViewDialog = () => { setViewDialogOpen(false); }; // 提交优化请求 const handleOptimize = async advice => { const model = getModelFromLocalStorage(); if (!model) { setSnackbar({ open: true, message: '请先选择模型，可以在顶部导航栏选择', severity: 'error' }); setOptimizeDialog(prev => ({ ...prev, open: false })); return; } try { setOptimizeDialog(prev => ({ ...prev, loading: true })); const language = i18n.language === 'zh-CN' ? '中文' : 'en'; const response = await fetch(`/api/projects/${projectId}/datasets/optimize`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ datasetId, model, advice, language }) }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || '优化失败'); } const data = await response.json(); // 更新数据集 setDataset(data.dataset); setAnswerValue(data.dataset.answer); setCotValue(data.dataset.cot || ''); setSnackbar({ open: true, message: 'AI智能优化成功', severity: 'success' }); } catch (error) { setSnackbar({ open: true, message: error.message || '优化失败', severity: 'error' }); } finally { setOptimizeDialog({ open: false, loading: false }); } }; if (loading) { return ( ); } if (!dataset) { return ( {t('datasets.noData')} ); } return ( {/* 顶部导航栏 */} {t('datasets.datasetDetail')} {t('datasets.stats', { total: datasets.length, confirmed: datasets.filter(d => d.confirmed).length, percentage: Math.round((datasets.filter(d => d.confirmed).length / datasets.length) * 100) })} handleNavigate('prev')}> handleNavigate('next')}> {/* 主要内容 */} {t('datasets.question')} {dataset.question} setEditingAnswer(true)} onChange={e => setAnswerValue(e.target.value)} onSave={() => handleSave('answer', answerValue)} onCancel={() => { setEditingAnswer(false); setAnswerValue(dataset.answer); }} onOptimize={handleOpenOptimizeDialog} /> setEditingCot(true)} onChange={e => setCotValue(e.target.value)} onSave={() => handleSave('cot', cotValue)} onCancel={() => { setEditingCot(false); setCotValue(dataset.cot || ''); }} /> {t('datasets.metadata')} {dataset.questionLabel && ( )} handleViewChunk(dataset.chunkId)} sx={{ cursor: 'pointer' }} /> {dataset.confirmed && ( )} setSnackbar(prev => ({ ...prev, open: false }))} anchorOrigin={{ vertical: 'bottom', horizontal: 'center' }} > setSnackbar(prev => ({ ...prev, open: false }))} severity={snackbar.severity} sx={{ width: '100%' }} > {snackbar.message} {/* AI优化对话框 */} {/* 文本块详情对话框 */} ); } ``` ## /app/projects/[projectId]/datasets/page.js ```js path="/app/projects/[projectId]/datasets/page.js" 'use client'; import { useState, useEffect } from 'react'; import { Container, Box, Typography, Button, IconButton, Paper, Table, TableBody, TableCell, TableContainer, TableHead, TableRow, CircularProgress, Alert, Snackbar, Chip, Dialog, DialogTitle, DialogContent, DialogActions, TablePagination, Card, Divider, useTheme, alpha, InputBase, Tooltip, Checkbox, LinearProgress } from '@mui/material'; import DeleteIcon from '@mui/icons-material/Delete'; import VisibilityIcon from '@mui/icons-material/Visibility'; import SearchIcon from '@mui/icons-material/Search'; import FileDownloadIcon from '@mui/icons-material/FileDownload'; import { useRouter } from 'next/navigation'; import ExportDatasetDialog from '@/components/ExportDatasetDialog'; import { useTranslation } from 'react-i18next'; import { processInParallel } from '@/lib/util/async'; // 数据集列表组件 const DatasetList = ({ datasets, onViewDetails, onDelete, page, rowsPerPage, onPageChange, onRowsPerPageChange, total, selectedIds, onSelectAll, onSelectItem }) => { const theme = useTheme(); const { t } = useTranslation(); const bgColor = theme.palette.mode === 'dark' ? theme.palette.primary.dark : theme.palette.primary.light; const color = theme.palette.mode === 'dark' ? theme.palette.getContrastText(theme.palette.primary.main) : theme.palette.getContrastText(theme.palette.primary.contrastText); return ( 0 && selectedIds.length < datasets.length} checked={datasets.length > 0 && selectedIds.length === datasets.length} onChange={onSelectAll} /> {t('datasets.question')} {t('datasets.createdAt')} {t('datasets.model')} {t('datasets.domainTag')} {t('datasets.cot')} {t('datasets.answer')} {/* {t('datasets.chunkId')} */} {t('common.actions')} {datasets.map((dataset, index) => ( onViewDetails(dataset.id)}> { e.stopPropagation(); onSelectItem(dataset.id); }} onClick={e => e.stopPropagation()} /> {dataset.confirmed && ( )} {dataset.question} {new Date(dataset.createdAt).toLocaleString('zh-CN')} {dataset.questionLabel ? ( ) : ( {t('datasets.noTag')} )} {dataset.answer} {/* {dataset.chunkId} */} { e.stopPropagation(); onViewDetails(dataset.id); }} sx={{ color: theme.palette.primary.main, '&:hover': { backgroundColor: alpha(theme.palette.primary.main, 0.1) } }}> { e.stopPropagation(); onDelete(dataset); }} sx={{ color: theme.palette.error.main, '&:hover': { backgroundColor: alpha(theme.palette.error.main, 0.1) } }}> ))} {datasets.length === 0 && ( {t('datasets.noData')} )}
t('datasets.pagination', { from, to, count })} sx={{ borderTop: `1px solid ${theme.palette.divider}`, '.MuiTablePagination-selectLabel, .MuiTablePagination-displayedRows': { fontWeight: 'medium' } }} /> ); }; // 删除确认对话框 const DeleteConfirmDialog = ({ open, datasets, onClose, onConfirm, batch, progress, deleting }) => { const theme = useTheme(); const { t } = useTranslation(); const dataset = datasets?.[0]; return ( {t('common.confirmDelete')} {batch ? t('datasets.batchconfirmDeleteMessage', { count: datasets.length }) : t('common.confirmDeleteDataSet')} {batch ? ( '' ) : ( {t('datasets.question')}： {dataset?.question} )} {deleting && progress ? ( {progress.percentage}% {t('datasets.batchDeleteProgress', { completed: progress.completed, total: progress.total })} {t('datasets.batchDeleteCount', { count: progress.datasetCount })} ) : ( '' )} ); }; // 主页面组件 export default function DatasetsPage({ params }) { const { projectId } = params; const router = useRouter(); const theme = useTheme(); const [datasets, setDatasets] = useState([]); const [loading, setLoading] = useState(true); const [snackbar, setSnackbar] = useState({ open: false, message: '', severity: 'success' }); const [deleteDialog, setDeleteDialog] = useState({ open: false, datasets: null, // 是否批量删除 batch: false, // 是否正在删除 deleting: false }); const [page, setPage] = useState(0); const [rowsPerPage, setRowsPerPage] = useState(10); const [searchQuery, setSearchQuery] = useState(''); const [exportDialog, setExportDialog] = useState({ open: false }); const [selectedIds, setselectedIds] = useState([]); const { t } = useTranslation(); // 删除进度状态 const [deleteProgress, setDeteleProgress] = useState({ total: 0, // 总删除问题数量 completed: 0, // 已删除完成的数量 percentage: 0 // 进度百分比 }); // 3. 添加打开导出对话框的处理函数 const handleOpenExportDialog = () => { setExportDialog({ open: true }); }; // 4. 添加关闭导出对话框的处理函数 const handleCloseExportDialog = () => { setExportDialog({ open: false }); }; // 获取数据集列表 const fetchDatasets = async () => { try { setLoading(true); const response = await fetch(`/api/projects/${projectId}/datasets`); if (!response.ok) throw new Error(t('datasets.fetchFailed')); const data = await response.json(); setDatasets(data); } catch (error) { setSnackbar({ open: true, message: error.message, severity: 'error' }); } finally { setLoading(false); } }; useEffect(() => { fetchDatasets(); }, [projectId]); // 处理页码变化 const handlePageChange = (event, newPage) => { setPage(newPage); }; // 处理每页行数变化 const handleRowsPerPageChange = event => { setRowsPerPage(parseInt(event.target.value, 10)); setPage(0); }; // 过滤数据 const filteredDatasets = datasets.filter( dataset => dataset.question.toLowerCase().includes(searchQuery.toLowerCase()) || (dataset.questionLabel && dataset.questionLabel.toLowerCase().includes(searchQuery.toLowerCase())) || dataset.answer.toLowerCase().includes(searchQuery.toLowerCase()) || dataset.chunkId.toLowerCase().includes(searchQuery.toLowerCase()) ); // 获取当前页的数据 const getCurrentPageData = () => { const start = page * rowsPerPage; const end = start + rowsPerPage; return filteredDatasets.slice(start, end); }; // 打开删除确认框 const handleOpenDeleteDialog = dataset => { setDeleteDialog({ open: true, datasets: [dataset] }); }; // 关闭删除确认框 const handleCloseDeleteDialog = () => { setDeleteDialog({ open: false, dataset: null }); }; const handleBatchDeleteDataset = async () => { setDeleteDialog({ open: true, datasets: datasets.filter(dataset => selectedIds.includes(dataset.id)), batch: true, count: selectedIds.length }); }; const resetProgress = () => { setDeteleProgress({ total: deleteDialog.count, completed: 0, percentage: 0 }); }; const handleDeleteConfirm = async () => { if (deleteDialog.batch) { setDeleteDialog({ ...deleteDialog, deleting: true }); await handleBatchDelete(); resetProgress(); } else { const [dataset] = deleteDialog.datasets; if (!dataset) return; await handleDelete(dataset); } // 刷新数据 fetchDatasets(); // 关闭确认框 handleCloseDeleteDialog(); }; // 批量删除数据集 const handleBatchDelete = async () => { // TODO: 并发删除存在问题，这里只能同时删除1个，待优化 await processInParallel(deleteDialog.datasets, handleDelete, 1, (cur, total) => { setDeteleProgress({ total: total, completed: cur, percentage: Math.floor((cur / total) * 100) }); }); }; // 删除数据集 const handleDelete = async dataset => { try { const response = await fetch(`/api/projects/${projectId}/datasets?id=${dataset.id}`, { method: 'DELETE' }); if (!response.ok) throw new Error(t('datasets.deleteFailed')); setSnackbar({ open: true, message: t('datasets.deleteSuccess'), severity: 'success' }); } catch (error) { setSnackbar({ open: true, message: error.message, severity: 'error' }); } }; // 导出数据集 const handleExportDatasets = exportOptions => { try { // 根据选项筛选数据 let dataToExport = [...filteredDatasets]; // 如果只导出已确认的数据集 if (exportOptions.confirmedOnly) { dataToExport = dataToExport.filter(dataset => dataset.confirmed); } // 根据选择的格式转换数据 let formattedData; // 不同文件格式 let mimeType = 'application/json'; if (exportOptions.formatType === 'alpaca') { formattedData = dataToExport.map(({ question, answer, cot }) => ({ instruction: question, input: '', output: cot && exportOptions.includeCOT ? `${cot}\n${answer}` : answer, system: exportOptions.systemPrompt || '' })); } else if (exportOptions.formatType === 'sharegpt') { formattedData = dataToExport.map(({ question, answer, cot }) => { const messages = []; // 添加系统提示词（如果有） if (exportOptions.systemPrompt) { messages.push({ role: 'system', content: exportOptions.systemPrompt }); } // 添加用户问题 messages.push({ role: 'user', content: question }); // 添加助手回答 messages.push({ role: 'assistant', content: cot && exportOptions.includeCOT ? `${cot}\n${answer}` : answer }); return { messages }; }); } else if (exportOptions.formatType === 'custom') { // 处理自定义格式 const { questionField, answerField, cotField, includeLabels, includeChunk } = exportOptions.customFields; formattedData = dataToExport.map(({ question, answer, cot, questionLabel: labels, chunkId }) => { const item = { [questionField]: question, [answerField]: answer }; // 如果有思维链且用户选择包含思维链，则添加思维链字段 if (cot && exportOptions.includeCOT && cotField) { item[cotField] = cot; } // 如果需要包含标签 if (includeLabels && labels && labels.length > 0) { item.label = labels.split(' ')[1]; } // 如果需要包含文本块 if (includeChunk && chunkId) { item.chunk = chunkId; } return item; }); } // 处理不同的文件格式 let content; let fileExtension; if (exportOptions.fileFormat === 'jsonl') { // JSONL 格式：每行一个 JSON 对象 content = formattedData.map(item => JSON.stringify(item)).join('\n'); fileExtension = 'jsonl'; } else if (exportOptions.fileFormat === 'csv') { // CSV 格式 const headers = Object.keys(formattedData[0] || {}); const csvRows = [ // 添加表头 headers.join(','), // 添加数据行 ...formattedData.map(item => headers .map(header => { // 处理包含逗号、换行符或双引号的字段 let field = item[header]?.toString() || ''; if (exportOptions.formatType === 'sharegpt') field = JSON.stringify(item[header]); if (field.includes(',') || field.includes('\n') || field.includes('"')) { field = `"${field.replace(/"/g, '""')}"`; } return field; }) .join(',') ) ]; content = csvRows.join('\n'); fileExtension = 'csv'; } else { // 默认 JSON 格式 content = JSON.stringify(formattedData, null, 2); fileExtension = 'json'; } // 创建 Blob 对象 const blob = new Blob([content], { type: mimeType || 'application/json' }); // 创建下载链接 const url = URL.createObjectURL(blob); const a = document.createElement('a'); a.href = url; const formatSuffix = exportOptions.formatType === 'alpaca' ? 'alpaca' : 'sharegpt'; a.download = `datasets-${projectId}-${formatSuffix}-${new Date().toISOString().slice(0, 10)}.${fileExtension}`; // 触发下载 document.body.appendChild(a); a.click(); // 清理 document.body.removeChild(a); URL.revokeObjectURL(url); // 关闭导出对话框 handleCloseExportDialog(); setSnackbar({ open: true, message: '数据集导出成功', severity: 'success' }); } catch (error) { setSnackbar({ open: true, message: '导出失败: ' + error.message, severity: 'error' }); } }; // 查看详情 const handleViewDetails = id => { router.push(`/projects/${projectId}/datasets/${id}`); }; // 关闭提示框 const handleCloseSnackbar = () => { setSnackbar(prev => ({ ...prev, open: false })); }; // 处理全选/取消全选 const handleSelectAll = event => { if (event.target.checked) { setselectedIds(getCurrentPageData().map(dataset => dataset.id)); } else { setselectedIds([]); } }; // 处理单个选择 const handleSelectItem = id => { setselectedIds(prev => { if (prev.includes(id)) { return prev.filter(item => item !== id); } else { return [...prev, id]; } }); }; if (loading) { return ( {t('datasets.loading')} ); } return ( {t('datasets.management')} {t('datasets.stats', { total: datasets.length, confirmed: datasets.filter(d => d.confirmed).length, percentage: datasets.length > 0 ? Math.round((datasets.filter(d => d.confirmed).length / datasets.length) * 100) : 0 })} { setSearchQuery(e.target.value); setPage(0); }} /> {selectedIds.length ? ( {t('datasets.selected', { count: selectedIds.length })} ) : ( '' )} {snackbar.message} ); } ``` ## /app/projects/[projectId]/layout.js ```js path="/app/projects/[projectId]/layout.js" 'use client'; import Navbar from '@/components/Navbar'; import { useState, useEffect } from 'react'; import { Box, CircularProgress, Typography, Button } from '@mui/material'; import { useRouter } from 'next/navigation'; import { useTranslation } from 'react-i18next'; export default function ProjectLayout({ children, params }) { const router = useRouter(); const { projectId } = params; const [projects, setProjects] = useState([]); const [currentProject, setCurrentProject] = useState(null); const [models, setModels] = useState([]); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); const [t] = useTranslation(); // 定义获取数据的函数 const fetchData = async () => { try { setLoading(true); // 获取用户创建的项目详情 const projectsResponse = await fetch(`/api/projects`); if (!projectsResponse.ok) { throw new Error(t('projects.fetchFailed')); } const projectsData = await projectsResponse.json(); setProjects(projectsData); // 获取当前项目详情 const projectResponse = await fetch(`/api/projects/${projectId}`); if (!projectResponse.ok) { // 如果项目不存在，跳转到首页 if (projectResponse.status === 404) { router.push('/'); return; } throw new Error('获取项目详情失败'); } const projectData = await projectResponse.json(); setCurrentProject(projectData); // 获取当前项目的模型配置 const modelsResponse = await fetch(`/api/projects/${projectId}/models`); if (modelsResponse.ok) { const modelsData = await modelsResponse.json(); if (modelsData && modelsData.length > 0) { // 将 API 返回的模型配置转换为 Navbar 需要的格式 const formattedModels = modelsData.map(model => ({ id: `${model.name}-${model.providerId}`, provider: model.provider, name: model.name, ...model })); setModels(formattedModels); } } else { console.warn('获取模型配置失败，使用默认配置'); // 如果项目有旧的模型配置，使用项目的模型配置 if (projectData.modelConfig && projectData.modelConfig.provider) { const customModel = { id: `${projectData.modelConfig.modelName}-${projectData.modelConfig.provider.toLowerCase()}`, provider: projectData.modelConfig.provider, name: projectData.modelConfig.modelName }; setModels([customModel]); } } } catch (error) { console.error('加载项目数据出错:', error); setError(error.message); } finally { setLoading(false); } }; // 初始加载数据 useEffect(() => { // 如果 projectId 是 undefined 或 "undefined"，直接重定向到首页 if (!projectId || projectId === 'undefined') { router.push('/'); return; } fetchData(); }, [projectId, router]); // 监听模型配置变化的事件 useEffect(() => { // 创建一个自定义事件监听器，用于在模型配置变化时刷新数据 const handleModelConfigChange = () => { console.log('检测到模型配置变化，重新获取模型数据'); // 使用一个标志来防止无限循环 fetchModelData(); }; // 监听模型选择变化的事件 const handleModelSelectionChange = () => { console.log('检测到模型选择变化'); // 如果需要在模型选择变化时执行特定操作，可以在这里添加 // 例如更新当前选中的模型或其他状态 }; // 添加事件监听器 window.addEventListener('model-config-changed', handleModelConfigChange); window.addEventListener('model-selection-changed', handleModelSelectionChange); // 清理函数 return () => { window.removeEventListener('model-config-changed', handleModelConfigChange); window.removeEventListener('model-selection-changed', handleModelSelectionChange); }; }, [projectId]); // 只获取模型数据，不获取项目数据，避免不必要的渲染 const fetchModelData = async () => { try { const modelsResponse = await fetch(`/api/projects/${projectId}/models`); if (!modelsResponse.ok) { throw new Error('获取模型数据失败'); } const modelsData = await modelsResponse.json(); if (modelsData && modelsData.length > 0) { // 将 API 返回的模型配置转换为 Navbar 需要的格式 const formattedModels = modelsData.map(model => ({ id: `${model.name}-${model.providerId}`, provider: model.provider, name: model.name, ...model })); setModels(formattedModels); } } catch (error) { console.error('获取模型数据出错:', error); // 不设置 error 状态，避免触发重新渲染 } }; if (loading) { return ( 加载项目数据... ); } if (error) { return ( {t('projects.fetchFailed')}: {error} ); } return ( <>
{children}
); } ``` ## /app/projects/[projectId]/page.js ```js path="/app/projects/[projectId]/page.js" 'use client'; import { useEffect } from 'react'; import { useRouter } from 'next/navigation'; export default function ProjectPage({ params }) { const router = useRouter(); const { projectId } = params; // 默认重定向到文本分割页面 useEffect(() => { router.push(`/projects/${projectId}/text-split`); }, [projectId, router]); return null; } ``` ## /app/projects/[projectId]/playground/page.js ```js path="/app/projects/[projectId]/playground/page.js" 'use client'; import React from 'react'; import { Box, Typography, Paper, Alert } from '@mui/material'; import { useParams } from 'next/navigation'; import { useTheme } from '@mui/material/styles'; import ChatArea from '@/components/playground/ChatArea'; import MessageInput from '@/components/playground/MessageInput'; import PlaygroundHeader from '@/components/playground/PlaygroundHeader'; import useModelPlayground from '@/hooks/useModelPlayground'; import { playgroundStyles } from '@/styles/playground'; import { useTranslation } from 'react-i18next'; export default function ModelPlayground() { const theme = useTheme(); const params = useParams(); const { projectId } = params; const styles = playgroundStyles(theme); const { t } = useTranslation(); const { availableModels, selectedModels, loading, userInput, conversations, error, outputMode, uploadedImage, handleModelSelection, handleInputChange, handleImageUpload, handleRemoveImage, handleSendMessage, handleClearConversations, handleOutputModeChange, getModelName } = useModelPlayground(projectId); return ( {t('playground.title')} {error && ( {error} )} ); } ``` ## /app/projects/[projectId]/questions/components/QuestionEditDialog.js ```js path="/app/projects/[projectId]/questions/components/QuestionEditDialog.js" 'use client'; import { useState, useEffect, useMemo } from 'react'; import { useTranslation } from 'react-i18next'; import { Dialog, DialogTitle, DialogContent, DialogActions, TextField, Button, Box, Autocomplete, TextField as MuiTextField } from '@mui/material'; export default function QuestionEditDialog({ open, onClose, onSubmit, initialData, chunks, tags, mode = 'create' // 'create' or 'edit' }) { const { t } = useTranslation(); // 获取文本块的标题 const getChunkTitle = chunkId => { const chunk = chunks.find(c => c.id === chunkId); return chunk?.filename || chunkId; // 直接使用文件名 }; const [formData, setFormData] = useState({ question: '', chunkId: '', label: '' // 默认不选中任何标签 }); useEffect(() => { if (initialData) { console.log('初始数据:', initialData); // 查看传入的初始数据 setFormData({ question: initialData.question || '', chunkId: initialData.chunkId || '', label: initialData.label || 'other' // 改用 label 而不是 label }); } else { setFormData({ question: '', chunkId: '', label: '' }); } }, [initialData]); const handleSubmit = () => { onSubmit(formData); onClose(); }; const flattenTags = (tags, prefix = '') => { let flatTags = []; const traverse = node => { flatTags.push({ id: node.label, // 使用标签名作为 id label: node.label, // 直接使用原始标签名 originalLabel: node.label }); if (node.child && node.child.length > 0) { node.child.forEach(child => traverse(child)); } }; tags.forEach(tag => traverse(tag)); flatTags.push({ id: 'other', label: t('datasets.uncategorized'), originalLabel: 'other' }); return flatTags; }; const flattenedTags = useMemo(() => flattenTags(tags), [tags, t]); useEffect(() => { if (initialData) { setFormData({ question: initialData.question || '', chunkId: initialData.chunkId || '', label: initialData.label || 'other' }); } else { setFormData({ question: '', chunkId: '', label: '' // 新建时默认为空 }); } }, [initialData]); // 修改 return 中的 Autocomplete 组件 { return tag.label; }} value={(() => { const foundTag = flattenedTags.find(tag => tag.id === formData.label); const defaultTag = flattenedTags.find(tag => tag.id === 'other'); return foundTag || defaultTag; })()} onChange={(e, newValue) => { setFormData({ ...formData, label: newValue ? newValue.id : 'other' }); }} renderInput={params => ( )} />; return ( {mode === 'create' ? t('questions.createQuestion') : t('questions.editQuestion')} setFormData({ ...formData, question: e.target.value })} /> getChunkTitle(chunk.id)} value={chunks.find(chunk => chunk.id === formData.chunkId) || null} onChange={(e, newValue) => setFormData({ ...formData, chunkId: newValue ? newValue.id : '' })} renderInput={params => ( )} /> tag.label} value={flattenedTags.find(tag => tag.id === formData.label) || null} onChange={(e, newValue) => setFormData({ ...formData, label: newValue ? newValue.id : '' })} renderInput={params => ( )} /> ); } ``` ## /app/projects/[projectId]/questions/hooks/useQuestionEdit.js ```js path="/app/projects/[projectId]/questions/hooks/useQuestionEdit.js" 'use client'; import { useState } from 'react'; import { useTranslation } from 'react-i18next'; import request from '@/lib/util/request'; export function useQuestionEdit(projectId, onSuccess) { const { t } = useTranslation(); const [editDialogOpen, setEditDialogOpen] = useState(false); const [editMode, setEditMode] = useState('create'); const [editingQuestion, setEditingQuestion] = useState(null); const handleOpenCreateDialog = () => { setEditMode('create'); setEditingQuestion(null); setEditDialogOpen(true); }; const handleOpenEditDialog = question => { setEditMode('edit'); setEditingQuestion(question); setEditDialogOpen(true); }; const handleCloseDialog = () => { setEditDialogOpen(false); setEditingQuestion(null); }; const handleSubmitQuestion = async formData => { try { const response = await request(`/api/projects/${projectId}/questions`, { method: editMode === 'create' ? 'POST' : 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify( editMode === 'create' ? { question: formData.question, chunkId: formData.chunkId, label: formData.label } : { question: formData.question, oldQuestion: editingQuestion.question, chunkId: formData.chunkId, label: formData.label, oldChunkId: editingQuestion.chunkId } ) }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('questions.operationFailed')); } // 获取更新后的问题数据 const updatedQuestion = await response.json(); // 直接更新问题列表中的数据，而不是重新获取整个列表 if (onSuccess) { onSuccess(updatedQuestion); } handleCloseDialog(); } catch (error) { console.error('操作失败:', error); } }; return { editDialogOpen, editMode, editingQuestion, handleOpenCreateDialog, handleOpenEditDialog, handleCloseDialog, handleSubmitQuestion }; } ``` ## /app/projects/[projectId]/questions/page.js ```js path="/app/projects/[projectId]/questions/page.js" 'use client'; import { useState, useEffect } from 'react'; import { useTranslation } from 'react-i18next'; import { Container, Typography, Box, Button, Paper, Tabs, Tab, CircularProgress, Divider, Checkbox, Snackbar, Alert, TextField, InputAdornment, Stack, Dialog, DialogActions, DialogContent, DialogContentText, DialogTitle, LinearProgress, Select, MenuItem, useTheme } from '@mui/material'; import AutoFixHighIcon from '@mui/icons-material/AutoFixHigh'; import DeleteIcon from '@mui/icons-material/Delete'; import i18n from '@/lib/i18n'; import SearchIcon from '@mui/icons-material/Search'; import AddIcon from '@mui/icons-material/Add'; import QuestionListView from '@/components/questions/QuestionListView'; import QuestionTreeView from '@/components/questions/QuestionTreeView'; import TabPanel from '@/components/text-split/components/TabPanel'; import request from '@/lib/util/request'; import useTaskSettings from '@/hooks/useTaskSettings'; import QuestionEditDialog from './components/QuestionEditDialog'; import { useQuestionEdit } from './hooks/useQuestionEdit'; import { useSnackbar } from '@/hooks/useSnackbar'; export default function QuestionsPage({ params }) { const { t } = useTranslation(); const theme = useTheme(); const { projectId } = params; const [loading, setLoading] = useState(true); const [error, setError] = useState(null); const [questions, setQuestions] = useState([]); const [tags, setTags] = useState([]); const [chunks, setChunks] = useState([]); const [activeTab, setActiveTab] = useState(0); const [selectedQuestions, setSelectedQuestions] = useState([]); const [searchTerm, setSearchTerm] = useState(''); const [processing, setProcessing] = useState(false); const [answerFilter, setAnswerFilter] = useState('all'); // 'all', 'answered', 'unanswered' const [snackbar, setSnackbar] = useState({ open: false, message: '', severity: 'success' }); const { taskSettings } = useTaskSettings(projectId); // 进度状态 const [progress, setProgress] = useState({ total: 0, // 总共选择的问题数量 completed: 0, // 已处理完成的数量 percentage: 0, // 进度百分比 datasetCount: 0 // 已生成的数据集数量 }); const { showSuccess, SnackbarComponent } = useSnackbar(); const { editDialogOpen, editMode, editingQuestion, handleOpenCreateDialog, handleOpenEditDialog, handleCloseDialog, handleSubmitQuestion } = useQuestionEdit(projectId, updatedQuestion => { // 直接更新 questions 数组中的数据 setQuestions(prevQuestions => { if (editMode === 'create') { return [...prevQuestions, updatedQuestion]; } else { return prevQuestions.map(q => (q.question === editingQuestion.question ? updatedQuestion : q)); } }); showSuccess(t('questions.operationSuccess')); }); const [confirmDialog, setConfirmDialog] = useState({ open: false, title: '', content: '', confirmAction: null }); const fetchData = async currentPage => { if (!currentPage) { setLoading(true); } try { // 获取标签树 const tagsResponse = await fetch(`/api/projects/${projectId}/tags`); if (!tagsResponse.ok) { throw new Error(t('common.fetchError')); } const tagsData = await tagsResponse.json(); setTags(tagsData.tags || []); // 获取问题列表 const questionsResponse = await fetch(`/api/projects/${projectId}/questions`); if (!questionsResponse.ok) { throw new Error(t('common.fetchError')); } const questionsData = await questionsResponse.json(); setQuestions(questionsData || []); // 获取文本块列表 const response = await fetch(`/api/projects/${projectId}/split`); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('common.fetchError')); } const data = await response.json(); setChunks(data.chunks || []); } catch (error) { console.error(t('common.fetchError'), error); setError(error.message); setSnackbar({ open: true, message: error.message, severity: 'error' }); } finally { if (!currentPage) { setLoading(false); } } }; // 获取所有数据 useEffect(() => { fetchData(); }, [projectId]); // 处理标签页切换 const handleTabChange = (event, newValue) => { setActiveTab(newValue); }; // 处理问题选择 const handleSelectQuestion = (questionKey, newSelected) => { if (newSelected) { // 处理批量选择的情况 setSelectedQuestions(newSelected); } else { // 处理单个问题选择的情况 setSelectedQuestions(prev => { if (prev.includes(questionKey)) { return prev.filter(id => id !== questionKey); } else { return [...prev, questionKey]; } }); } }; // 全选/取消全选 const handleSelectAll = () => { if (selectedQuestions.length > 0) { setSelectedQuestions([]); } else { const filteredQuestions = questions.filter(question => { const matchesSearch = searchTerm === '' || question.question.toLowerCase().includes(searchTerm.toLowerCase()) || (question.label && question.label.toLowerCase().includes(searchTerm.toLowerCase())); let matchesAnswerFilter = true; if (answerFilter === 'answered') { matchesAnswerFilter = question.dataSites && question.dataSites.length > 0; } else if (answerFilter === 'unanswered') { matchesAnswerFilter = !question.dataSites || question.dataSites.length === 0; } return matchesSearch && matchesAnswerFilter; }); const filteredQuestionKeys = filteredQuestions.map(question => JSON.stringify({ question: question.question, chunkId: question.chunkId }) ); setSelectedQuestions(filteredQuestionKeys); } }; // 从本地存储获取模型参数 const getModelFromLocalStorage = () => { if (typeof window === 'undefined') return null; try { // 从 localStorage 获取当前选择的模型信息 let model = null; // 尝试从 localStorage 获取完整的模型信息 const modelInfoStr = localStorage.getItem('selectedModelInfo'); if (modelInfoStr) { try { model = JSON.parse(modelInfoStr); } catch (e) { console.error(t('models.parseError'), e); return null; } } // 如果没有模型 ID 或模型信息，返回 null if (!model) { return null; } return model; } catch (error) { console.error(t('models.configNotFound'), error); return null; } }; // 生成单个问题的数据集 const handleGenerateDataset = async (questionId, chunkId) => { try { // 获取模型参数 const model = getModelFromLocalStorage(); if (!model) { setSnackbar({ open: true, message: t('models.configNotFound'), severity: 'error' }); return null; } // 显示处理中提示 setSnackbar({ open: true, message: t('datasets.generating'), severity: 'info' }); // 调用API生成数据集 const currentLanguage = i18n.language === 'zh-CN' ? '中文' : 'en'; const response = await request(`/api/projects/${projectId}/datasets`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ questionId, chunkId, model, language: currentLanguage }) }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('datasets.generateFailed')); } const result = await response.json(); setSnackbar({ open: false }); fetchData(1); return result.dataset; } catch (error) { console.error(t('datasets.generateError'), error); setSnackbar({ open: true, message: error.message || t('datasets.generateFailed'), severity: 'error' }); return null; } }; // 并行处理数组的辅助函数，限制并发数 const processInParallel = async (items, processFunction, concurrencyLimit) => { const results = []; const inProgress = new Set(); const queue = [...items]; while (queue.length > 0 || inProgress.size > 0) { // 如果有空闲槽位且队列中还有任务，启动新任务 while (inProgress.size < concurrencyLimit && queue.length > 0) { const item = queue.shift(); const promise = processFunction(item).then(result => { inProgress.delete(promise); return result; }); inProgress.add(promise); results.push(promise); } // 等待其中一个任务完成 if (inProgress.size > 0) { await Promise.race(inProgress); } } return Promise.all(results); }; const handleBatchGenerateAnswers = async () => { if (selectedQuestions.length === 0) { setSnackbar({ open: true, message: t('questions.noQuestionsSelected'), severity: 'warning' }); return; } // 获取模型参数 const model = getModelFromLocalStorage(); if (!model) { setSnackbar({ open: true, message: t('models.configNotFound'), severity: 'error' }); return; } try { setProgress({ total: selectedQuestions.length, completed: 0, percentage: 0, datasetCount: 0 }); // 然后设置处理状态为真，确保进度条显示 setProcessing(true); setSnackbar({ open: true, message: t('questions.batchGenerateStart', { count: selectedQuestions.length }), severity: 'info' }); // 单个问题处理函数 const processQuestion = async key => { try { // 从问题键中提取 chunkId 和 questionId const lastDashIndex = key.lastIndexOf('-'); if (lastDashIndex === -1) { console.error(t('questions.invalidQuestionKey'), key); // 更新进度状态（即使失败也计入已处理） setProgress(prev => { const completed = prev.completed + 1; const percentage = Math.round((completed / prev.total) * 100); return { ...prev, completed, percentage }; }); return { success: false, key, error: t('questions.invalidQuestionKey') }; } const { question: questionId, chunkId } = JSON.parse(key); console.log('开始生成数据集:', { chunkId, questionId }); const language = i18n.language === 'zh-CN' ? '中文' : 'en'; // 调用API生成数据集 const response = await request(`/api/projects/${projectId}/datasets`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ questionId, chunkId, model, language }) }); if (!response.ok) { const errorData = await response.json(); console.error(t('datasets.generateError'), errorData.error || t('datasets.generateFailed')); // 更新进度状态（即使失败也计入已处理） setProgress(prev => { const completed = prev.completed + 1; const percentage = Math.round((completed / prev.total) * 100); return { ...prev, completed, percentage }; }); return { success: false, key, error: errorData.error || t('datasets.generateFailed') }; } const data = await response.json(); // 更新进度状态 setProgress(prev => { const completed = prev.completed + 1; const percentage = Math.round((completed / prev.total) * 100); const datasetCount = prev.datasetCount + 1; return { ...prev, completed, percentage, datasetCount }; }); console.log(`数据集生成成功: ${questionId}`); return { success: true, key, data: data.dataset }; } catch (error) { console.error('生成数据集失败:', error); // 更新进度状态（即使失败也计入已处理） setProgress(prev => { const completed = prev.completed + 1; const percentage = Math.round((completed / prev.total) * 100); return { ...prev, completed, percentage }; }); return { success: false, key, error: error.message }; } }; // 并行处理所有问题，最多同时处理2个 const results = await processInParallel(selectedQuestions, processQuestion, taskSettings.concurrencyLimit); // 刷新数据 fetchData(1); // 处理完成后设置结果消息 const successCount = results.filter(r => r.success).length; const failCount = results.filter(r => !r.success).length; if (failCount > 0) { setSnackbar({ open: true, message: t('datasets.partialSuccess', { successCount, total: selectedQuestions.length, failCount }), severity: 'warning' }); } else { setSnackbar({ open: true, message: t('common.success', { successCount }), severity: 'success' }); } } catch (error) { console.error('生成数据集出错:', error); setSnackbar({ open: true, message: error.message || '生成数据集失败', severity: 'error' }); } finally { // 延迟关闭处理状态，确保用户可以看到完成的进度 setTimeout(() => { setProcessing(false); // 再次延迟重置进度状态 setTimeout(() => { setProgress({ total: 0, completed: 0, percentage: 0, datasetCount: 0 }); }, 500); }, 2000); // 延迟关闭处理状态，让用户看到完成的进度 } }; // 关闭提示框 const handleCloseSnackbar = () => { setSnackbar(prev => ({ ...prev, open: false })); }; // 处理删除问题 const confirmDeleteQuestion = (questionId, chunkId) => { // 根据 questionId 找到对应的问题对象 const question = questions.find(q => q.question === questionId && q.chunkId === chunkId); const questionText = question ? question.question : questionId; // 显示确认对话框 setConfirmDialog({ open: true, title: t('common.confirmDelete'), content: t('common.confirmDeleteQuestion'), confirmAction: () => executeDeleteQuestion(questionId, chunkId) }); }; // 执行删除问题的操作 const executeDeleteQuestion = async (questionId, chunkId) => { try { // 显示删除中的提示 setSnackbar({ open: true, message: t('common.deleting'), severity: 'info' }); // 调用删除问题的 API const response = await fetch(`/api/projects/${projectId}/questions/${encodeURIComponent(questionId)}`, { method: 'DELETE', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ chunkId }) }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || '删除问题失败'); } // 从列表中移除已删除的问题 setQuestions(prev => prev.filter(q => !(q.question === questionId && q.chunkId === chunkId))); // 从选中列表中移除已删除的问题 const questionKey = JSON.stringify({ question: questionId, chunkId }); setSelectedQuestions(prev => prev.filter(id => id !== questionKey)); // 显示成功提示 setSnackbar({ open: true, message: t('common.deleteSuccess'), severity: 'success' }); } catch (error) { console.error('删除问题失败:', error); setSnackbar({ open: true, message: error.message || '删除问题失败', severity: 'error' }); } }; // 处理删除问题的入口函数 const handleDeleteQuestion = (questionId, chunkId) => { confirmDeleteQuestion(questionId, chunkId); }; // 确认批量删除问题 const confirmBatchDeleteQuestions = () => { if (selectedQuestions.length === 0) { setSnackbar({ open: true, message: '请先选择问题', severity: 'warning' }); return; } // 显示确认对话框 setConfirmDialog({ open: true, title: '确认批量删除问题', content: `您确定要删除选中的 ${selectedQuestions.length} 个问题吗？此操作不可恢复。`, confirmAction: executeBatchDeleteQuestions }); }; // 执行批量删除问题 const executeBatchDeleteQuestions = async () => { try { // 显示删除中的提示 setSnackbar({ open: true, message: `正在删除 ${selectedQuestions.length} 个问题...`, severity: 'info' }); // 存储成功删除的问题数量 let successCount = 0; // 逐个删除问题，完全模仿单个删除的逻辑 for (const key of selectedQuestions) { try { const { question: questionId, chunkId } = JSON.parse(key); console.log('开始删除问题:', { chunkId, questionId }); // 调用删除问题的 API const response = await fetch(`/api/projects/${projectId}/questions/${encodeURIComponent(questionId)}`, { method: 'DELETE', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ chunkId }) }); if (!response.ok) { const errorData = await response.json(); console.error(`删除问题失败:`, errorData.error || '删除问题失败'); continue; } // 从列表中移除已删除的问题，完全复制单个删除的逻辑 setQuestions(prev => prev.filter(q => !(q.question === questionId && q.chunkId === chunkId))); successCount++; console.log(`问题删除成功: ${questionId}`); } catch (error) { console.error('删除问题失败:', error); } } // 清空选中列表 setSelectedQuestions([]); // 显示成功提示 setSnackbar({ open: true, message: successCount === selectedQuestions.length ? `成功删除 ${successCount} 个问题` : `删除完成，成功: ${successCount}, 失败: ${selectedQuestions.length - successCount}`, severity: successCount === selectedQuestions.length ? 'success' : 'warning' }); } catch (error) { console.error('批量删除问题失败:', error); setSnackbar({ open: true, message: error.message || '批量删除问题失败', severity: 'error' }); } }; // 处理批量删除问题的入口函数 const handleBatchDeleteQuestions = () => { confirmBatchDeleteQuestions(); }; // 获取文本块内容 const getChunkContent = chunkId => { const chunk = chunks.find(c => c.id === chunkId); return chunk ? chunk.content : ''; }; if (loading) { return ( ); } if (error) { return ( {error} ); } // 计算问题总数 const totalQuestions = questions.length; return ( {/* 处理中的进度显示 - 全局蒙版样式 */} {processing && ( {t('datasets.generatingDataset')} {progress.percentage}% {t('questions.generatingProgress', { completed: progress.completed, total: progress.total })} {t('questions.generatedCount', { count: progress.datasetCount })} {t('questions.pleaseWait')} )} {t('questions.title')} ( { questions.filter(question => { const matchesSearch = searchTerm === '' || question.question.toLowerCase().includes(searchTerm.toLowerCase()) || (question.label && question.label.toLowerCase().includes(searchTerm.toLowerCase())); let matchesAnswerFilter = true; if (answerFilter === 'answered') { matchesAnswerFilter = question.dataSites && question.dataSites.length > 0; } else if (answerFilter === 'unanswered') { matchesAnswerFilter = !question.dataSites || question.dataSites.length === 0; } return matchesSearch && matchesAnswerFilter; }).length } ) 0 && selectedQuestions.length === totalQuestions} indeterminate={selectedQuestions.length > 0 && selectedQuestions.length < totalQuestions} onChange={handleSelectAll} /> {selectedQuestions.length > 0 ? t('questions.selectedCount', { count: selectedQuestions.length }) : t('questions.selectAll')} ( {t('questions.totalCount', { count: questions.filter(question => { const matchesSearch = searchTerm === '' || question.question.toLowerCase().includes(searchTerm.toLowerCase()) || (question.label && question.label.toLowerCase().includes(searchTerm.toLowerCase())); let matchesAnswerFilter = true; if (answerFilter === 'answered') { matchesAnswerFilter = question.dataSites && question.dataSites.length > 0; } else if (answerFilter === 'unanswered') { matchesAnswerFilter = !question.dataSites || question.dataSites.length === 0; } return matchesSearch && matchesAnswerFilter; }).length })} ) setSearchTerm(e.target.value)} InputProps={{ startAdornment: ( ) }} /> { // 搜索词筛选 const matchesSearch = searchTerm === '' || question.question.toLowerCase().includes(searchTerm.toLowerCase()) || (question.label && question.label.toLowerCase().includes(searchTerm.toLowerCase())); // 答案状态筛选 let matchesAnswerFilter = true; if (answerFilter === 'answered') { matchesAnswerFilter = question.dataSites && question.dataSites.length > 0; } else if (answerFilter === 'unanswered') { matchesAnswerFilter = !question.dataSites || question.dataSites.length === 0; } return matchesSearch && matchesAnswerFilter; })} chunks={chunks} selectedQuestions={selectedQuestions} onSelectQuestion={handleSelectQuestion} onDeleteQuestion={handleDeleteQuestion} onGenerateDataset={handleGenerateDataset} onEditQuestion={handleOpenEditDialog} projectId={projectId} /> { // 搜索词筛选 const matchesSearch = searchTerm === '' || question.question.toLowerCase().includes(searchTerm.toLowerCase()) || (question.label && question.label.toLowerCase().includes(searchTerm.toLowerCase())); // 答案状态筛选 let matchesAnswerFilter = true; if (answerFilter === 'answered') { matchesAnswerFilter = question.dataSites && question.dataSites.length > 0; } else if (answerFilter === 'unanswered') { matchesAnswerFilter = !question.dataSites || question.dataSites.length === 0; } return matchesSearch && matchesAnswerFilter; })} chunks={chunks} tags={tags} selectedQuestions={selectedQuestions} onSelectQuestion={handleSelectQuestion} onDeleteQuestion={handleDeleteQuestion} onGenerateDataset={handleGenerateDataset} onEditQuestion={handleOpenEditDialog} projectId={projectId} /> {snackbar.message} {/* 确认对话框 */} setConfirmDialog({ ...confirmDialog, open: false })} aria-labelledby="alert-dialog-title" aria-describedby="alert-dialog-description" > {confirmDialog.title} {confirmDialog.content} ); } ``` ## /app/projects/[projectId]/settings/components/PromptSettings.js ```js path="/app/projects/[projectId]/settings/components/PromptSettings.js" import React, { useState, useEffect } from 'react'; import { Box, TextField, Button, Typography, Alert, Grid, Card, CardContent } from '@mui/material'; import SaveIcon from '@mui/icons-material/Save'; import { useTranslation } from 'react-i18next'; import fetchWithRetry from '@/lib/util/request'; import { useSnackbar } from '@/hooks/useSnackbar'; export default function PromptSettings({ projectId }) { const { t } = useTranslation(); const { showSuccess, showError, SnackbarComponent } = useSnackbar(); const [prompts, setPrompts] = useState({ globalPrompt: '', questionPrompt: '', answerPrompt: '', labelPrompt: '', domainTreePrompt: '' }); const [loading, setLoading] = useState(false); // 加载提示词配置 useEffect(() => { const loadPrompts = async () => { try { const response = await fetchWithRetry(`/api/projects/${projectId}/config`); const config = await response.json(); // 提取提示词相关的字段 const promptFields = { globalPrompt: config.globalPrompt || '', questionPrompt: config.questionPrompt || '', answerPrompt: config.answerPrompt || '', labelPrompt: config.labelPrompt || '', domainTreePrompt: config.domainTreePrompt || '' }; setPrompts(promptFields); } catch (error) { console.error('加载提示词配置失败:', error); showError(t('settings.loadPromptsFailed')); } }; loadPrompts(); }, [projectId]); // 保存提示词配置 const handleSave = async () => { setLoading(true); try { const response = await fetchWithRetry(`/api/projects/${projectId}/config`, { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ prompts: { ...prompts } }) }); const data = await response.json(); if (response.ok) { showSuccess(t('settings.savePromptsSuccess')); } else { throw new Error(data.error || '保存失败'); } } catch (error) { console.error('保存提示词配置失败:', error); showError(t('settings.savePromptsFailed')); } finally { setLoading(false); } }; const handleChange = field => event => { setPrompts(prev => ({ ...prev, [field]: event.target.value })); }; return ( {t('settings.promptsDescription')} {/* 全局提示词 */} {t('settings.globalPrompt')} {/* 生成问题提示词 */} {t('settings.questionPrompt')} {/* 生成答案提示词 */} {t('settings.answerPrompt')} {/* 问题打标提示词 */} {t('settings.labelPrompt')} {/* 构建领域树提示词 */} {t('settings.domainTreePrompt')} ); } ``` ## /app/projects/[projectId]/settings/page.js ```js path="/app/projects/[projectId]/settings/page.js" 'use client'; import { useState, useEffect } from 'react'; import { Container, Typography, Box, Tabs, Tab, Paper, Alert, CircularProgress } from '@mui/material'; import { useSearchParams, useRouter } from 'next/navigation'; import { useTranslation } from 'react-i18next'; // 导入设置组件 import BasicSettings from '@/components/settings/BasicSettings'; import ModelSettings from '@/components/settings/ModelSettings'; import TaskSettings from '@/components/settings/TaskSettings'; import PromptSettings from './components/PromptSettings'; // 定义 TAB 枚举 const TABS = { BASIC: 'basic', MODEL: 'model', TASK: 'task', PROMPTS: 'prompts' }; export default function SettingsPage({ params }) { const { t } = useTranslation(); const { projectId } = params; const searchParams = useSearchParams(); const router = useRouter(); const [activeTab, setActiveTab] = useState(TABS.BASIC); const [projectExists, setProjectExists] = useState(true); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); // 从 URL hash 中获取当前 tab useEffect(() => { const tab = searchParams.get('tab'); if (tab && Object.values(TABS).includes(tab)) { setActiveTab(tab); } }, [searchParams]); // 检查项目是否存在 useEffect(() => { async function checkProject() { try { setLoading(true); const response = await fetch(`/api/projects/${projectId}`); if (!response.ok) { if (response.status === 404) { setProjectExists(false); } else { throw new Error(t('projects.fetchFailed')); } } else { setProjectExists(true); } } catch (error) { console.error('获取项目详情出错:', error); setError(error.message); } finally { setLoading(false); } } checkProject(); }, [projectId, t]); // 处理 tab 切换 const handleTabChange = (event, newValue) => { setActiveTab(newValue); // 更新 URL hash router.push(`/projects/${projectId}/settings?tab=${newValue}`); }; if (loading) { return ( ); } if (!projectExists) { return ( {t('projects.notExist')} ); } if (error) { return ( {error} ); } return ( {t('settings.title')} {activeTab === TABS.BASIC && } {activeTab === TABS.MODEL && } {activeTab === TABS.TASK && } {activeTab === TABS.PROMPTS && } ); } ``` ## /app/projects/[projectId]/text-split/page.js ```js path="/app/projects/[projectId]/text-split/page.js" 'use client'; import { useState, useEffect } from 'react'; import { useTranslation } from 'react-i18next'; import i18n from '@/lib/i18n'; import { Container, Typography, Box, Tabs, Tab, Alert, CircularProgress, Snackbar, Backdrop, Paper, LinearProgress, Select, MenuItem } from '@mui/material'; import FileUploader from '@/components/text-split/FileUploader'; import ChunkList from '@/components/text-split/ChunkList'; import DomainAnalysis from '@/components/text-split/DomainAnalysis'; import request from '@/lib/util/request'; import { processInParallel } from '@/lib/util/async'; import useTaskSettings from '@/hooks/useTaskSettings'; import { finished } from 'stream'; export default function TextSplitPage({ params }) { const { t } = useTranslation(); const { projectId } = params; const [activeTab, setActiveTab] = useState(0); const [chunks, setChunks] = useState([]); const [showChunks, setShowChunks] = useState([]); const [tocData, setTocData] = useState(''); const [tags, setTags] = useState([]); const [loading, setLoading] = useState(false); const [processing, setProcessing] = useState(false); const [pdfProcessing, setPdfProcessing] = useState(false); const [error, setError] = useState(null); // 可以是字符串或对象 { severity, message } const {taskSettings } = useTaskSettings(projectId); const [pdfStrategy,setPdfStrategy]= useState("default"); const [questionFilter, setQuestionFilter] = useState('all'); // 'all', 'generated', 'ungenerated' const [selectedViosnModel,setSelectedViosnModel]= useState(''); // 进度状态 const [progress, setProgress] = useState({ total: 0, // 总共选择的文本块数量 completed: 0, // 已处理完成的数量 percentage: 0, // 进度百分比 questionCount: 0 // 已生成的问题数量 }); // 加载文本块数据 useEffect(() => { fetchChunks(); }, []); // 获取文本块列表 const fetchChunks = async () => { try { setLoading(true); const response = await fetch(`/api/projects/${projectId}/split`); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('textSplit.fetchChunksFailed')); } const data = await response.json(); setChunks(data.chunks || []); // Apply filter when setting showChunks const filteredChunks = (data.chunks || []).filter(chunk => { if (questionFilter === 'generated') { return chunk.questions && chunk.questions.length > 0; } else if (questionFilter === 'ungenerated') { return !chunk.questions || chunk.questions.length === 0; } return true; }); setShowChunks(filteredChunks); // 如果有文件结果，处理详细信息 if (data.toc) { console.log(t('textSplit.fileResultReceived'), data.fileResult); // 如果有目录结构，设置目录数据 setTocData(data.toc); } // 如果有标签，设置标签数据 if (data.tags) { setTags(data.tags); } } catch (error) { console.error(t('textSplit.fetchChunksError'), error); setError(error.message); } finally { setLoading(false); } }; // 处理标签切换 const handleTabChange = (event, newValue) => { setActiveTab(newValue); }; // 处理文件上传成功 const handleUploadSuccess = async (fileNames, model,pdfFiles) => { console.log(t('textSplit.fileUploadSuccess'), fileNames); //上传完处理PDF文件 try{ setPdfProcessing(true); setError(null); // 重置进度状态 setProgress({ total: pdfFiles.length, completed: 0, percentage: 0, questionCount: 0 }); const currentLanguage = i18n.language === 'zh-CN' ? '中文' : 'en'; for(const file of pdfFiles){ const response = await fetch(`/api/projects/${projectId}/pdf?fileName=`+file.name+`&strategy=`+pdfStrategy+`¤tLanguage=`+currentLanguage+`&modelId=`+selectedViosnModel); if (!response.ok) { const errorData = await response.json(); throw new Error(t('textSplit.pdfProcessingFailed') + errorData.error); } const data = await response.json(); // 更新进度状态 setProgress(prev => { const completed = prev.completed + 1; const percentage = Math.round((completed / prev.total) * 100); return { ...prev, completed, percentage }; }); } }catch(error){ console.error(t('textSplit.pdfProcessingFailed'), error); setError({ severity: 'error', message: error.message }); }finally{ setPdfProcessing(false); // 重置进度状态 setTimeout(() => { setProgress({ total: 0, completed: 0, percentage: 0, questionCount: 0 }); }, 1000); // 延迟重置，让用户看到完成的进度 } // 如果有文件上传成功，自动处理第一个文件 if (fileNames && fileNames.length > 0) { handleSplitText(fileNames[0], model); } }; // 处理文本分割 const handleSplitText = async (fileName, model) => { try { setProcessing(true); const language = i18n.language === 'zh-CN' ? '中文' : 'en'; const response = await fetch(`/api/projects/${projectId}/split`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ fileName, model, language }) }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('textSplit.splitTextFailed')); } const data = await response.json(); // 更新文本块列表 setChunks(prev => { const newChunks = [...prev]; data.chunks.forEach(chunk => { if (!newChunks.find(c => c.id === chunk.id)) { newChunks.push(chunk); } }); return newChunks; }); setShowChunks(prev => { const newChunks = [...prev]; data.chunks.forEach(chunk => { if (!newChunks.find(c => c.id === chunk.id)) { newChunks.push(chunk); } }); return newChunks; }); // 更新目录结构 if (data.toc) { setTocData(data.toc); } // 自动切换到智能分割标签 setActiveTab(0); location.reload(); } catch (error) { console.error(t('textSplit.splitTextError'), error); setError(error.message); } finally { setProcessing(false); } }; // 处理删除文本块 const handleDeleteChunk = async chunkId => { try { const response = await fetch(`/api/projects/${projectId}/chunks/${encodeURIComponent(chunkId)}`, { method: 'DELETE' }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('textSplit.deleteChunkFailed')); } // 更新文本块列表 setChunks(prev => prev.filter(chunk => chunk.id !== chunkId)); setShowChunks(prev => prev.filter(chunk => chunk.id !== chunkId)); } catch (error) { console.error(t('textSplit.deleteChunkError'), error); setError(error.message); } }; // 处理生成问题 const handleGenerateQuestions = async chunkIds => { try { setProcessing(true); setError(null); // 重置进度状态 setProgress({ total: chunkIds.length, completed: 0, percentage: 0, questionCount: 0 }); let model = null; // 尝试从 localStorage 获取完整的模型信息 const modelInfoStr = localStorage.getItem('selectedModelInfo'); if (modelInfoStr) { try { model = JSON.parse(modelInfoStr); } catch (e) { console.error('解析模型信息出错:', e); // 继续执行，将在下面尝试获取模型信息 } } // 如果仍然没有模型信息，抛出错误 if (!model) { throw new Error(t('textSplit.selectModelFirst')); } // 如果是单个文本块，直接调用单个生成接口 if (chunkIds.length === 1) { const chunkId = chunkIds[0]; // 获取当前语言环境 const currentLanguage = i18n.language === 'zh-CN' ? '中文' : 'en'; const response = await request(`/api/projects/${projectId}/chunks/${encodeURIComponent(chunkId)}/questions`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ model, language: currentLanguage }) }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('textSplit.generateQuestionsFailed', { chunkId })); } const data = await response.json(); console.log(t('textSplit.questionsGenerated', { chunkId, total: data.total })); setError({ severity: 'success', message: t('textSplit.questionsGeneratedSuccess', { total: data.total }) }); } else { // 如果是多个文本块，循环调用单个文本块的问题生成接口，限制并行数为2 let totalQuestions = 0; let successCount = 0; let errorCount = 0; // 单个文本块处理函数 const processChunk = async chunkId => { try { // 获取当前语言环境 const currentLanguage = i18n.language === 'zh-CN' ? '中文' : 'en'; const response = await request( `/api/projects/${projectId}/chunks/${encodeURIComponent(chunkId)}/questions`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ model, language: currentLanguage }) } ); if (!response.ok) { const errorData = await response.json(); console.error(t('textSplit.generateQuestionsForChunkFailed', { chunkId }), errorData.error); errorCount++; return { success: false, chunkId, error: errorData.error }; } const data = await response.json(); console.log(t('textSplit.questionsGenerated', { chunkId, total: data.total })); // 更新进度状态 setProgress(prev => { const completed = prev.completed + 1; const percentage = Math.round((completed / prev.total) * 100); const questionCount = prev.questionCount + (data.total || 0); return { ...prev, completed, percentage, questionCount }; }); totalQuestions += data.total || 0; successCount++; return { success: true, chunkId, total: data.total }; } catch (error) { console.error(t('textSplit.generateQuestionsForChunkError', { chunkId }), error); errorCount++; // 更新进度状态（即使失败也计入已处理） setProgress(prev => { const completed = prev.completed + 1; const percentage = Math.round((completed / prev.total) * 100); return { ...prev, completed, percentage }; }); return { success: false, chunkId, error: error.message }; } }; // 并行处理所有文本块，最多同时处理2个 await processInParallel(chunkIds, processChunk, taskSettings.concurrencyLimit); // 处理完成后设置结果消息 if (errorCount > 0) { setError({ severity: 'warning', message: t('textSplit.partialSuccess', { successCount, total: chunkIds.length, errorCount }) }); } else { setError({ severity: 'success', message: t('textSplit.allSuccess', { successCount, totalQuestions }) }); } } // 刷新文本块列表 fetchChunks(); } catch (error) { console.error(t('textSplit.generateQuestionsError'), error); setError({ severity: 'error', message: error.message }); } finally { setProcessing(false); // 重置进度状态 setTimeout(() => { setProgress({ total: 0, completed: 0, percentage: 0, questionCount: 0 }); }, 1000); // 延迟重置，让用户看到完成的进度 } }; // 处理文本块编辑 const handleEditChunk = async (chunkId, newContent) => { try { setProcessing(true); setError(null); const response = await fetch(`/api/projects/${projectId}/chunks/${encodeURIComponent(chunkId)}`, { method: 'PATCH', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ content: newContent }) }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || t('textSplit.editChunkFailed')); } // 更新成功后刷新文本块列表 fetchChunks(); setError({ severity: 'success', message: t('textSplit.editChunkSuccess') }); } catch (error) { console.error(t('textSplit.editChunkError'), error); setError({ severity: 'error', message: error.message }); } finally { setProcessing(false); } }; // 处理文件删除 const handleFileDeleted = (fileName, filesCount) => { console.log(t('textSplit.fileDeleted', { fileName })); // 从 localStorage 获取当前选择的模型信息 let selectedModelInfo = null; // 尝试从 localStorage 获取完整的模型信息 const modelInfoStr = localStorage.getItem('selectedModelInfo'); if (modelInfoStr) { try { selectedModelInfo = JSON.parse(modelInfoStr); } catch (e) { throw new Error(t('textSplit.modelInfoParseError')); } } else { throw new Error(t('textSplit.selectModelFirst')); } //如果多个文件的情况下，删除的不是最后一个文件，就复用handleSplitText重新构建领域树 if (filesCount > 1) { handleSplitText(['rebuildToc.md'], selectedModelInfo); } else { //删除最后一个文件仅刷新界面即可 location.reload(); } }; // 关闭错误提示 const handleCloseError = () => { setError(null); }; // 处理错误或成功提示 const renderAlert = () => { if (!error) return null; const severity = error.severity || 'error'; const message = typeof error === 'string' ? error : error.message; return ( {message} ); }; // 处理筛选器变更 const handleQuestionFilterChange = value => { setQuestionFilter(value); // 应用筛选 const filteredChunks = chunks.filter(chunk => { if (value === 'generated') { return chunk.questions && chunk.questions.length > 0; } else if (value === 'ungenerated') { return !chunk.questions || chunk.questions.length === 0; } return true; }); setShowChunks(filteredChunks); }; const handleSelected = array => { if (array.length > 0) { let selectedChunks = []; for (let i = 0; i < array.length; i++) { const name = array[i].replace(/\.md$/, ''); console.log(name); const tempChunks = chunks.filter(item => item.id.includes(name)); tempChunks.forEach(item => { selectedChunks.push(item); }); } setShowChunks(selectedChunks); console.log(selectedChunks); } else { const allChunks = chunks; setShowChunks(allChunks); } }; return ( {/* 文件上传组件 */} {/* 标签页 */} {/* 智能分割标签内容 */} {activeTab === 0 && ( )} {/* 领域分析标签内容 */} {activeTab === 1 && } {/* 加载中蒙版 */} theme.zIndex.drawer + 1, position: 'fixed', backdropFilter: 'blur(3px)' }} open={loading} > {t('textSplit.loading')} {t('textSplit.fetchingDocuments')} {/* 处理中蒙版 */} theme.zIndex.drawer + 1, position: 'fixed', backdropFilter: 'blur(3px)' }} open={processing} > {t('textSplit.processing')} {progress.total > 1 ? ( {t('textSplit.progressStatus', { total: progress.total, completed: progress.completed })} {progress.percentage}% {t('textSplit.questionsGenerated', { total: progress.questionCount })} ) : ( {t('textSplit.processingPleaseWait')} )} {/* PDF处理中蒙版 */} theme.zIndex.drawer + 1, position: 'fixed', backdropFilter: 'blur(3px)' }} open={pdfProcessing} > {t('textSplit.pdfProcessing')} {progress.total > 1 ? ( {t('textSplit.pdfProcessStatus', { total: progress.total, completed: progress.completed })} {progress.percentage}% ) : ( {t('textSplit.processingPleaseWait')} )} {/* 错误或成功提示 */} {renderAlert()} ); } ``` ## /commitlint.config.mjs ```mjs path="/commitlint.config.mjs" export default { extends: ['@commitlint/config-conventional'] }; ``` The content has been capped at 50000 tokens, and files over NaN bytes have been omitted. The user could consider applying other filters to refine the result. The better and more specific the context, the better the LLM can follow instructions. If the context seems verbose, the user can refine the filter using uithub. Thank you for using https://uithub.com - Perfect LLM context for any GitHub repo.