Add append-only batch node onboarding and recovery checks
Some checks are pending
Docker Build and Publish / build (push) Waiting to run
Some checks are pending
Docker Build and Publish / build (push) Waiting to run
This commit is contained in:
hellcat
parent
522666c339
commit
6af676421b
@ -1,6 +1,6 @@
|
||||
# dpl 项目维护
|
||||
|
||||
开始维护前阅读 [维护入口](docs/maintenance/README.md)，再按任务范围阅读架构、业务约束、发布恢复与验收状态。用户最新明确要求优先；文档中的历史状态必须按当前代码和运行环境复核。
|
||||
先读 [CoDocs 项目总览](/Users/hellcat/Documents/CoDocs/项目总览.md) 与首次会话的 [持续维护协议](/Users/hellcat/Documents/CoDocs/00-协作规范/维护协议.md)，再阅读 [维护入口](docs/maintenance/README.md)，再按任务范围阅读架构、业务约束、发布恢复与验收状态。用户最新明确要求优先；文档中的历史状态必须按当前代码和运行环境复核。
|
||||
|
||||
- 保留用户及其他任务既有修改，提交只包含本次范围；不要重置或清理无关文件。
|
||||
- 默认 Git 发布：提交、推送，再让服务器更新到明确版本并验证；未经明确要求不得打包覆盖。
|
||||
@ -10,4 +10,10 @@
|
||||
- 本地开发优先 dev/热更新；构建用于发布或构建验证。相关检查与命令见 docs/maintenance/ARCHITECTURE.md。
|
||||
- 后台变更实际验证显示、切换、保存与权限；HTTP 200 不代表界面可用。测试、提交、发布和真实验收分别报告，未执行的步骤不能写成完成。
|
||||
|
||||
## 批量节点接入：默认启动一个子助手
|
||||
|
||||
用户要求执行批量节点接入（明确说批量，或同批接入两个及以上节点）时，主协调者必须默认启动一个子助手，无需再次询问是否使用助手：核查验收助手负责候选服务器、端口、中转与已有资源冲突核查，并独立审查脚本、测试并发/查重/恢复及执行接入验收。主协调者负责统一参数、展示编号、端口、唯一账本及生产写入。完整分工、执行关卡与消耗控制见 [批量接入协作流程](docs/maintenance/NODE_BATCH.md#多智能体协作流程)。
|
||||
|
||||
这项授权针对当前任务的子助手，不自动新建或分叉普通聊天，也不授权向已有聊天发消息。仅讨论方案、维护脚本/文档或单节点任务不自动触发。若子助手工具不可用、并发名额不足或助手失败，说明具体缺口并恢复/等待；未经用户明确允许，不静默省略角色或跳过验收。当前批次已有对应助手时复用，不重复启动。具体线上变更仍以用户本次接入授权为限；项目规则不启动无人值守接入。
|
||||
|
||||
本文件提供维护入口，不扩大当前任务授权，也不启动无人值守自动维护。
|
||||
|
||||
@ -2,11 +2,11 @@
|
||||
|
||||
## 目标与连接
|
||||
|
||||
本项目生产站点为 https://dp-link.net，后台 /dove；后端目录 /www/wwwroot/dp-link.net，服务器逻辑名 site-jp02。通过既定白色跳板连接；具体连接资料参见本机 /Users/hellcat/Documents/CoDocs/项目关系.md 与 vps连接资料.md，不把它们的内容提交到公开仓库。
|
||||
本项目生产站点为 https://dp-link.net，后台 /dove；后端目录 /www/wwwroot/dp-link.net，服务器逻辑名 site-jp02。通过既定白色跳板连接；项目与连接入口参见 [dpl 档案](/Users/hellcat/Documents/CoDocs/10-项目/dpl.md) 和 [私有资源索引](/Users/hellcat/Documents/CoDocs/30-私有资源/README.md)，不把它们的内容提交到公开仓库。
|
||||
|
||||
本地 http://dpl.loc/dove 已配置到当前 Laravel public/。本机 Nginx 配置 /usr/local/etc/nginx/servers/dpl.loc.conf，PHP-FPM 使用现有 9000 端口。系统级服务重载可能需要 Mac 管理员授权；不持久保存系统密码。
|
||||
|
||||
本地开发与联调访问规则（用户于 2026-10-11 指定）：本地不访问 tp-link.net，需要访问远端站点时使用 https://dpl.tomshell.com.de 代替；本地站点仍使用 http://dpl.loc。该访问约定不自动修改生产支付回调域名或生产配置。
|
||||
本地开发与联调访问规则：本地站点仍使用 `http://dpl.loc`；本地访问线上项目域名必须查询 [CoDocs 反代表](/Users/hellcat/Documents/CoDocs/20-基础设施/访问与反代.md) 并走明确登记的白色机反代，dpl 主站入口为 `https://dpl.tomshell.com.de`。缺映射/用途不符时询问用户，不直接或变相访问源站。该访问约定不自动修改生产支付回调域名或生产配置。
|
||||
|
||||
## Git 发布顺序
|
||||
|
||||
|
||||
149
docs/maintenance/NODE_BATCH.md
Normal file
149
docs/maintenance/NODE_BATCH.md
Normal file
@ -0,0 +1,149 @@
|
||||
# 节点批量接入
|
||||
|
||||
本专题说明批量调度；单节点操作、append 的保护范围与验收仍以 [执行手册](NODE_RUNBOOK.md) 为准，展示编号以 [命名规则](NODE_ONBOARDING.md#节点命名与编号) 为准。
|
||||
|
||||
2026-10-16：新增标准库 Python 调度器、命令适配器和离线进程测试。没有部署、连接服务器或增加生产节点。调度器可以运行经过审阅的任务适配器；**没有内置 Cloudflare/中转后台 API，也没有替使用者实现各站点的 SSH 登录与现场检查。**生产任务必须先按下述协议配置命令，不能将离线测试通过视为生产全链路可用。
|
||||
|
||||
## 文件与运行方式
|
||||
|
||||
| 文件 | 用途 |
|
||||
| --- | --- |
|
||||
| `scripts/node-onboarding/batch.py` | 参数校验、资源预留、并发、进程间锁、逐阶段账本与恢复 |
|
||||
| `scripts/node-onboarding/command-adapter.py` | 调用经过摘要固定的任务命令；不经过 shell，缺命令返回等待 |
|
||||
| `tests/node-onboarding/test_batch.py` | 临时目录模拟远端，验证并发、重复运行和回包丢失；不加载 Laravel/.env |
|
||||
|
||||
在仓库根目录运行，计划文件与状态目录用绝对路径，放在受保护任务目录。计划只含非秘密参数和引用；不要加入密码、令牌、私钥内容、含订阅令牌的完整 URL 或命令行凭据。
|
||||
|
||||
```sh
|
||||
python3 scripts/node-onboarding/batch.py plan --plan /private/task/batch.json --state-root /private/task/ledger
|
||||
python3 scripts/node-onboarding/batch.py run --plan /private/task/batch.json --state-root /private/task/ledger --jobs 4 --timeout 900 --execute
|
||||
python3 scripts/node-onboarding/batch.py status --plan /private/task/batch.json --state-root /private/task/ledger
|
||||
```
|
||||
|
||||
`plan` 仅做本地参数校验，不预留资源、不访问网络、不生成状态目录。`status` 只读账本，不触发命令或创建账本。`run --execute` 才可能执行写入。退出码 0 表示本次要求完成；2 表示校验失败、等待、冲突、未知或未完成，不应自动当成临时网络错误无限重试。默认并发 4，上限 16；timeout 是每次适配器调用的超时秒数，不是整个批次的时限。
|
||||
|
||||
## 参数格式
|
||||
|
||||
以下使用保留示例地址，不能直接运行生产任务。`provision_commit` 填已审阅、已推送的完整 Git 提交，bundle 与客户端二进制路径必须由前检核验。VPS ID 与实际地址从 CoDocs 对应资源读取，不能从展示编号推断。所有节点放在同一个 `nodes` 数组，SOCKS 端口各不相同。
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"mode": "append",
|
||||
"batch_id": "jp-batch-20261016",
|
||||
"context": {
|
||||
"panel_vps": "VPS-001",
|
||||
"gateway_vps": "VPS-002",
|
||||
"jump_vps": "VPS-003",
|
||||
"app_root": "/absolute/panel/app",
|
||||
"panel_url": "https://panel.example.invalid",
|
||||
"subscription_url": "https://subscribe.example.invalid",
|
||||
"provision_commit": "0000000000000000000000000000000000000000",
|
||||
"bundle": "/private/task/reviewed.bundle",
|
||||
"client_binary": "/absolute/gateway/sing-box"
|
||||
},
|
||||
"adapter": ["/absolute/python3", "/absolute/repo/scripts/node-onboarding/command-adapter.py"],
|
||||
"command_profile": {
|
||||
"files": {"/private/task/handler.py": "填写该文件的SHA256"},
|
||||
"phases": {
|
||||
"preflight": {
|
||||
"probe": ["/absolute/python3", "/private/task/handler.py", "--plan", "{plan}", "--node", "{node}", "--phase", "{phase}", "--action", "{action}", "--state-dir", "{state_dir}"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"nodes": [{
|
||||
"key": "jp-new-a",
|
||||
"country": "JP", "sequence": 2, "flag": "🇯🇵", "country_name": "日本",
|
||||
"name": "🇯🇵 日本 JP - 02",
|
||||
"vps_id": "VPS-011", "node_address": "192.0.2.11", "listen_port": 24443,
|
||||
"relay_host": "relay.example.invalid", "relay_port": 47011,
|
||||
"tls_domain": "jp-new-a.example.invalid", "formal_group_id": 1,
|
||||
"expected_exit": "192.0.2.11", "socks_port": 19011
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
示例只声明 preflight 命令，其他阶段必然等待。实际任务可为八个阶段逐一登记 `probe`，并在 stage/dns/provision/relay/publish 登记有权限执行的 `apply`。缺少 API 的阶段可以只提供读取实际状态的 probe，由操作者完成外部操作后续跑。不要使用“手工勾选成功”或静态回执替代现场 probe。
|
||||
|
||||
`files` 必须覆盖本地适配器、handler 及其加载的操作配置/脚本；不要只固定 Python 解释器。任一摘要变化停止，不接受在原批次中改命令后直接续跑。远端代码、安装包、二进制也由前检按明确 Git 提交/摘要核验。通用调度器不会自动检查命令是否遵守这些语义，也不会替命令构造 SSH 配置。`adapter` 亦可指向符合协议的专用程序，不必使用命令适配器。
|
||||
|
||||
## 调度与操作协议
|
||||
|
||||
所有候选节点先通过 preflight，才能开始任何 stage。每个节点依次执行以下流程；独立节点同时推进，共享写入由锁串行执行。
|
||||
|
||||
| 阶段 | 并发边界 | 任务命令必须核验的事实 |
|
||||
| --- | --- | --- |
|
||||
| preflight | 按 VPS 并发，全批屏障 | 当前代码/工具/系统依赖；VPS 与地址；空闲端口；现有机器/节点/组/名称/域名/中转冲突；目标正式组；保留资源基线 |
|
||||
| stage | 面板串行 | 参数与 manifest 完整匹配；隔离 QA 和新节点确属本任务；创建后回读对象及私有传输文件 |
|
||||
| dns | 全局串行 | 本次域名实际 DNS 指向；有权限的 API 才能 apply；无 API 则等待 |
|
||||
| provision | 按 VPS 并发 | 已推送提交、安装包摘要、独立服务/配置归属、证书/SNI、旧服务保持；不能仅按文件存在判定完成 |
|
||||
| relay | 全局串行 | 独立规则的入口和目标完全匹配；网关侧严格 TLS/SNI 连通；不得占用旧规则 |
|
||||
| verify | 按 VPS 并发 | 从隔离网关获取真实完整 QA 订阅，用 `verify-node.py --run` 验证预期出口/国家并核对实际流量上报；报告绑定本请求 |
|
||||
| publish | 面板串行 | `promote.php --mode=append` 先预览后应用；只开放本次新节点到指定已有组并禁用 QA；不能重写套餐、用户组或订阅地址 |
|
||||
| audit | 按 VPS 并发 | 正式多节点订阅完整、旧节点/网关/套餐/用户关联/订阅地址符合基线、QA 禁用且订阅拒绝/不下发；新节点仍可用 |
|
||||
|
||||
preflight/verify/audit 仅调用 `probe`，没有 apply。verify 的 probe 会产生隔离测试流量，可以启动/停止本次临时客户端；不能修改正式配置。已经完成 audit 的节点，重跑只做 preflight 与完整 audit，避免重新激活 QA。尚未完成的节点会从第一阶段重新逐项 probe，确认后才越过。
|
||||
|
||||
适配器接受 `--plan FILE --node FILE --phase PHASE --action probe|apply --state-dir DIR`。node 文件包含 `request_sha256`，它绑定 context 与完整节点参数。标准输出必须只含一条 JSON，字段如下：
|
||||
|
||||
```json
|
||||
{"key":"jp-new-a","phase":"stage","request_sha256":"从node文件原样读取","status":"absent","retry_safe":true}
|
||||
```
|
||||
|
||||
| status | 含义与调度行为 |
|
||||
| --- | --- |
|
||||
| ready | 当前现场满足本阶段所有要求，可以继续；不能只依据上次成功日志 |
|
||||
| absent | 已权威确认本阶段尚未执行；只有同时明确 `retry_safe: true` 才能调用 apply |
|
||||
| waiting | 外部依赖未就绪或需人工处理，本节点停下；不占工作线程反复轮询 |
|
||||
| unknown | 现场无法确认，不写入，不自动重试 apply |
|
||||
| conflict | 资源占用、参数漂移或归属冲突；阻止该节点，通知其余任务停止继续写入 |
|
||||
|
||||
apply 返回成功后仍必须 probe 确认。退出码非零、输出损坏、身份不符、超时或 Ctrl-C 后，远端结果一律按未知处理；重跑仍从现场读取。apply 命令必须支持安全处理部分完成的现场；不能把“有进程/有目录但未完成”随意返回 absent。stage 的现有对象没有匹配 manifest、或事务结果与 manifest 不一致时应 conflict/unknown，人工核实，不能自动创建第二套对象。
|
||||
|
||||
命令适配器的占位符仅支持 plan/node/state_dir/key/phase/action；每个元素作为独立 argv 传递，不解释 shell。操作脚本需要参数时自行读取受保护的 JSON。禁止在 argv/标准输出加入凭据；密码与令牌由 handler 按 CoDocs 私有引用按需读取，SSH 保持经白色机跳转；本地 HTTP 继续使用明确登记的反代。不要为了批量化改变既有访问规则。
|
||||
|
||||
## 查重、锁与断点恢复
|
||||
|
||||
计划内拒绝重复的部署 key、展示名、国家编号、VPS ID、实际落地 IP、TLS 域名、中转入口域名+端口与网关 SOCKS 端口。展示名格式与旗帜/国家/两位编号必须匹配。**这些是本地候选查重，已在线资源的查重必须由 preflight/stage 实际查询。**同一物理地址使用不同 VPS 别名也会被拦下。
|
||||
|
||||
状态目录是唯一协调入口：SQLite 事务原子预留身份，同目录 fcntl 锁协调进程。VPS/IP/TLS 与中转入口预留跨面板；名称/编号/key 按面板应用划分；SOCKS 按网关划分。相同节点参数可重跑；不同参数不能借用同一批次或占用已有预留。预留不自动释放，不提供强制接管命令。
|
||||
|
||||
所有参与聊天/进程必须使用**同一台协调机、同一个状态目录**。另建账本、复制状态目录、在另一台主机启动独立协调器都不能获得跨机锁保证。对共享面板的非批量手工操作也必须协调暂停，不能指望本地锁约束后台人工修改。每次写入前的远端归属/参数检查仍必要。
|
||||
|
||||
中断后保留原计划、账本、远端 manifest 和操作日志。先用 status 查看，再以同一计划、同一状态目录运行相同 run 命令；不要删除账本、改 batch_id 或清除 manifest 来“解除报错”。冲突需先核对原始请求与现场，不能通过修改参数静默接管。未知结果宁可停住；恢复不保证所有断点都能无人干预推进。
|
||||
|
||||
SIGINT/SIGTERM 会停止派发后续写入，终止当前本地适配器进程组；**这不保证已发出的远端命令停止，也不自动回滚**。其他已开始的节点可能已完成步骤。已完成节点保留成果；未完成节点留在其最后安全阶段。需要回退时按执行手册逐字段预览与回退，不做全批删除。
|
||||
|
||||
## 多智能体协作流程
|
||||
|
||||
用户于 2026-10-16 明确授权：以后执行批量节点接入，默认启动一个子助手，按本节分工执行，不再重复询问是否需要助手。触发入口见项目根 [AGENTS.md](../../AGENTS.md)。这里的“启动”由主协调者使用子助手工具完成；`batch.py` 本身不调用模型或创建助手。明确要求批量，或同批接入两个及以上节点时触发；仅讨论、修改脚本/文档、单节点接入不自动触发。用户当次明确调整角色或数量时，以最新要求为准。
|
||||
|
||||
| 角色 | 固定职责 | 写入与协作边界 |
|
||||
| --- | --- | --- |
|
||||
| 主协调者 | 明确本批范围，统一参数、展示编号与端口，固定计划，持有唯一账本，安排生产写入并汇总结果 | 只有主协调者启动本批正式调度和共享资源写入；不以助手已派发代替工作完成 |
|
||||
| 核查验收助手 | 接入前核查候选服务器身份、既有节点/机器、端口、中转、DNS 和归属冲突；独立审查脚本与参数、测试并发/查重/恢复；开放前后核对 QA、正式订阅和旧资源基线 | 分阶段完成两类职责，返回每节点结论、证据及缺口；不自行分配最终编号、创建节点、改后台或另起调度。真实联网验收的对象、端口和时机由主协调者统一安排 |
|
||||
|
||||
执行顺序：
|
||||
|
||||
1. 主协调者在生产写入前启动/复用一个核查验收助手，给出当前任务范围、必要资料入口、受保护资源、分阶段输出和完成标准。当前批次已存在对应子助手时继续使用，不在每个节点或每次续跑时重新创建。
|
||||
2. 助手分阶段核查资源与审查方案；主协调者并行准备参数与调度。助手完成冲突核查和相关隔离测试/审查后，主协调者解决影响写入的缺口，固定同一参数单，再运行全批 preflight。检查发现变更时重新核对受影响项，不把旧回执沿用到不同参数。
|
||||
3. 独立节点由脚本按 `--jobs` 并发，面板/DNS/中转按共享锁执行。正式开放前，核查验收助手核对本节点隔离 QA、严格 TLS、实际出口与上报证据；主协调者确认后执行 publish。助手不得另起同批调度、重复 stage、共用浏览器会话切换账户或绕开账本。
|
||||
4. 正式开放后，核查验收助手检查正式订阅、QA 禁用/拒绝、旧资源基线并报告通过或具体缺口。主协调者核实后逐节点报告完成、等待或需用户操作；批次全部完成必须有对应实际验收，不能以子助手结束作为完成证明。
|
||||
5. 子助手不可用、并发名额不足或失败时，主协调者明确报告，复用/恢复角色或等待名额；不自动追加更多助手、跳过必需角色或把未知写成通过。暂停和停止指令必须传达给子助手，并停止派发后续写入；已发出的远端操作仍按断点恢复规则回读。
|
||||
|
||||
主协调者与一个子助手分别消耗处理其输入和输出所需的 token，总量不是固定倍数。主协调者默认仅传必要背景、资料路径、节点参数与任务边界，避免复制整段历史；每项检查明确归属，汇报用结论、证据位置与缺口，不反复粘贴完整日志。独立验收保留必要的交叉核对，不为节省 token 省略验收。没有固定 token 配额或价格承诺。
|
||||
|
||||
该常驻授权仅允许启动当前任务的子助手；不自动新建/分叉侧边栏普通聊天，向已有聊天派消息仍需用户明确授权。节点并发数由脚本控制，助手数保持一个；不会为十个节点启动十个助手。项目规则不扩大实际生产变更、凭据、访问路线或后台自动化授权。
|
||||
|
||||
## 本地验证
|
||||
|
||||
本地测试：
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s tests/node-onboarding -p 'test_batch.py' -v
|
||||
```
|
||||
|
||||
本次 11 项测试通过，覆盖：plan/status 无副作用，计划内重复与命名约束，全批前检屏障，独立节点重叠执行/面板串行，完整重跑不重复写，模拟写成后回包丢失，SIGTERM 中断后续跑，跨批预留/参数漂移，两进程竞争、错误身份回执、命令摘要变更及配置完整的适配器流程。测试没有验证生产命令的正确性、DNS/中转 API、真实 SSH/网络、现场负载或十节点耗时。生产批次的候选参数、命令 profile 与现场验收须在具体获授权的接入任务中另行核实。
|
||||
|
||||
|
||||
2026-10-17追加面板隔离测试：`php vendor/bin/phpunit --no-configuration tests/node-onboarding/PanelChangeIsolatedTest.php`，9 tests / 65 assertions通过。真实Eloquent与SQLite内存库，无.env/Redis/队列/网络；覆盖append保护范围、QA禁用、幂等、事务失败/通知失败恢复和冲突拒绝。NodeSync为测试替身，生产通知与用户下发仍需现场验收。
|
||||
@ -1,5 +1,7 @@
|
||||
# 节点与订阅网关接入
|
||||
|
||||
后续接入的参数与命令唯一入口：[节点接入执行手册](NODE_RUNBOOK.md)。首次初始化与追加节点分别执行，不自动改订阅地址、套餐或用户关联。
|
||||
|
||||
首节点历史核对日期：2026-10-11。本文是后续新增 dpl 节点的执行入口。首节点部署、服务器侧完整订阅联网与正式套餐开放已完成；用户明确要求跳过本地连通测试。Cloudflare 本聊天已正常访问，DNS/严格 TLS、有效订阅响应、HTTP/2 与 gzip 解码已复核；两个专用订阅区域的全域缓存绕过、两机场订阅路径 BIC 例外及原机场严格回源已部署并复验。已实施范围与可选未实施调优见下文。凭据、机器令牌、用户 UUID 与订阅令牌只存受保护文件，不写入本文或 Git。
|
||||
|
||||
## 2026-10-16 日本节点接入
|
||||
@ -23,11 +25,19 @@
|
||||
|
||||
订阅：客户端 → c2.deeplink-subscribe.com（Cloudflare 橙云 HTTPS）→ gateway-01 宿主 Nginx → dpl 独立 SubSieve → dpl 面板。
|
||||
|
||||
流量：客户端 → haiwi.lshan.eu.org:47101 → node-us01 64.44.115.18:24443 → 目标网络。AnyTLS 使用 us01.deeplink-subscribe.com 的有效证书，订阅要求严格校验。普通 Cloudflare 橙云只用于订阅 HTTP，不用于原生 AnyTLS 流量。
|
||||
流量链路按项目现有接入记录分列（2026-10-11/16历史现场结果，本次未联网复验）：
|
||||
|
||||
| 部署 key | VPS（不代表展示编号） | 中转入口 | 落地 | TLS 名称 | 证据 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| us01 | node-us01 | haiwi.lshan.eu.org:47101 | 64.44.115.18:24443 | us01.deeplink-subscribe.com | 2026-10-11 首节点验收 |
|
||||
| jp02 | node-jp02 | haiwi.lshan.eu.org:47202 | 193.111.31.52:24443 | jp02.deeplink-subscribe.com | 本文2026-10-16日本节点接入 |
|
||||
|
||||
客户端 → 对应中转入口 → 对应落地 → 目标网络。两条链路均使用严格 AnyTLS TLS；普通 Cloudflare 橙云用于订阅 HTTP，不用于原生 AnyTLS 流量。完整对象关联见[CoDocs DPL档案](/Users/hellcat/Documents/CoDocs/10-项目/dpl.md)。
|
||||
|
||||
| 资源 | 本次新增 | 必须保留的原有资源 |
|
||||
| --- | --- | --- |
|
||||
| node-us01 | dpl-node，TCP 24443，机器/节点 ID 1 | XrayR，TCP 443 |
|
||||
| node-us01 | dpl-node，TCP 24443，机器/节点 ID 1（2026-10-11） | XrayR，TCP 443 |
|
||||
| node-jp02 | dpl-node，TCP 24443，机器/节点 ID 2（2026-10-16） | XrayR，TCP 443；原中转 46202 → 443 |
|
||||
| 中转 | dpl-us01-anytls，47101 → 24443 | us01，46101 → 443；其他规则 |
|
||||
| gateway-01 | dpl-subscribe-gateway、dpl-subscribe-admin | subscribe-gateway、subscribe-admin |
|
||||
| 网关回环端口 | 18080 / 18443，管理 16444 | 8080 / 8443，管理 64444 |
|
||||
@ -90,70 +100,20 @@ VPS 代号、数据库节点/机器 ID、部署 key 与展示编号各自独立
|
||||
|
||||
## 新节点参数单
|
||||
|
||||
| 参数 | 首次实例 | 后续填写规则 |
|
||||
| --- | --- | --- |
|
||||
| key | us01 | 唯一、稳定的部署标识；不用于推导展示序号 |
|
||||
| name | 🇺🇸 美国 US - 01 | 按[节点命名与编号](#节点命名与编号)从生产清单分配，按国家从01递增，不沿用VPS序号 |
|
||||
| server | node-us01 | 从 CoDocs 项目关系及连接资料确认 |
|
||||
| listen-port | 24443 | 当场核对 TCP/UDP 和防火墙，不复用过期判断 |
|
||||
| relay-host / relay-port | haiwi.lshan.eu.org / 47101 | 新增规则，不改原规则 |
|
||||
| tls-domain | us01.deeplink-subscribe.com | DNS only，有效证书 |
|
||||
| group / plans | 首次隔离组；套餐 1、6、7、8 | 当前真实 ID 和授权集合 |
|
||||
| backup | /root/codex-backups/dpl/node-onboarding-20261011-us01 | 每节点独立时间/key 目录 |
|
||||
参数、冲突检查与模式选择统一见[执行手册](NODE_RUNBOOK.md#统一参数单与冲突检查)，不沿用首节点示例作为后续默认值。命名仍以本文前一节为准。
|
||||
|
||||
## 重复执行步骤
|
||||
|
||||
1. 读取根 AGENTS、维护入口和本文。经 CoDocs 白色机中转访问服务器；本地网页使用指定访问域名。记录原服务 PID/启动时间/端口、中转规则、Docker ID/启动时间、原订阅结果；审计资源、防火墙、所有证书钩子。受保护备份本次将修改的配置和 Git 差异，不输出私密内容。
|
||||
2. 非敏感脚本和模板先 Git 提交/推送，再更新服务器到明确提交，保留生产原有差异。无法直接拉取时可传输已推送提交的 Git bundle 后 fetch/fast-forward，不能用压缩包覆盖。scripts/node-onboarding 可 subtree split 为独立运维仓库；面板 .env 不送到节点。
|
||||
3. 在生产应用根目录按参数单运行 stage.php，创建机器、AnyTLS 节点、隔离组和 100MB/一天的临时用户。节点虽启用显示，正式用户没有隔离组权限。相同 manifest 重跑只报告已创建；同名资源存在却缺少 manifest 时停止，不自动接管。
|
||||
按[统一接入顺序](NODE_RUNBOOK.md#统一接入顺序)执行；先区分首次 initialize 与已有组 append。新本地promote.php必须显式mode；append只修改新节点及本次QA，默认预览、apply才写入。该版本尚未测试或发布，线上旧脚本不能直接按新语义调用。
|
||||
|
||||
```sh
|
||||
php scripts/node-onboarding/stage.php \
|
||||
--key=us01 --name='🇺🇸 美国 US - 01' \
|
||||
--relay-host=haiwi.lshan.eu.org --relay-port=47101 \
|
||||
--listen-port=24443 --tls-domain=us01.deeplink-subscribe.com \
|
||||
--backup=/root/codex-backups/dpl/node-onboarding-20261011-us01
|
||||
```
|
||||
|
||||
4. 将受保护备份的 node-config.yml、node-secret.env 经白机安装到节点 /etc/dpl-node/config.yml、secret.env，均 600；运维仓库 checkout 明确 Git 版本到 /opt/dpl-node/provision。首次安装使用下面命令。脚本检查参数、配置、端口、已有后端及 XrayR 状态，校验官方二进制并安装独立 unit，不自动启动。已有 dpl-node 时不能重跑覆盖；升级须单独复核。
|
||||
|
||||
```sh
|
||||
bash /opt/dpl-node/provision/install-node.sh \
|
||||
--listen-port=24443 --tls-domain=us01.deeplink-subscribe.com
|
||||
```
|
||||
|
||||
5. 配置 DNS only、有效证书并审计/安装仅匹配新域名的续期钩子，再启用 dpl-node。在中转后台只新增 dpl-<key>-anytls，核对保存后的入口/目标和诊断结果。本次未启用收发 Proxy Protocol，以后启用前须同时确认入口和后端支持。
|
||||
6. 后续 dpl 节点复用 c2 及当前网关，不再次更换 NS。另一个机场需要不同上游时才新增独立实例；不得改已有上游、运行原项目 setup/update/down 或复用旧卷。
|
||||
7. 在网关创建本次验收目录（700），以 600 保存 stage.php 生成的 QA 私密文件和实际订阅；准备官方客户端并校验官方发布摘要。首轮已校验的 Linux 1.14.3 客户端保留于 /opt/dpl-gateway/acceptance/，下述命令可复用它。拉取临时用户完整订阅到私密文件；sing-box 使用正常版本 UA（如 sing-box/1.14.3），不要使用覆盖版本识别的裸 flag=sing-box。核验名称、入口、SNI、严格 TLS、缓存头和真实 IP；先官方客户端 check，再完整配置实际联网，核对出口、DNS、上下行计费与上报。
|
||||
|
||||
```sh
|
||||
python3 /opt/dpl-gateway/provision/verify-node.py \
|
||||
--qa /var/lib/dpl-sgw/qa/qa.json \
|
||||
--binary /opt/dpl-gateway/acceptance/sing-box-1.14.3-linux-amd64/sing-box \
|
||||
--subscription /var/lib/dpl-sgw/qa/subscription.json \
|
||||
--expected-exit 64.44.115.18
|
||||
```
|
||||
|
||||
helper 仅替换 TUN 等入站为临时本机 SOCKS，保留订阅 DNS、分流及出站，退出时停止客户端；失败日志留在同一私密目录。这不等于手机 GUI、系统 TUN、多地区、超额或多设备并发验收。某环境到入口不通时分段诊断并换环境核验，HTTP 200 不能替代联网。
|
||||
|
||||
8. 上述验收通过后，在生产根目录运行 readiness.php 和 promote.php，显式指定当前套餐集合。已有正式组时额外传 --group=<ID>。只关联所选尚无组的套餐及这些套餐现有无组用户，不改价格、周期、流量或设备限制。
|
||||
|
||||
```sh
|
||||
php scripts/node-onboarding/readiness.php \
|
||||
--backup=/root/codex-backups/dpl/node-onboarding-20261011-us01
|
||||
php scripts/node-onboarding/promote.php \
|
||||
--backup=/root/codex-backups/dpl/node-onboarding-20261011-us01 \
|
||||
--plans=1,6,7,8
|
||||
```
|
||||
|
||||
首次隔离组改为基础节点、订阅域名设置为 c2，以后复用正式组。脚本要求节点在线且临时用户已有流量，保存 before-promotion.json，最后禁用临时用户并写 promoted.json。清理本次客户端和私密传输副本，保留服务器备份、禁用用户与审计记录。
|
||||
|
||||
9. 回归原机场服务/规则/订阅，实际核对正式组权限、面板节点及订阅域名保存，确认测试用户禁用且不再下发。更新本文及 STATUS，分别记录提交、发布、真实验收及未完成项。
|
||||
依赖检查在创建面板资源前完成；证书演练、完整订阅验收、精确回退与现有脚本限制均在执行手册维护，不在多处复制命令。本地新增面板正式开放/精确回退与多节点验证入口，均尚未测试；细节与奶油接手测试清单见执行手册。远程服务/DNS/中转仍按专项恢复。
|
||||
|
||||
数据库默认 sing-box 模板已迁移新 DNS/路由格式，保留原分流和选择逻辑；现代本地 DoH 使用默认 dialer，不绕到空 direct 出站。仓库原默认资源和其他机场模板不改。apply-template.php 仅接受原默认模板，或与显式 expected-current-sha256 完全匹配的旧版本，逐次备份后只更新该模板/缓存。用户定制或未知模板须先审查，不能强行覆盖。
|
||||
|
||||
## 回退与排障
|
||||
|
||||
节点变更先按[精确回退入口与证据契约](NODE_RUNBOOK.md#精确回退入口与证据契约)确定本次对象及 before/after。下列专项恢复说明继续适用；历史 before-promotion 不保证包含所有 after 值，不能盲目恢复。
|
||||
|
||||
- 单节点：停用本次节点/新增中转，停止新 dpl-node；保留 XrayR、原规则和审计数据。
|
||||
- 正式关联：依据 before-promotion.json 精确恢复本次套餐/用户组及订阅域名，先确认当前值仍是本次写入值，避免覆盖后续业务。不恢复整库。
|
||||
- 模板：从对应私密模板备份恢复该条内容并清理模板缓存，不替换整个设置表。
|
||||
|
||||
254
docs/maintenance/NODE_RUNBOOK.md
Normal file
254
docs/maintenance/NODE_RUNBOOK.md
Normal file
@ -0,0 +1,254 @@
|
||||
# DPL 节点接入执行手册
|
||||
|
||||
入口：[节点关系与历史验收](NODE_ONBOARDING.md)。本手册定义后续授权任务的步骤。2026-10-16 已补齐本地脚本草稿；用户要求测试交给奶油，本轮未运行脚本、语法检查或功能测试，未触及线上。**新脚本必须先在隔离环境测试，再按Git流程审阅发布；本地存在不代表线上已更新。**已有节点展示命名按原入口的[节点命名与编号](NODE_ONBOARDING.md#节点命名与编号)，此处不另建规则。
|
||||
|
||||
## 先选择变更模式
|
||||
|
||||
| 模式 | 适用情形 | 允许修改的范围 |
|
||||
| --- | --- | --- |
|
||||
| initialize | 首次建立 DPL 网关、基础节点组及明确授权的套餐可见范围 | 本次参数单与授权清单逐项列明；订阅地址、套餐/用户关联必须分别确认 |
|
||||
| append | 已有网关/正式节点组上增加节点 | 新机器、新节点、新独立中转、节点 DNS/证书和 QA 隔离资源；验收后只把新节点加入指定已有组并禁用 QA |
|
||||
|
||||
默认判断依据是现有资源与目标，不把“新节点”当作首次初始化。append 保持现有订阅地址、网关、套餐全部字段、正式用户分组及旧节点不变。扩展套餐覆盖另列变更任务。未知已有组/节点状态时先解决该依赖，不能通过重跑初始化猜测。
|
||||
|
||||
## 统一参数单与冲突检查
|
||||
|
||||
在本次服务器受保护任务目录保存参数，目录 700、文件 600；非秘密参数也不进公开日志。以下是待填写模板，没有可直接用于新节点的默认值。文件只使用赋值，不含执行命令；人工审阅后才加载。
|
||||
|
||||
```sh
|
||||
MODE='' # initialize 或 append
|
||||
KEY='' # 稳定部署 key
|
||||
NODE_NAME='' # 按既有命名规则确定；与 VPS 代号独立
|
||||
VPS_ID='' # CoDocs 稳定 ID
|
||||
NODE_ADDRESS='' # 实际落地地址，按私有资源读取
|
||||
LISTEN_PORT=''
|
||||
RELAY_HOST=''
|
||||
RELAY_PORT=''
|
||||
TLS_DOMAIN=''
|
||||
FORMAL_GROUP_ID='' # append 必须填已有正式组
|
||||
PLAN_IDS='' # 仅 initialize 或另有明确套餐变更授权时填
|
||||
BACKUP='' # 绝对路径，每次任务独立，不复用其他节点目录
|
||||
QA_DIR='' # 网关本次独立验收目录
|
||||
CLIENT_BINARY='' # 已校验摘要的官方客户端绝对路径
|
||||
EXPECTED_EXIT=''
|
||||
EXPECTED_COUNTRY=''
|
||||
LOCAL_SOCKS_PORT='' # 网关上本次隔离客户端空闲端口
|
||||
APP_ROOT='' # 面板应用路径，从项目档案核实
|
||||
PROVISION_ROOT='' # 面板上本次已确认脚本路径
|
||||
NODE_PROVISION='' # 节点上已确认的运维 Git 路径
|
||||
GIT_COMMIT='' # 已推送且经过审阅的明确提交
|
||||
PANEL_URL='' # 节点回连面板的明确HTTPS地址
|
||||
SUBSCRIBE_URL='' # 仅initialize明确修改的订阅地址
|
||||
QA_REPORT='' # 本次隔离QA的私有实时验收报告
|
||||
FORMAL_EXPECTATIONS='' # 正式多节点预期的私有JSON文件
|
||||
FORMAL_REPORT='' # 正式多节点验证报告
|
||||
```
|
||||
|
||||
填写后逐项核对：key/展示名、机器名、面板 ID、VPS、落地监听端口、中转入口的域名+端口、源站目标、SNI、正式组、DNS 现值、备份路径和 QA 端口。相同 key 有 manifest 时，只回读已有对象并确认参数完全一致；新 stage.php 重跑要求 manifest 记录的完整参数一致，并核对节点、机器和中转端点；旧 manifest 没有 parameters 时停止，不能自动采纳。相同资源无 manifest、冲突端口或未授权的覆盖均停止，不自动接管。
|
||||
|
||||
机密字段单独沿用 stage.php 生成的私密文件；不把机器令牌、QA UUID、订阅令牌写进上述参数单、文档或命令行。参数按角色分发，不把面板 .env 复制到节点。
|
||||
|
||||
可在审阅参数文件后运行以下本地字段检查；它不验证线上冲突或资源归属：
|
||||
|
||||
```sh
|
||||
set -eu
|
||||
: "${MODE:?}" "${KEY:?}" "${NODE_NAME:?}" "${VPS_ID:?}" "${NODE_ADDRESS:?}"
|
||||
: "${LISTEN_PORT:?}" "${RELAY_HOST:?}" "${RELAY_PORT:?}" "${TLS_DOMAIN:?}"
|
||||
: "${BACKUP:?}" "${QA_DIR:?}" "${CLIENT_BINARY:?}" "${EXPECTED_EXIT:?}" "${EXPECTED_COUNTRY:?}"
|
||||
: "${LOCAL_SOCKS_PORT:?}" "${APP_ROOT:?}" "${PROVISION_ROOT:?}" "${NODE_PROVISION:?}" "${GIT_COMMIT:?}"
|
||||
: "${PANEL_URL:?}" "${QA_REPORT:?}"
|
||||
case "$MODE" in
|
||||
append) : "${FORMAL_GROUP_ID:?}"; test -z "$PLAN_IDS" ;;
|
||||
initialize) : "${PLAN_IDS:?}" "${SUBSCRIBE_URL:?}" ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
```
|
||||
|
||||
## 创建面板记录之前完成前置检查
|
||||
|
||||
只在具体接入任务授权后，通过 CoDocs 白色机跳板及登记的后台反代执行以下检查。本次文档维护没有执行这些动作。
|
||||
|
||||
| 位置 | 必须确认 | 未通过时 |
|
||||
| --- | --- | --- |
|
||||
| 面板 | Git 提交、当前组和节点清单、现有订阅设置、套餐/用户分组快照、应用运行环境、脚本实际版本 | 先补证据与备份，不创建机器/节点/QA 记录 |
|
||||
| 节点 | OS/架构、内存/磁盘、时间同步、systemd、Git、curl、sha256sum、ss、证书工具，以及命令实际路径 | 先明确缺失工具和安装来源；安装属于接入任务变更，文档整理不安装 |
|
||||
| 软件来源 | 发行版源可下载、包版本与依赖可满足；运维仓库和官方二进制来源/摘要可验证 | 不把 apt 索引存在当可安装；不自动换不明镜像或全量升级 |
|
||||
| 证书 | 已有 renewal 配置、pre/post/deploy hooks、Certbot 是 apt 还是 Snap、可执行路径、timer/cron 的真实调用参数和运行用户 | 确定新证书隔离方案及可生效调度后再签发；不能仅添加 renewal 文件就视为自动续期完成 |
|
||||
| 节点隔离 | 现有 XrayR 等服务的 PID/启动时间、TCP/UDP 监听、防火墙与新端口、旧证书和目录权限 | 冲突先处理；安装脚本只检查 TCP，不能代替全项前检 |
|
||||
| 中转 | 后台/登录账户按 CoDocs 读取，当前规则快照、目标落地、隧道及入口端口、Proxy Protocol 两端设置 | 缺入口先问，不改原规则；创建后回读实际值 |
|
||||
| 网关 | 容器 ID/启动时间/重启计数、CPU/内存、独立 QA 路径和端口、Python3、官方客户端及其摘要 | 补齐验收依赖后再 stage，不边创建业务记录边找工具 |
|
||||
|
||||
历史 jp02 接入曾因 Debian11 源包 404 临时使用 Debian 官方 HTTPS 归档安装 Git；这是该次处理记录，不是以后默认改源或安装命令。先核对目标当下发行版与来源，不在本手册固化过期包 URL。
|
||||
|
||||
前置完成后，保存本次允许改变的对象、每个对象的 before 值、预期 after 值和精确回退依据。不要打印整库快照或秘密。
|
||||
|
||||
## 本地脚本能力与边界（尚未测试/发布）
|
||||
|
||||
| 工具 | 新行为 | 边界 |
|
||||
| --- | --- | --- |
|
||||
| stage.php | 明确panel-url、完整参数/对象重跑核对、端点冲突检查；受保护manifest可恢复传输文件 | 旧manifest不自动采纳；依赖与实际资源前检仍由接入任务完成 |
|
||||
| promote.php | 强制mode；默认只预览；append只写新节点group_ids及QA banned；initialize显式plans/subscribe-url/可选group-name | 真正写入需apply及15分钟内匹配该QA、端点与SNI的实时验证报告；不根据节点名推导身份 |
|
||||
| readiness.php | 只统计显式plans，缺省不统计套餐；可检查给定group的冲突套餐 | 只读快照，不代替真实订阅/计费/界面验收 |
|
||||
| verify-node.py | 默认离线结构核对；显式run才启动客户端/联网；可按预期逐一验证多节点、SNI、密码、出口IP及国家 | 正式多节点需要完整预期清单；只证明实际执行的探针，不代替业务/手机/TUN/全路由验收 |
|
||||
| rollback.php | 默认预览；基于新promotion-journal逐字段比较after再恢复before；保持QA禁用，隔离本次节点 | 冲突整次停止，不覆盖后续变化；只覆盖面板变更，不停止远程服务或修改DNS/中转；旧before-promotion需专项人工核对 |
|
||||
| operations.php | 私有文件锁、原子写日志、事务内记录写前意图、提交后完成标记；中断后按before/after识别已应用或冲突 | 文件与数据库不是同一个事务；状态未知需依据日志恢复，不盲目重新stage |
|
||||
| install-node.sh / renew-node.sh | 沿用已登记的独立安装与专用续期hook | 本次未更改，不代表前置工具/安装源/实际timer已经验证 |
|
||||
|
||||
新脚本不修改节点展示名、不分配编号。操作日志不包含密码/UUID/机器令牌；脚本不在错误输出中打印数据库异常全文或客户端配置。首次运行前必须由奶油进行隔离测试，特别核对事务、缓存/节点通知、重复执行和崩溃恢复。
|
||||
|
||||
## 统一接入顺序
|
||||
|
||||
1. 完成参数单、模式确认、前检和恢复点；非秘密运维代码先提交/推送，再更新目标至明确 Git 提交。已有线上差异保留。拉取受限时只能传输已推送提交的 Git bundle 后 fetch/fast-forward，不用归档覆盖。
|
||||
2. 在确认的面板应用根目录，用下列统一参数创建隔离资源；创建后把实际 node_id、machine_id、group_id、test_user_id 与 key/VPS 对应写回本次任务 manifest，不能从服务器名推导面板 ID。
|
||||
|
||||
```sh
|
||||
cd "$APP_ROOT"
|
||||
php "$PROVISION_ROOT/stage.php" \
|
||||
--key="$KEY" --name="$NODE_NAME" \
|
||||
--relay-host="$RELAY_HOST" --relay-port="$RELAY_PORT" \
|
||||
--listen-port="$LISTEN_PORT" --tls-domain="$TLS_DOMAIN" \
|
||||
--panel-url="$PANEL_URL" --backup="$BACKUP"
|
||||
```
|
||||
|
||||
3. 按既有受保护传输方式安装 node-config.yml 和 node-secret.env 至节点的独立目录，文件 600、父目录 700；实际运维 Git 路径使用 NODE_PROVISION，保留二进制固定版本/摘要核验。节点未安装本服务时才运行：
|
||||
|
||||
```sh
|
||||
bash "$NODE_PROVISION/install-node.sh" \
|
||||
--listen-port="$LISTEN_PORT" --tls-domain="$TLS_DOMAIN"
|
||||
```
|
||||
|
||||
4. 配置本次 DNS only、证书和隔离续期策略，再启动新服务；中转只新增本次规则，保存后回读入口、目标及诊断结果。Proxy Protocol 默认保持本次参数单记录，不能单端打开。已有网关按 append 复用，不重建网关、NS、旧卷或订阅配置。
|
||||
5. 按下节执行隔离验收；不因 service active、内核磁盘无 JSON 或 HTTP 200 单独判断成功/故障。Xboard-Node 可使用内存配置，须按实际版本、启动日志、证书和请求链路判断。
|
||||
6. 验收通过后按下方“面板变更命令”预览，再对同一组参数明确apply。append只改新节点与QA；initialize才允许修改明确列出的套餐/用户分组及订阅地址。保存前后日志并回读。未通过隔离测试的新版本不能直接带到生产执行。
|
||||
7. 回读正式订阅与原资源，检查 QA 禁用、订阅拒绝、下发集合无 QA，确认旧节点、旧中转、套餐/用户和订阅设置保持预期。同步 CoDocs 对象关系、节点登记及 STATUS；提交/推送/部署/现场验收分别报告。
|
||||
|
||||
## 证书钩子与演练
|
||||
|
||||
新证书只挂本项目专用 hook。历史已验证的调用方式为 `/bin/sh /opt/dpl-node/provision/renew-node.sh`；不能假定脚本有执行权限后直接执行。路径实际不同时按当前运维检出路径填写。renew-node.sh 依据 `/etc/dpl-node/certificate-domain` 和 RENEWED_LINEAGE 精确匹配，只操作 dpl-node。
|
||||
|
||||
目标实际 Certbot 命令路径记为 CERTBOT_BIN；先审计目录钩子、renewal 配置及 Snap/apt timer 的调用方式。经确认的演练入口：
|
||||
|
||||
```sh
|
||||
: "${CERTBOT_BIN:?}"
|
||||
"$CERTBOT_BIN" renew --cert-name "$TLS_DOMAIN" --dry-run \
|
||||
--no-random-sleep-on-renew --run-deploy-hooks --no-directory-hooks
|
||||
```
|
||||
|
||||
`--no-directory-hooks` 不替代审计 renewal 配置内的显式钩子。必须确认该证书配置仍明确调用专用 `/bin/sh …/renew-node.sh`，并检查演练后的服务状态及旧服务 PID/启动时间。关闭随机等待只用于本次演练可控等待，不等于修改常规定时调度。BrokenPipe/超时只记未确认，回查进程和最终结果后再决定是否重试，不重复发起签发。
|
||||
|
||||
续期完成条件包括：证书配置正确、调度 enabled/下一次触发可见、调度参数确实覆盖新证书及专用 hook、dry-run 退出状态和旧服务保护证据齐全。证书到期时间引用证书/域名对应记录，不把一次演练日期当作长期验收。
|
||||
|
||||
## 统一验收入口
|
||||
|
||||
每次在 QA_DIR 保存 700/600 的 QA 私密文件与完整实际订阅；文件名统一为 qa.json、subscription.json、验收结果和受保护日志，参数与 stage manifest 对照，不使用首节点默认路径。本地浏览器、服务器请求均按各自授权路线执行。
|
||||
|
||||
在已授权的网关环境，先用本次参数里的官方客户端检查完整配置，再运行完整隔离订阅：
|
||||
|
||||
```sh
|
||||
"$CLIENT_BINARY" check -c "$QA_DIR/subscription.json"
|
||||
python3 /opt/dpl-gateway/provision/verify-node.py \
|
||||
--qa "$QA_DIR/qa.json" --subscription "$QA_DIR/subscription.json" \
|
||||
--binary "$CLIENT_BINARY" --port "$LOCAL_SOCKS_PORT" \
|
||||
--expected-exit "$EXPECTED_EXIT" --expected-country "$EXPECTED_COUNTRY" \
|
||||
--report "$QA_REPORT" --run
|
||||
```
|
||||
|
||||
上面脚本路径是既有网关检出约定，执行前核对实际 Git 版本与路径，若不同则使用本次确认路径，不复制临时代码替代。QA可按host/port唯一匹配本次节点，冲突时显式target-tag；正式多节点使用下方expectations。验证保留DNS/分流及全部出站，替换临时SOCKS入站，并仅为api.ipify.org与www.cloudflare.com两个诊断目的地添加指定出站规则，从而逐节点验证；不是原订阅所有路由均已验收。
|
||||
|
||||
| 验收层 | 必须保留的结果 |
|
||||
| --- | --- |
|
||||
| 参数一致 | 订阅包含本次入口/端口/SNI、严格 TLS 开启；机器/节点与 VPS 对应正确 |
|
||||
| 协议/联网 | 官方客户端配置检查与完整启动结果、严格 TLS、目标 HTTPS、出口 IP 与 EXPECTED_EXIT 一致；脚本强制比对 EXPECTED_COUNTRY；仅结构核对不产生live_verified报告 |
|
||||
| 订阅 | 有效用户标准/短路径按本次范围检查，JSON/gzip、private/no-store、无 HIT/Age/挑战；不要在日志保存令牌 |
|
||||
| 上报 | readiness.php回读节点在线、last_check/last_push及QA上下行；按显式plans/group统计，缺省不选任何套餐 |
|
||||
| 正式开放 | 正式有效用户订阅与完整expectations逐一对应，并逐节点验证；不把单节点QA结果扩大成全部可用 |
|
||||
| QA 清理 | QA banned、其订阅拒绝、实际下发用户集合无 QA；客户端退出、临时本地秘密副本清理，服务器审计文件保留 |
|
||||
| 回归 | 原节点/中转、套餐字段/用户组、订阅 URL、网关容器 ID/启动时间/重启计数与 before 快照对照 |
|
||||
| 浏览器 | 正确反代入口，管理列表与保存结果回读；只查看后取消不算提交验收 |
|
||||
|
||||
面板回读命令入口如下；它只覆盖该脚本实际返回的节点/QA 字段，其他行须另留证据：
|
||||
|
||||
```sh
|
||||
cd "$APP_ROOT"
|
||||
php "$PROVISION_ROOT/readiness.php" --backup="$BACKUP" --group="$FORMAL_GROUP_ID"
|
||||
```
|
||||
|
||||
GUI/手机、TUN、多地区、并发及用户明确跳过的测试分别标为未执行或跳过。未确认结果不补成通过；统一表格不替代真实执行。
|
||||
|
||||
### 正式多节点验证
|
||||
|
||||
在私有文件 FORMAL_EXPECTATIONS 保存每个AnyTLS出站的明确预期，完整覆盖订阅中的全部AnyTLS标签。每条结构：`tag`、`host`、整数`port`、`sni`、`uuid`、`expected_exit`、`expected_country`；uuid是该正式用户的实际密码，不能复制进普通参数单或文档。标签必须与订阅精确一致；不重命名节点。
|
||||
|
||||
```sh
|
||||
python3 /opt/dpl-gateway/provision/verify-node.py \
|
||||
--subscription "$QA_DIR/formal-subscription.json" \
|
||||
--expectations "$FORMAL_EXPECTATIONS" --binary "$CLIENT_BINARY" \
|
||||
--port "$LOCAL_SOCKS_PORT" --report "$FORMAL_REPORT" --run
|
||||
```
|
||||
|
||||
不加run时只读取私有JSON并核对结构，不启动客户端、不发网络请求；run每次会先将旧报告失效，避免失败后误用旧成功结果。实时报告记录具体结果与时间，面板正式开放仅接受匹配manifest QA及端点、15分钟内的实时QA报告，正式多节点报告不能代替QA报告。
|
||||
|
||||
### 面板变更命令
|
||||
|
||||
以下命令默认只预览数据库字段变化；确认输出范围后，对同一命令追加 `--verification="$QA_REPORT" --apply` 才写入。append禁止传plans、subscribe-url和group-name；它不会修改现有套餐/正式用户/订阅地址。initialize对应已有套餐中的无组用户关联也属于实际写入范围，必须先看预览。
|
||||
|
||||
```sh
|
||||
cd "$APP_ROOT"
|
||||
php "$PROVISION_ROOT/promote.php" --backup="$BACKUP" --mode=append \
|
||||
--group="$FORMAL_GROUP_ID"
|
||||
```
|
||||
|
||||
首次初始化另用以下命令，按任务需要显式传group/group-name；没有这些参数时不自动改组名：
|
||||
|
||||
```sh
|
||||
cd "$APP_ROOT"
|
||||
php "$PROVISION_ROOT/promote.php" --backup="$BACKUP" --mode=initialize \
|
||||
--plans="$PLAN_IDS" --subscribe-url="$SUBSCRIBE_URL"
|
||||
```
|
||||
|
||||
promotion-journal.json记录允许写入字段、before/after与manifest。相同日志/参数重跑会比较实际字段，已应用则只确认；参数变化、已启动回退、旧promoted.json无新日志或对象冲突时停止。prepared可能意味着数据库未提交、已提交但后续通知/缓存/落盘失败；只有完整before或完整after匹配才恢复，中间/未知状态先人工处理。不要删除日志重新执行。
|
||||
|
||||
## 精确回退入口与证据契约
|
||||
|
||||
触发失败先停止正式开放，隔离新节点。回退按本次 manifest 指定对象执行；不恢复整库、不删除原卷、不停止旧节点。执行前取回当前值，只有当前值仍等于本次 after 才恢复 before；存在后续变化时停止本次自动回退，不写任何字段；专项人工恢复另行核对。
|
||||
|
||||
| 对象 | 必需恢复证据 | 动作及回读 |
|
||||
| --- | --- | --- |
|
||||
| append 新节点正式组 | 该 node_id 的 before/after group_ids、enabled/show 及实际变动字段 | 新rollback.php按日志恢复本次字段并禁用/隐藏节点；不动已有套餐/用户/订阅地址 |
|
||||
| initialize 正式关联 | 新promotion-journal.json、manifest.json；旧before-promotion/promoted仅用于历史专项回退 | 按 ID 逐条恢复本次变动的套餐/用户组、节点组、组名、订阅 URL；缺 after 不盲目回填，以实际变更证据核对 |
|
||||
| QA | test_user_id 与实际禁用状态 | 始终保持本次测试账号禁用、订阅拒绝、不下发；不因回退恢复 QA 活跃 |
|
||||
| 中转 | 新增规则的真实 ID、入口和目标 before/after | 只停用本次新增规则，再回读；原规则保持 |
|
||||
| 节点服务 | 新服务/unit/配置及安装前状态、原服务 PID/启动时间 | 在已授权节点上执行下方停止入口，只停止新 dpl-node，回查旧服务；不删证书/审计目录 |
|
||||
| DNS/证书续期 | 本次记录 ID/旧值/新值、renewal/hook/timer 原文件及权限 | 仅撤回本次增改；已共用证书/调度不能直接删除，恢复后核对下一次调度及原服务 |
|
||||
| 网关/模板/CF | 对应专项恢复点 | append 不应修改；initialize 或专项变更按原[恢复专题](NODE_ONBOARDING.md#回退与排障)逐项执行 |
|
||||
|
||||
节点上的停止入口（确认本服务确为本次新增后）：
|
||||
|
||||
```sh
|
||||
systemctl stop dpl-node
|
||||
systemctl is-active dpl-node || test "$?" -eq 3
|
||||
```
|
||||
|
||||
新日志的面板回退统一入口（默认预览，加apply才执行）：
|
||||
|
||||
```sh
|
||||
cd "$APP_ROOT"
|
||||
php "$PROVISION_ROOT/rollback.php" --backup="$BACKUP"
|
||||
```
|
||||
|
||||
回退逐字段核对当前值仍等于promotion记录的after，任何冲突整体停止；写回本次before、保持QA禁用，并禁用/隐藏本次节点。rollback-journal.json保存回退本身的意图与完成状态，重复执行也需匹配完整结果。旧日志不能自动补造after或升级成新日志。DNS、中转、远程服务、证书、网关/CF属于其他对象，仍按对应专项步骤和恢复证据处理；该面板脚本不包含跨服务器/服务商变更。
|
||||
|
||||
## 本次交付与交接测试范围
|
||||
|
||||
2026-10-16，本地已编写mode隔离的正式开放、参数化回读、多节点验证及面板字段回退，并补stage参数与重跑保护。**按用户要求尚未测试，包括未运行php -l、Python结构/联网验证或任何应用命令；未提交/推送/部署。** 原有运行结果仍引用接入历史，不替代新代码验收。
|
||||
|
||||
奶油应在全新SQLite内存库/隔离应用容器中测试，禁止加载项目本地真实.env、默认数据库/Redis/队列或使用正式凭据：
|
||||
|
||||
1. append仅改变本次node.group_ids、QA.banned，现有订阅设置/全部套餐/正式用户/旧节点逐字段不变；拒绝append夹带plans/url/group-name。
|
||||
2. initialize只修改明确套餐及其无组用户、指定订阅地址/可选组名；拒绝不存在ID、其他组冲突，验证原null订阅设置的创建及回退。
|
||||
3. 默认预览无数据库写入；apply要求近期真实QA报告且QA身份、端点/SNI正确；结构报告、过期/错误QA报告不能通过。
|
||||
4. 回读仅使用本次显式plans；无plans时不查询/统计隐含套餐集合。
|
||||
5. 应用/回退幂等；数据库提交前失败、提交后通知/缓存/日志失败、prepared+完整before/after与混合冲突，不能重复创建或覆盖后续变化；QA始终不复活。
|
||||
6. 多节点JSON用虚构身份；缺失/重复标签、端点/SNI/密码/严格TLS/国家/IP错误均失败。模拟子进程和传输，不发公网请求；默认无run绝不启动进程/开socket，失败后不能留下旧成功报告。
|
||||
7. stage相同完整参数恢复缺失传输文件；参数变化、旧manifest、现有同名/同中转端点资源或已promote时拒绝接管；检查所有秘密文件权限。
|
||||
8. 通过隔离测试之后才另行申请或接受具体线上发布/验收任务，本次授权不包含这些动作。节点命名仍由原维护任务处理，不因测试改名或分配编号。
|
||||
@ -4,7 +4,7 @@
|
||||
|
||||
## 接手顺序
|
||||
|
||||
1. 阅读全局及项目根目录 AGENTS.md、当前用户要求和本目录说明。用户最新明确边界优先，不扩大本次授权。
|
||||
1. 按全局规则先读 CoDocs 项目总览、首次会话的持续维护协议与 dpl 档案，再读项目根目录 AGENTS.md、当前用户要求和本目录说明。用户最新明确边界优先，不扩大本次授权。
|
||||
2. 确认所在仓库、分支、工作区差异与远端；分别识别用户既有修改、本次任务与其他聊天修改。不要覆盖、清理或混入其他任务。
|
||||
3. 按 [架构与配置](ARCHITECTURE.md) 沿实际调用链检查问题；按 [业务约束](BUSINESS_RULES.md) 判断影响。
|
||||
4. 修改后运行与影响范围相关的检查；后台界面必须浏览器验收，包括页面显示、保存、切换、权限。HTTP 200 不代表前端可用。
|
||||
@ -21,9 +21,13 @@
|
||||
| TELEGRAM_TICKETS.md | Telegram 工单接入、绑定、权限及故障处理 |
|
||||
| STATUS.md | 最近验证基线、已知限制与待验收项目 |
|
||||
| CLOUD_FLARE_AUDIT.md | 两机场 Cloudflare 审查、已部署调优、域名轮换与回退及可选未实施项 |
|
||||
| NODE_ONBOARDING.md | 共享资源隔离、订阅网关与新增节点流程及待验收状态 |
|
||||
| NODE_ONBOARDING.md | 两节点链路、命名规则、共享资源与历史验收 |
|
||||
| NODE_RUNBOOK.md | initialize/append参数、脚本入口、前检/证书/验收与面板回退；新本地代码未测试，含奶油测试清单 |
|
||||
| NODE_BATCH.md | 批量接入默认一个子助手的分工与关卡；append并发锁/查重/断点恢复、命令协议与测试边界 |
|
||||
| GATEWAY_ADMIN_PROXY.md | 两网关管理域名、白色机新旧区域反代、独立证书续期、恢复和验收边界 |
|
||||
|
||||
这些说明用于辅助 AI 接手，不代表 AI 会在聊天结束后自动持续运行。应用定时任务在服务器执行，AI 维护任务需要会话或明确配置的自动化来启动。
|
||||
|
||||
正式维护文档放在 docs/maintenance/，纳入 Git 并随相关代码一起更新。项目根目录 AGENTS.md 是 AI 自动接手入口；运行日志、截图、临时脚本与任务产物放在 codex/output/，继续默认忽略。受保护备份和私密配置按运行环境保存，不提交 Git。
|
||||
|
||||
本机共享 wwwroot 权限已于 2026-10-13 收紧；策略、服务身份、限制和恢复入口见 [CoDocs 本地目录权限](/Users/hellcat/Documents/CoDocs/20-基础设施/本地目录权限.md)。仅本机适用，未变更本项目业务或生产权限。
|
||||
|
||||
@ -2,6 +2,28 @@
|
||||
|
||||
文档核对日期：2026-10-11。以下区分本会话检查结果与用户验收确认，不承诺服务器一直保持此状态。
|
||||
|
||||
## 2026-10-17 三节点批次前检（执行中）
|
||||
|
||||
用户授权追加3节点，暂定node-jp03/04/05，展示编号JP - 02/03/04；经用户批准的精确SSH指纹及白色机跳转登录，现场确认root、Debian11、x86_64、JP出口、旧XrayR active、无dpl-node、24443空闲。面板现场只有US - 01与JP - 01，未创建新节点。新增PanelChangeIsolatedTest.php：9 tests / 65 assertions通过；批量进程11 tests重跑通过。面板测试使用真实Eloquent/SQLite内存库与通知替身，不覆盖真实Redis/observer/联网；生产追加尚未完成。
|
||||
|
||||
## 2026-10-16 批量接入默认单子助手规则
|
||||
|
||||
按用户明确要求，在项目 AGENTS.md 登记批量接入触发规则，在 [批量专题](NODE_BATCH.md#多智能体协作流程) 维护唯一完整流程：主协调者统一参数/编号/端口/账本与写入；按用户为节省 token 的最新要求，只启动一个核查验收助手，分阶段核查候选及冲突、审查脚本、测试恢复并独立验收。规定复用、失败/暂停处理、写入及正式开放关卡、最小上下文与避免重复检查。此次仅更新文档，没有启动新接入任务或工作助手，没有生产变更；先前单个显示测试助手不算本批分工验收。规则本地落盘不代表自动执行或生产验证；尚未提交/推送。
|
||||
|
||||
## 2026-10-16 批量调度本地验证
|
||||
|
||||
新增 `batch.py`、`command-adapter.py` 与[批量专题](NODE_BATCH.md)：append 模式，全批前检屏障，独立 VPS 并发，共享面板/DNS/中转锁，跨批资源预留，固定计划与逐阶段现场回读恢复。11 项独立进程离线测试通过，覆盖并发/串行边界、双进程去重、回包丢失与 SIGTERM 中断恢复、参数冲突、错误回执和命令摘要变更；没有加载 Laravel/.env 或访问网络。CoDocs 离线结构检查通过。该测试结果只覆盖新调度器与命令适配器，不覆盖下节其他聊天新写的 PHP 面板脚本及真实订阅验证器。
|
||||
|
||||
当前未提交/推送/部署，未新增线上节点。生产操作通过经过审阅的任务命令 profile 接入；没有内置 DNS/中转后台自动写入或 SSH 登录工具，缺少 handler 的阶段返回等待。真实生产命令、十节点耗时、远端现场及业务验收尚未验证，不能宣称已经支持现网一键接入。
|
||||
|
||||
## 2026-10-16 节点脚本本地草稿（未测试/未发布）
|
||||
|
||||
已在本地补promote.php的append/initialize模式隔离和默认预览，新增operations.php字段日志及rollback.php面板精确回退；readiness.php按显式套餐回读，verify-node.py支持完整多节点预期及IP/国家断言、默认不联网，stage.php移除固定面板地址并核对完整重跑参数。使用与限制见[NODE_RUNBOOK](NODE_RUNBOOK.md)。用户要求测试交给奶油，本次未运行测试/语法检查或应用命令，未改线上，未提交/推送/部署，不能视为已验证可用版本。节点命名未改；本次代码不更改展示名或分配编号。
|
||||
|
||||
## 2026-10-16 节点文档核对（仅离线）
|
||||
|
||||
核对两条既有节点链路并补 CoDocs 美国节点关联；新[执行手册](NODE_RUNBOOK.md)统一参数/前检/证书钩子与演练/验收及精确回退，首次初始化与追加模式分开。现有 promote.php 会写固定订阅地址并关联无组套餐/用户，不适合作为追加默认入口；readiness.php 套餐统计硬编码，verify-node.py 只接受单个 AnyTLS 出站，自动追加与统一回退工具仍缺失。文档已明确限制与现有管理界面操作入口，本次未改脚本或线上。以下运行结果均引用原任务证据，不是本轮复验；节点命名继续由原维护任务处理。
|
||||
|
||||
## 2026-10-16 日本节点
|
||||
|
||||
`node-jp02` 已按既有 Git 运维版本独立部署并加入基础节点组 1，节点/机器 ID 2、AnyTLS、名称 `🇯🇵 日本 JP - 01`。完整隔离订阅经中转实际联网、严格 TLS、日本出口、DNS/路由、上下行上报均通过；证书续期演练及专用 deploy hook 通过。正式有效订阅同时包含美国、日本节点，测试账号禁用且订阅 403。未改价格/套餐限制；原 XrayR 与原中转保留。管理面板实际登录、节点/服务器列表切换、在线状态及已保存参数回读通过；未额外提交编辑表单默认值。参数、Git 版本、备份和边界见 [节点接入记录](NODE_ONBOARDING.md#2026-10-16-日本节点接入)。
|
||||
|
||||
368
scripts/node-onboarding/batch.py
Normal file
368
scripts/node-onboarding/batch.py
Normal file
@ -0,0 +1,368 @@
|
||||
#!/usr/bin/env python3
|
||||
"""DPL append-only batch coordinator. Adapters must probe before and after writes."""
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import contextlib
|
||||
import fcntl
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import signal
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
PHASES = ('stage', 'dns', 'provision', 'relay', 'verify', 'publish', 'audit')
|
||||
SHARED = {'stage': 'panel', 'dns': 'dns', 'relay': 'relay', 'publish': 'panel'}
|
||||
STOP = threading.Event()
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True, ensure_ascii=False,
|
||||
separators=(',', ':')).encode()).hexdigest()
|
||||
|
||||
|
||||
def read_json(path):
|
||||
return json.loads(Path(path).read_text())
|
||||
|
||||
|
||||
def atomic_json(path, value):
|
||||
path = Path(path)
|
||||
tmp = path.with_name(path.name + '.tmp-' + str(os.getpid()) + '-' + str(threading.get_ident()))
|
||||
with tmp.open('x') as f:
|
||||
os.chmod(tmp, 0o600)
|
||||
json.dump(value, f, ensure_ascii=False, sort_keys=True)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def validate(plan):
|
||||
if plan.get('version') != 1 or plan.get('mode') != 'append':
|
||||
raise ValueError('Only version=1, mode=append is supported')
|
||||
if not re.fullmatch(r'[a-z0-9-]{1,64}', plan.get('batch_id', '')):
|
||||
raise ValueError('Invalid batch_id')
|
||||
context = plan.get('context', {})
|
||||
required = ('panel_vps', 'gateway_vps', 'jump_vps', 'app_root', 'panel_url',
|
||||
'subscription_url', 'provision_commit', 'bundle', 'client_binary')
|
||||
if any(not context.get(k) for k in required):
|
||||
raise ValueError('Missing explicit context fields')
|
||||
if not re.fullmatch(r'[0-9a-f]{40}', context['provision_commit']):
|
||||
raise ValueError('A full reviewed, pushed Git commit is required')
|
||||
for k in ('panel_url', 'subscription_url'):
|
||||
if not re.fullmatch(r'https://[a-z0-9.-]+', context[k]):
|
||||
raise ValueError('HTTPS origin without token/path required: ' + k)
|
||||
adapter = plan.get('adapter')
|
||||
if not isinstance(adapter, list) or not adapter or not all(isinstance(x, str) for x in adapter):
|
||||
raise ValueError('adapter must be an argv array, never a shell string')
|
||||
if not plan.get('nodes') or not isinstance(plan['nodes'], list):
|
||||
raise ValueError('A nonempty node list is required')
|
||||
seen = {k: set() for k in ('key', 'name', 'vps_id', 'node_address', 'tls_domain', 'relay', 'socks_port', 'number')}
|
||||
for n in plan['nodes']:
|
||||
if any(k not in n for k in ('key', 'country', 'sequence', 'flag', 'country_name', 'name',
|
||||
'vps_id', 'node_address', 'listen_port', 'relay_host',
|
||||
'relay_port', 'tls_domain', 'formal_group_id',
|
||||
'expected_exit', 'socks_port')):
|
||||
raise ValueError('Incomplete node specification')
|
||||
if not re.fullmatch(r'[a-z0-9-]{1,48}', n['key']):
|
||||
raise ValueError('Invalid deployment key')
|
||||
if not re.fullmatch(r'[A-Z]{2}', n['country']):
|
||||
raise ValueError('Country must be a two-letter code')
|
||||
if type(n['sequence']) is not int or not 1 <= n['sequence'] <= 99:
|
||||
raise ValueError('Display sequence must be 01..99')
|
||||
expected = f"{n['flag']} {n['country_name']} {n['country']} - {n['sequence']:02d}"
|
||||
flag = ''.join(chr(0x1F1E6 + ord(x) - ord('A')) for x in n['country'])
|
||||
if n['name'] != expected or n['flag'] != flag:
|
||||
raise ValueError('Display name/flag does not match country and sequence')
|
||||
for k in ('listen_port', 'relay_port', 'socks_port'):
|
||||
if type(n[k]) is not int or not 1024 <= n[k] <= 65535:
|
||||
raise ValueError('Invalid port: ' + k)
|
||||
if type(n['formal_group_id']) is not int or n['formal_group_id'] < 1:
|
||||
raise ValueError('An existing formal group is required')
|
||||
ipaddress.ip_address(n['node_address'])
|
||||
ipaddress.ip_address(n['expected_exit'])
|
||||
for k in ('relay_host', 'tls_domain'):
|
||||
if not re.fullmatch(r'[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?\.[a-z]{2,}', n[k]):
|
||||
raise ValueError('Invalid DNS name: ' + k)
|
||||
for k in ('vps_id',):
|
||||
if not re.fullmatch(r'VPS-[0-9]{3}', n[k]):
|
||||
raise ValueError('Use the exact CoDocs VPS ID')
|
||||
identities = {k: n[k] for k in ('key', 'name', 'vps_id', 'node_address', 'tls_domain', 'socks_port')}
|
||||
identities.update(relay=(n['relay_host'], n['relay_port']), number=(n['country'], n['sequence']))
|
||||
for k, value in identities.items():
|
||||
if value in seen[k]:
|
||||
raise ValueError('Duplicate batch identity: ' + k)
|
||||
seen[k].add(value)
|
||||
# The plan carries references, never secret values. Adapters load credentials themselves.
|
||||
def check(value):
|
||||
if isinstance(value, dict):
|
||||
for k, v in value.items():
|
||||
if re.search(r'password|secret|token|private_key', k, re.I):
|
||||
raise ValueError('Secret fields are not allowed in a batch plan')
|
||||
check(v)
|
||||
elif isinstance(value, list):
|
||||
for v in value:
|
||||
check(v)
|
||||
check(plan)
|
||||
return plan
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def lock(root, name):
|
||||
path = root / ('lock-' + hashlib.sha256(name.encode()).hexdigest())
|
||||
with path.open('a+') as f:
|
||||
os.chmod(path, 0o600)
|
||||
while True:
|
||||
try:
|
||||
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
break
|
||||
except BlockingIOError:
|
||||
if STOP.wait(.1):
|
||||
raise RuntimeError('Stopped while waiting for resource lock')
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
fcntl.flock(f, fcntl.LOCK_UN)
|
||||
|
||||
|
||||
class Ledger:
|
||||
def __init__(self, root):
|
||||
self.root = Path(root).resolve()
|
||||
if self.root == Path('/'):
|
||||
raise ValueError('Invalid state root')
|
||||
self.root.mkdir(parents=True, mode=0o700, exist_ok=True)
|
||||
os.chmod(self.root, 0o700)
|
||||
self.path = self.root / 'registry.sqlite3'
|
||||
with self.db() as c:
|
||||
c.executescript('''
|
||||
CREATE TABLE IF NOT EXISTS batches(id TEXT PRIMARY KEY, digest TEXT NOT NULL);
|
||||
CREATE TABLE IF NOT EXISTS claims(namespace TEXT, kind TEXT, value TEXT,
|
||||
owner TEXT NOT NULL, PRIMARY KEY(namespace,kind,value));
|
||||
CREATE TABLE IF NOT EXISTS phases(batch TEXT, node TEXT, phase TEXT,
|
||||
status TEXT NOT NULL, attempts INTEGER NOT NULL DEFAULT 0,
|
||||
updated REAL NOT NULL, PRIMARY KEY(batch,node,phase));
|
||||
''')
|
||||
os.chmod(self.path, 0o600)
|
||||
|
||||
def db(self):
|
||||
c = sqlite3.connect(self.path, timeout=30)
|
||||
c.execute('PRAGMA synchronous=FULL')
|
||||
return c
|
||||
|
||||
def reserve(self, plan):
|
||||
namespace = plan['context']['panel_vps'] + ':' + plan['context']['app_root']
|
||||
with self.db() as c:
|
||||
c.execute('BEGIN IMMEDIATE')
|
||||
prior = c.execute('SELECT digest FROM batches WHERE id=?', (plan['batch_id'],)).fetchone()
|
||||
if prior and prior[0] != digest(plan):
|
||||
raise ValueError('Batch plan changed; resume requires the original exact plan')
|
||||
for n in plan['nodes']:
|
||||
owner = digest({'context': plan['context'], 'node': n})
|
||||
claims = {k: n[k] for k in ('key', 'name', 'vps_id', 'node_address', 'tls_domain')}
|
||||
claims['relay'] = [n['relay_host'], n['relay_port']]
|
||||
claims['number'] = [n['country'], n['sequence']]
|
||||
for kind, value in claims.items():
|
||||
scope = namespace
|
||||
if kind == 'relay':
|
||||
scope = 'relay:' + n['relay_host']
|
||||
elif kind in ('vps_id', 'node_address', 'tls_domain'):
|
||||
scope = 'infrastructure'
|
||||
encoded = json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
existing = c.execute('SELECT owner FROM claims WHERE namespace=? AND kind=? AND value=?',
|
||||
(scope, kind, encoded)).fetchone()
|
||||
if existing and existing[0] != owner:
|
||||
raise ValueError('Another batch reserved identity: ' + kind)
|
||||
c.execute('INSERT OR IGNORE INTO claims VALUES(?,?,?,?)', (scope, kind, encoded, owner))
|
||||
# SOCKS listeners are shared by all batches using this gateway.
|
||||
scope, value = 'gateway:' + plan['context']['gateway_vps'], str(n['socks_port'])
|
||||
old = c.execute('SELECT owner FROM claims WHERE namespace=? AND kind=? AND value=?',
|
||||
(scope, 'socks_port', value)).fetchone()
|
||||
if old and old[0] != owner:
|
||||
raise ValueError('Another batch reserved the gateway SOCKS port')
|
||||
c.execute('INSERT OR IGNORE INTO claims VALUES(?,?,?,?)', (scope, 'socks_port', value, owner))
|
||||
c.execute('INSERT OR IGNORE INTO batches VALUES(?,?)', (plan['batch_id'], digest(plan)))
|
||||
|
||||
def state(self, batch, key, phase):
|
||||
with self.db() as c:
|
||||
row = c.execute('SELECT status FROM phases WHERE batch=? AND node=? AND phase=?',
|
||||
(batch, key, phase)).fetchone()
|
||||
return row[0] if row else 'pending'
|
||||
|
||||
def mark(self, batch, key, phase, status, attempt=False):
|
||||
with self.db() as c:
|
||||
c.execute('''INSERT INTO phases VALUES(?,?,?,?,?,?) ON CONFLICT(batch,node,phase)
|
||||
DO UPDATE SET status=excluded.status,attempts=phases.attempts+?,updated=excluded.updated''',
|
||||
(batch, key, phase, status, int(attempt), time.time(), int(attempt)))
|
||||
|
||||
|
||||
class Coordinator:
|
||||
def __init__(self, plan, root, jobs=4, timeout=900):
|
||||
self.plan = validate(plan)
|
||||
self.ledger = Ledger(root)
|
||||
self.jobs = jobs
|
||||
self.timeout = timeout
|
||||
self.ledger.reserve(plan)
|
||||
self.dir = self.ledger.root / plan['batch_id']
|
||||
self.dir.mkdir(mode=0o700, exist_ok=True)
|
||||
self.plan_file = self.dir / 'plan.json'
|
||||
atomic_json(self.plan_file, plan)
|
||||
|
||||
def call(self, n, phase, action):
|
||||
fingerprint = digest({'context': self.plan['context'], 'node': n})
|
||||
path = self.dir / (n['key'] + '.json')
|
||||
atomic_json(path, dict(n, request_sha256=fingerprint))
|
||||
argv = self.plan['adapter'] + ['--plan', str(self.plan_file), '--node', str(path),
|
||||
'--phase', phase, '--action', action, '--state-dir', str(self.dir)]
|
||||
with subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
|
||||
start_new_session=True) as p:
|
||||
deadline = time.monotonic() + self.timeout
|
||||
while True:
|
||||
try:
|
||||
output, _ = p.communicate(timeout=.2)
|
||||
break
|
||||
except subprocess.TimeoutExpired:
|
||||
if STOP.is_set() or time.monotonic() >= deadline:
|
||||
os.killpg(p.pid, signal.SIGTERM)
|
||||
try:
|
||||
p.communicate(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(p.pid, signal.SIGKILL)
|
||||
p.communicate()
|
||||
raise RuntimeError('Adapter interrupted; remote outcome must be probed')
|
||||
if p.returncode or len(output) > 65536:
|
||||
raise RuntimeError('Adapter failed; no stdout/stderr persisted')
|
||||
r = json.loads(output)
|
||||
if (r.get('key'), r.get('phase'), r.get('request_sha256')) != (n['key'], phase, fingerprint):
|
||||
raise RuntimeError('Adapter receipt identity mismatch')
|
||||
if r.get('status') not in ('ready', 'absent', 'waiting', 'unknown', 'conflict'):
|
||||
raise RuntimeError('Invalid adapter status')
|
||||
return r
|
||||
|
||||
def phase(self, n, phase):
|
||||
try:
|
||||
return self._phase(n, phase)
|
||||
except Exception:
|
||||
self.ledger.mark(self.plan['batch_id'], n['key'], phase, 'unknown')
|
||||
return False
|
||||
|
||||
def _phase(self, n, phase):
|
||||
batch, key = self.plan['batch_id'], n['key']
|
||||
resource = SHARED.get(phase, n['vps_id'])
|
||||
# Common panel/dns/relay lock names across batches and processes.
|
||||
namespace = self.plan['context']['panel_vps']
|
||||
if resource in ('dns', 'relay'):
|
||||
namespace = 'dpl-external'
|
||||
with lock(self.ledger.root, namespace + ':' + resource):
|
||||
r = self.call(n, phase, 'probe')
|
||||
if r['status'] == 'ready':
|
||||
self.ledger.mark(batch, key, phase, 'done')
|
||||
return True
|
||||
if r['status'] == 'conflict':
|
||||
STOP.set()
|
||||
if phase in ('preflight', 'verify', 'audit') or r['status'] != 'absent' or r.get('retry_safe') is not True:
|
||||
self.ledger.mark(batch, key, phase, r['status'])
|
||||
return False
|
||||
if STOP.is_set():
|
||||
return False
|
||||
# Persist intent before a write; never infer success from the command exit code.
|
||||
self.ledger.mark(batch, key, phase, 'running', attempt=True)
|
||||
applied = self.call(n, phase, 'apply')
|
||||
if applied['status'] == 'conflict':
|
||||
self.ledger.mark(batch, key, phase, 'conflict')
|
||||
STOP.set()
|
||||
return False
|
||||
r = self.call(n, phase, 'probe')
|
||||
self.ledger.mark(batch, key, phase, 'done' if r['status'] == 'ready' else r['status'])
|
||||
if r['status'] == 'conflict':
|
||||
STOP.set()
|
||||
return r['status'] == 'ready'
|
||||
|
||||
def worker(self, n):
|
||||
try:
|
||||
with lock(self.ledger.root, 'node:' + self.plan['context']['panel_vps'] + ':' + n['key']):
|
||||
self.ledger.mark(self.plan['batch_id'], n['key'], 'coordinator', 'running')
|
||||
# Already finished batches are re-audited; disabled QA need not be re-enabled.
|
||||
if self.ledger.state(self.plan['batch_id'], n['key'], 'audit') == 'done':
|
||||
success = self.phase(n, 'audit')
|
||||
else:
|
||||
success = True
|
||||
for phase in PHASES:
|
||||
if STOP.is_set() or not self.phase(n, phase):
|
||||
success = False
|
||||
break
|
||||
self.ledger.mark(self.plan['batch_id'], n['key'], 'coordinator', 'done' if success else 'waiting')
|
||||
return success
|
||||
except Exception:
|
||||
# Never print adapter payloads, commands, credentials or exception contents.
|
||||
self.ledger.mark(self.plan['batch_id'], n['key'], 'coordinator', 'unknown')
|
||||
return False
|
||||
|
||||
def run(self):
|
||||
# Whole-batch barrier: no staging until every candidate passed current preflight.
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=self.jobs) as pool:
|
||||
def check(n):
|
||||
try:
|
||||
return self.phase(n, 'preflight')
|
||||
except Exception:
|
||||
self.ledger.mark(self.plan['batch_id'], n['key'], 'preflight', 'unknown')
|
||||
return False
|
||||
checks = list(pool.map(check, self.plan['nodes']))
|
||||
if not all(checks):
|
||||
return False
|
||||
return all(list(pool.map(self.worker, self.plan['nodes'])))
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('command', choices=('plan', 'run', 'status'))
|
||||
p.add_argument('--plan', required=True)
|
||||
p.add_argument('--state-root', required=True)
|
||||
p.add_argument('--jobs', type=int, default=4)
|
||||
p.add_argument('--timeout', type=int, default=900)
|
||||
p.add_argument('--execute', action='store_true', help='Required for a run that may write remotely')
|
||||
a = p.parse_args()
|
||||
os.umask(0o077)
|
||||
if not 1 <= a.jobs <= 16 or not 1 <= a.timeout <= 3600:
|
||||
p.error('jobs must be 1..16; timeout 1..3600')
|
||||
try:
|
||||
plan = validate(read_json(a.plan))
|
||||
if a.command == 'plan':
|
||||
print(json.dumps({'batch_id': plan['batch_id'], 'nodes': len(plan['nodes']),
|
||||
'plan_sha256': digest(plan), 'writes': False}))
|
||||
return 0
|
||||
if a.command == 'run' and not a.execute:
|
||||
p.error('run requires --execute; plan does not perform remote requests')
|
||||
if a.command == 'status':
|
||||
path = Path(a.state_root).resolve() / 'registry.sqlite3'
|
||||
rows = {}
|
||||
if path.exists():
|
||||
with sqlite3.connect(path.as_uri() + '?mode=ro', uri=True) as c:
|
||||
prior = c.execute('SELECT digest FROM batches WHERE id=?', (plan['batch_id'],)).fetchone()
|
||||
if prior and prior[0] != digest(plan):
|
||||
raise ValueError('Batch plan differs from its reserved plan')
|
||||
rows = {(key, phase): status for key, phase, status in c.execute(
|
||||
'SELECT node,phase,status FROM phases WHERE batch=?', (plan['batch_id'],))}
|
||||
print(json.dumps({n['key']: {phase: rows.get((n['key'], phase), 'pending')
|
||||
for phase in ('preflight',) + PHASES + ('coordinator',)}
|
||||
for n in plan['nodes']}))
|
||||
return 0
|
||||
runner = Coordinator(plan, a.state_root, a.jobs, a.timeout)
|
||||
for sig in (signal.SIGINT, signal.SIGTERM):
|
||||
signal.signal(sig, lambda *_: STOP.set())
|
||||
success = runner.run()
|
||||
print(json.dumps({'batch_id': plan['batch_id'], 'complete': success,
|
||||
'next': None if success else 'Inspect status, resolve waiting/conflict, rerun the exact plan'}))
|
||||
return 0 if success else 2
|
||||
except Exception as exc:
|
||||
# Validation errors have no adapter/credential material.
|
||||
print(json.dumps({'error': str(exc) if isinstance(exc, ValueError) else 'Coordinator failed; inspect protected state'}))
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
66
scripts/node-onboarding/command-adapter.py
Normal file
66
scripts/node-onboarding/command-adapter.py
Normal file
@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run reviewed, pinned site-specific commands for batch.py without a shell.
|
||||
|
||||
Missing integration is a waiting gate, never an assumed successful operation.
|
||||
Handlers receive paths to the frozen request, not passwords on the command line.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
for field in ('plan', 'node', 'phase', 'action', 'state-dir'):
|
||||
p.add_argument('--' + field, required=True)
|
||||
a = p.parse_args()
|
||||
node = json.loads(Path(a.node).read_text())
|
||||
plan = json.loads(Path(a.plan).read_text())
|
||||
identity = {'key': node['key'], 'phase': a.phase, 'request_sha256': node['request_sha256']}
|
||||
try:
|
||||
profile = plan['command_profile']
|
||||
# Every runnable local script/config must be pinned; pinning the interpreter
|
||||
# alone does not pin scripts, remote code or the commands' dependencies.
|
||||
files = profile['files']
|
||||
if not files:
|
||||
raise ValueError('No pinned handler files')
|
||||
for filename, expected in files.items():
|
||||
if not Path(filename).is_absolute():
|
||||
raise ValueError('Absolute handler paths required')
|
||||
if hashlib.sha256(Path(filename).read_bytes()).hexdigest() != expected:
|
||||
raise ValueError('Handler changed')
|
||||
command = profile['phases'].get(a.phase, {}).get(a.action)
|
||||
if command is None:
|
||||
result = dict(identity, status='waiting', reason='handler_not_configured')
|
||||
else:
|
||||
if not isinstance(command, list) or not command or not all(isinstance(x, str) for x in command):
|
||||
raise ValueError('Command must be argv')
|
||||
values = dict(plan=a.plan, node=a.node, state_dir=a.state_dir,
|
||||
key=node['key'], phase=a.phase, action=a.action)
|
||||
argv = [x.format_map(values) for x in command]
|
||||
if not Path(argv[0]).is_absolute():
|
||||
raise ValueError('Absolute executable required')
|
||||
completed = subprocess.run(argv, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
|
||||
if completed.returncode or len(completed.stdout) > 65536:
|
||||
raise ValueError('Handler failed')
|
||||
result = json.loads(completed.stdout)
|
||||
if any(result.get(k) != v for k, v in identity.items()):
|
||||
raise ValueError('Wrong handler receipt')
|
||||
if result.get('status') not in ('ready', 'absent', 'waiting', 'unknown', 'conflict'):
|
||||
raise ValueError('Wrong handler status')
|
||||
# Only structured nonsecret receipt fields are allowed through to the runner.
|
||||
safe = {**identity, 'status': result['status']}
|
||||
if result.get('retry_safe') is True:
|
||||
safe['retry_safe'] = True
|
||||
print(json.dumps(safe))
|
||||
return 0
|
||||
except Exception:
|
||||
print(json.dumps(dict(identity, status='unknown')))
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
332
scripts/node-onboarding/operations.php
Normal file
332
scripts/node-onboarding/operations.php
Normal file
@ -0,0 +1,332 @@
|
||||
<?php
|
||||
/** Shared CLI helpers; loading this file does not bootstrap Laravel or open a connection. */
|
||||
namespace DplOnboarding;
|
||||
|
||||
use App\Models\{Server, ServerGroup, User, Plan, Setting};
|
||||
use Illuminate\Support\Facades\{DB, Cache};
|
||||
use RuntimeException;
|
||||
|
||||
function ids(string $value): array
|
||||
{
|
||||
if (!preg_match('/^[1-9][0-9]*(,[1-9][0-9]*)*$/D', $value)) {
|
||||
throw new RuntimeException('Expected explicit comma-separated positive IDs');
|
||||
}
|
||||
$ids = array_map('intval', explode(',', $value));
|
||||
if (count($ids) !== count(array_unique($ids))) throw new RuntimeException('Duplicate IDs');
|
||||
return $ids;
|
||||
}
|
||||
|
||||
function readJson(string $path): array
|
||||
{
|
||||
if (!is_file($path) || is_link($path)) throw new RuntimeException('Required regular journal/manifest missing');
|
||||
$data = json_decode(file_get_contents($path), true, flags: JSON_THROW_ON_ERROR);
|
||||
if (!is_array($data)) throw new RuntimeException('Invalid JSON document');
|
||||
return $data;
|
||||
}
|
||||
|
||||
function writeJson(string $path, array $data): void
|
||||
{
|
||||
if (is_link($path)) throw new RuntimeException('Refusing journal symlink');
|
||||
$tmp = tempnam(dirname($path), '.onboarding-');
|
||||
if ($tmp === false) throw new RuntimeException('Cannot create journal');
|
||||
try {
|
||||
chmod($tmp, 0600);
|
||||
$body = json_encode($data, JSON_THROW_ON_ERROR | JSON_PRETTY_PRINT | JSON_UNESCAPED_UNICODE) . "\n";
|
||||
if (file_put_contents($tmp, $body) !== strlen($body) || !rename($tmp, $path)) {
|
||||
throw new RuntimeException('Cannot persist journal');
|
||||
}
|
||||
} finally {
|
||||
if (is_file($tmp)) unlink($tmp);
|
||||
}
|
||||
}
|
||||
|
||||
function manifest(string $dir): array
|
||||
{
|
||||
$m = readJson($dir . '/manifest.json');
|
||||
foreach (['node_id','machine_id','group_id','test_user_id'] as $field) {
|
||||
if (!isset($m[$field]) || !preg_match('/^[1-9][0-9]*$/D', (string)$m[$field])) {
|
||||
throw new RuntimeException('Manifest missing positive object ID');
|
||||
}
|
||||
$m[$field] = (int)$m[$field];
|
||||
}
|
||||
if (!isset($m['key']) || !preg_match('/^[a-z0-9-]+$/D', $m['key'])) throw new RuntimeException('Invalid manifest key');
|
||||
return $m;
|
||||
}
|
||||
|
||||
function locked(string $dir, callable $fn): mixed
|
||||
{
|
||||
if (!is_dir($dir) || is_link($dir) || (fileperms($dir)&0077)!==0) throw new RuntimeException('Existing private backup directory required');
|
||||
$path = $dir . '/operation.lock';
|
||||
if (is_link($path)) throw new RuntimeException('Refusing lock symlink');
|
||||
$h = fopen($path, 'c');
|
||||
if ($h === false) throw new RuntimeException('Cannot open operation lock');
|
||||
chmod($path, 0600);
|
||||
try {
|
||||
if (!flock($h, LOCK_EX | LOCK_NB)) throw new RuntimeException('Another operation is active');
|
||||
return $fn();
|
||||
} finally { flock($h, LOCK_UN); fclose($h); }
|
||||
}
|
||||
|
||||
function boot(): void
|
||||
{
|
||||
require getcwd() . '/vendor/autoload.php';
|
||||
$app = require getcwd() . '/bootstrap/app.php';
|
||||
$app->make(\Illuminate\Contracts\Console\Kernel::class)->bootstrap();
|
||||
}
|
||||
|
||||
final class PanelChange
|
||||
{
|
||||
private const MODELS = ['node'=>Server::class,'qa'=>User::class,'user'=>User::class,
|
||||
'plan'=>Plan::class,'group'=>ServerGroup::class,'setting'=>Setting::class];
|
||||
private const FIELDS = ['node'=>['group_ids'],'qa'=>['banned'],'user'=>['group_id'],
|
||||
'plan'=>['group_id'],'group'=>['name'],'setting'=>['value']];
|
||||
|
||||
private static function normalize(string $field, mixed $value): mixed
|
||||
{
|
||||
return match ($field) {
|
||||
'group_ids' => array_map('strval', $value ?? []),
|
||||
'group_id' => $value === null ? null : (int)$value,
|
||||
'banned','enabled','show' => (bool)$value,
|
||||
default => $value,
|
||||
};
|
||||
}
|
||||
|
||||
private static function object(string $kind, int|string $id): mixed
|
||||
{
|
||||
$class = self::MODELS[$kind] ?? throw new RuntimeException('Unsupported journal object');
|
||||
return $class::where($kind === 'setting' ? 'name' : 'id', $id)->lockForUpdate()->first();
|
||||
}
|
||||
|
||||
private static function value(mixed $model, string $field): mixed
|
||||
{
|
||||
return $model === null ? null : self::normalize($field, $model->getAttribute($field));
|
||||
}
|
||||
|
||||
private static function operation(string $kind, mixed $model, string $field, mixed $after, int|string|null $id = null): ?array
|
||||
{
|
||||
if (!in_array($field, self::FIELDS[$kind] ?? [], true)) throw new RuntimeException('Unsupported journal field');
|
||||
$before = self::value($model, $field);
|
||||
$after = self::normalize($field, $after);
|
||||
if ($model !== null && $before === $after) return null;
|
||||
return ['kind'=>$kind,'id'=>$id ?? $model->getKey(),'field'=>$field,
|
||||
'before_exists'=>$model !== null,'before'=>$before,'after'=>$after];
|
||||
}
|
||||
|
||||
private static function context(array $m): array
|
||||
{
|
||||
$node = self::object('node', $m['node_id']) ?? throw new RuntimeException('Manifest node missing');
|
||||
$qa = self::object('qa', $m['test_user_id']) ?? throw new RuntimeException('Manifest QA missing');
|
||||
if ((int)$node->machine_id !== $m['machine_id'] || $node->type !== 'anytls') {
|
||||
throw new RuntimeException('Manifest node/machine mismatch');
|
||||
}
|
||||
if ($qa->is_admin || (int)$qa->group_id !== $m['group_id'] ||
|
||||
!str_starts_with($qa->email, 'node-qa-' . $m['key'] . '-') ||
|
||||
!str_ends_with($qa->email, '@example.invalid')) {
|
||||
throw new RuntimeException('Manifest does not identify an isolated QA user');
|
||||
}
|
||||
return [$node, $qa];
|
||||
}
|
||||
|
||||
private static function checkJournal(array $j, array $m): void
|
||||
{
|
||||
if (($j['version'] ?? null) !== 1 || ($j['manifest'] ?? null) !== $m || !is_array($j['operations'] ?? null)) {
|
||||
throw new RuntimeException('Journal/manifest mismatch or unsupported version');
|
||||
}
|
||||
foreach ($j['operations'] as $op) {
|
||||
if (!in_array($op['field'], self::FIELDS[$op['kind']] ?? [], true)) throw new RuntimeException('Invalid journal operation');
|
||||
if ($op['kind']==='node' && $op['id']!==$m['node_id']) throw new RuntimeException('Wrong node in journal');
|
||||
if ($op['kind']==='qa' && $op['id']!==$m['test_user_id']) throw new RuntimeException('Wrong QA in journal');
|
||||
if ($op['kind']==='setting' && $op['id']!=='subscribe_url') throw new RuntimeException('Wrong setting in journal');
|
||||
if ($j['request']['mode']==='append' && !in_array($op['kind'], ['node','qa'], true)) {
|
||||
throw new RuntimeException('Append journal contains shared-resource writes');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static function current(array $op): array
|
||||
{
|
||||
$model = self::object($op['kind'], $op['id']);
|
||||
return [$model, $model !== null, self::value($model, $op['field'])];
|
||||
}
|
||||
|
||||
private static function applyOperation(array $op): void
|
||||
{
|
||||
[$model, $exists, $value] = self::current($op);
|
||||
if ($exists !== $op['before_exists'] || $value !== $op['before']) throw new RuntimeException('Object changed before application');
|
||||
if (!$exists) {
|
||||
if ($op['kind'] !== 'setting') throw new RuntimeException('Missing target');
|
||||
$model = new Setting(['name'=>$op['id']]);
|
||||
}
|
||||
$model->setAttribute($op['field'], $op['after']);
|
||||
// Keep model observers for node full-sync notifications; QA does not send notifications.
|
||||
$op['kind'] === 'qa' ? $model->saveQuietly() : $model->save();
|
||||
}
|
||||
|
||||
private static function flushSettings(array $ops): void
|
||||
{
|
||||
if (array_filter($ops, fn($op) => $op['kind']==='setting')) {
|
||||
// Called after commit; clearing request-local state also makes retry recovery safe.
|
||||
Cache::store('redis')->forget(\App\Support\Setting::CACHE_KEY);
|
||||
app()->forgetInstance(\App\Support\Setting::class);
|
||||
}
|
||||
}
|
||||
|
||||
private static function notifyCommitted(array $m): void
|
||||
{
|
||||
// Explicitly repeat idempotent invalidations after commit. If an observer's
|
||||
// after-commit callback failed, prepared+after recovery must still notify.
|
||||
\App\Services\NodeSyncService::notifyFullSync($m['node_id']);
|
||||
\App\Services\NodeSyncService::notifyMachineNodesChanged($m['machine_id']);
|
||||
}
|
||||
|
||||
private static function verifyEvidence(array $request, array $m, mixed $node): void
|
||||
{
|
||||
$report = readJson($request['verification'] ?? throw new RuntimeException('Missing --verification live QA report'));
|
||||
$at = $report['verified_at'] ?? 0;
|
||||
if (($report['state'] ?? '') !== 'live_verified' || ($report['qa_user_id'] ?? null) !== $m['test_user_id'] ||
|
||||
!is_int($at) || $at > time() || time()-$at > 900) {
|
||||
throw new RuntimeException('Recent live verification for this manifest QA required');
|
||||
}
|
||||
$sni = $node->protocol_settings['tls']['server_name'] ?? null;
|
||||
$targets = array_filter($report['results'] ?? [], fn($row) =>
|
||||
($row['live'] ?? false) === true && ($row['strict_tls'] ?? false) === true &&
|
||||
($row['relay'] ?? null) === $node->host && (int)($row['relay_port'] ?? 0) === (int)$node->port &&
|
||||
($row['sni'] ?? null) === $sni);
|
||||
if (count($targets)!==1) throw new RuntimeException('Live evidence endpoint/SNI mismatch');
|
||||
}
|
||||
|
||||
public static function promote(string $dir, array $request, bool $apply): array
|
||||
{
|
||||
$mode = $request['mode'] ?? '';
|
||||
if (!in_array($mode, ['append','initialize'], true)) throw new RuntimeException('Explicit append/initialize mode required');
|
||||
$request['group'] = isset($request['group']) ? (int)$request['group'] : null;
|
||||
if ($mode==='append' && (!$request['group'] || isset($request['plans']) || isset($request['subscribe-url']) || isset($request['group-name']))) {
|
||||
throw new RuntimeException('Append requires group and forbids plans, subscribe-url and group-name');
|
||||
}
|
||||
if ($mode==='initialize') {
|
||||
$request['plan_ids'] = ids($request['plans'] ?? '');
|
||||
$url = $request['subscribe-url'] ?? '';
|
||||
$parts = parse_url($url);
|
||||
if (!is_array($parts) || ($parts['scheme'] ?? '')!=='https' || empty($parts['host']) || isset($parts['user']) || isset($parts['pass']) || isset($parts['query']) || isset($parts['fragment'])) {
|
||||
throw new RuntimeException('Initialize requires an explicit HTTPS subscribe-url without credentials/query');
|
||||
}
|
||||
}
|
||||
ksort($request);
|
||||
$m = manifest($dir); $path = $dir . '/promotion-journal.json';
|
||||
$result = DB::transaction(function () use ($dir,$request,$mode,$m,$path,$apply) {
|
||||
[$node,$qa] = self::context($m);
|
||||
if (is_file($path)) {
|
||||
$j = readJson($path); self::checkJournal($j,$m);
|
||||
if ($j['request'] !== $request) throw new RuntimeException('Existing operation has different parameters');
|
||||
if (is_file($dir.'/rollback-journal.json')) throw new RuntimeException('Rollback started; use a new reviewed task');
|
||||
$before = $after = true;
|
||||
foreach ($j['operations'] as $op) {
|
||||
[, $exists,$value] = self::current($op);
|
||||
$before = $before && $exists===$op['before_exists'] && $value===$op['before'];
|
||||
$after = $after && $exists && $value===$op['after'];
|
||||
}
|
||||
if (!$after && (!$before || $j['state']!=='prepared')) throw new RuntimeException('Journal conflict; inspect before retrying');
|
||||
if (!$apply) return ['state'=>$after?'already_applied':'preview','journal'=>$j];
|
||||
if (!$after) {
|
||||
self::verifyEvidence($request,$m,$node);
|
||||
if (!$node->is_online || (int)$qa->u+(int)$qa->d<=0 || $qa->banned) throw new RuntimeException('QA evidence no longer ready');
|
||||
foreach ($j['operations'] as $op) self::applyOperation($op);
|
||||
}
|
||||
return ['state'=>'applied','journal'=>$j];
|
||||
}
|
||||
if (is_file($dir.'/promoted.json')) throw new RuntimeException('Legacy promotion exists without new journal; do not adopt automatically');
|
||||
if (!$node->enabled || !$node->show || !$node->is_online || (int)$qa->u + (int)$qa->d <= 0 || $qa->banned) throw new RuntimeException('Fresh online/active QA traffic evidence missing');
|
||||
if (self::value($node,'group_ids') !== [(string)$m['group_id']]) throw new RuntimeException('Node no longer in its isolated stage group');
|
||||
$groupId = $request['group'] ?? $m['group_id'];
|
||||
$group = self::object('group',$groupId) ?? throw new RuntimeException('Target group missing');
|
||||
if ($mode==='append' && $groupId===$m['group_id']) throw new RuntimeException('Append requires an existing formal group');
|
||||
$ops = [self::operation('node',$node,'group_ids',[(string)$groupId]), self::operation('qa',$qa,'banned',true)];
|
||||
if ($mode==='initialize') {
|
||||
$plans = Plan::whereIn('id',$request['plan_ids'])->orderBy('id')->lockForUpdate()->get();
|
||||
if ($plans->count()!==count($request['plan_ids'])) throw new RuntimeException('Plan IDs do not all exist');
|
||||
foreach ($plans as $plan) {
|
||||
if ($plan->group_id !== null && (int)$plan->group_id!==$groupId) throw new RuntimeException('Plan belongs to another group');
|
||||
$ops[] = self::operation('plan',$plan,'group_id',$groupId);
|
||||
}
|
||||
// Only the explicitly selected plans' unassigned users can change in initialize mode.
|
||||
foreach (User::whereIn('plan_id',$request['plan_ids'])->whereNull('group_id')->orderBy('id')->lockForUpdate()->get() as $user) {
|
||||
$ops[] = self::operation('user',$user,'group_id',$groupId);
|
||||
}
|
||||
if (isset($request['group-name'])) $ops[] = self::operation('group',$group,'name',$request['group-name']);
|
||||
$setting = self::object('setting','subscribe_url');
|
||||
$ops[] = self::operation('setting',$setting,'value',$request['subscribe-url'],'subscribe_url');
|
||||
}
|
||||
if ($apply) self::verifyEvidence($request,$m,$node);
|
||||
$j = ['version'=>1,'state'=>'prepared','manifest'=>$m,'request'=>$request,
|
||||
'operations'=>array_values(array_filter($ops)),
|
||||
'node_guard'=>['enabled'=>(bool)$node->enabled,'show'=>(bool)$node->show]];
|
||||
if (!$apply) return ['state'=>'preview','journal'=>$j];
|
||||
writeJson($path,$j); // Durable intent before any DB write; recover prepared+after on retry.
|
||||
foreach ($j['operations'] as $op) self::applyOperation($op);
|
||||
return ['state'=>'applied','journal'=>$j];
|
||||
},1);
|
||||
if ($apply) {
|
||||
self::flushSettings($result['journal']['operations']);
|
||||
self::notifyCommitted($m);
|
||||
$result['journal']['state']='applied'; writeJson($path,$result['journal']);
|
||||
}
|
||||
return $result;
|
||||
}
|
||||
|
||||
public static function rollback(string $dir, bool $apply): array
|
||||
{
|
||||
$m = manifest($dir); $j = readJson($dir.'/promotion-journal.json'); self::checkJournal($j,$m);
|
||||
$rollbackPath=$dir.'/rollback-journal.json';
|
||||
$result=DB::transaction(function () use ($m,$j,$apply,$rollbackPath) {
|
||||
[$node,$qa]=self::context($m);
|
||||
if (!$qa->banned) throw new RuntimeException('QA must remain disabled during rollback');
|
||||
if (is_file($rollbackPath)) {
|
||||
$r=readJson($rollbackPath);
|
||||
if (($r['promotion']??null)!==$j) throw new RuntimeException('Rollback journal mismatch');
|
||||
} else {
|
||||
// Rollback itself has a durable before/after journal, including node containment.
|
||||
$undo=[];
|
||||
foreach (array_reverse($j['operations']) as $op) {
|
||||
if ($op['kind']==='qa') continue; // Never reactivate test users.
|
||||
[$model,$exists,$value]=self::current($op);
|
||||
if (!$exists || $value!==$op['after']) throw new RuntimeException('Rollback conflict: current value differs from recorded after');
|
||||
$undo[]=['kind'=>$op['kind'],'id'=>$op['id'],'field'=>$op['field'],
|
||||
'before_exists'=>true,'before'=>$op['after'],'after'=>$op['before'],
|
||||
'delete'=>!$op['before_exists']];
|
||||
}
|
||||
foreach (['enabled','show'] as $field) {
|
||||
if ((bool)$node->$field!==$j['node_guard'][$field]) throw new RuntimeException('Node containment field changed after promotion');
|
||||
if ($node->$field) $undo[]=['kind'=>'node','id'=>$m['node_id'],'field'=>$field,
|
||||
'before_exists'=>true,'before'=>true,'after'=>false,'delete'=>false];
|
||||
}
|
||||
if (!$qa->banned) throw new RuntimeException('QA reactivated after promotion; resolve conflict before rollback');
|
||||
$r=['version'=>1,'state'=>'prepared','promotion'=>$j,'operations'=>$undo];
|
||||
}
|
||||
$before=$after=true;
|
||||
foreach ($r['operations'] as $op) {
|
||||
// Only rollback containment adds these two node fields.
|
||||
if (!in_array($op['field'],self::FIELDS[$op['kind']]??[],true) && !($op['kind']==='node' && $op['id']===$m['node_id'] && in_array($op['field'],['enabled','show'],true))) {
|
||||
throw new RuntimeException('Invalid rollback field');
|
||||
}
|
||||
[, $exists,$value]=self::current($op);
|
||||
$before=$before && $exists && $value===$op['before'];
|
||||
$after=$after && ($op['delete'] ? !$exists : ($exists && $value===$op['after']));
|
||||
}
|
||||
if (!$after && (!$before || $r['state']!=='prepared')) throw new RuntimeException('Rollback conflicts; no objects restored');
|
||||
if (!$apply) return ['state'=>$after?'already_rolled_back':'preview','journal'=>$r];
|
||||
if (!is_file($rollbackPath)) writeJson($rollbackPath,$r);
|
||||
if (!$after) foreach ($r['operations'] as $op) {
|
||||
[$model]=self::current($op);
|
||||
if ($op['delete']) $model->delete();
|
||||
else { $model->setAttribute($op['field'],$op['after']); $model->save(); }
|
||||
}
|
||||
return ['state'=>'rolled_back','journal'=>$r];
|
||||
},1);
|
||||
if ($apply) {
|
||||
self::flushSettings($j['operations']);
|
||||
self::notifyCommitted($m);
|
||||
$result['journal']['state']='rolled_back';writeJson($rollbackPath,$result['journal']);
|
||||
}
|
||||
return $result;
|
||||
}
|
||||
}
|
||||
@ -1,43 +1,25 @@
|
||||
<?php
|
||||
// Run only after real relay, subscription, certificate and traffic checks pass.
|
||||
require getcwd() . '/vendor/autoload.php';
|
||||
$app = require getcwd() . '/bootstrap/app.php';
|
||||
$app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
|
||||
use App\Models\Plan;
|
||||
use App\Models\Server;
|
||||
use App\Models\ServerGroup;
|
||||
use App\Models\User;
|
||||
use Illuminate\Support\Facades\DB;
|
||||
$opts = getopt('', ['backup:', 'plans:', 'group:']);
|
||||
$dir = $opts['backup'] ?? throw new RuntimeException('Missing backup');
|
||||
$ids = array_map('intval', explode(',', $opts['plans'] ?? ''));
|
||||
if (!$ids || in_array(0, $ids, true)) throw new RuntimeException('Explicit plan IDs required');
|
||||
$m = json_decode(file_get_contents("$dir/manifest.json"), true, flags: JSON_THROW_ON_ERROR);
|
||||
$groupId = (int)($opts['group'] ?? $m['group_id']);
|
||||
$node = Server::findOrFail($m['node_id']);
|
||||
$qa = User::findOrFail($m['test_user_id']);
|
||||
if (!$node->is_online || $qa->u + $qa->d <= 0) throw new RuntimeException('Online and traffic evidence missing');
|
||||
if (is_file("$dir/promoted.json")) { echo "Already promoted; verify current state separately\n"; exit; }
|
||||
$plans = Plan::whereIn('id', $ids)->get();
|
||||
if ($plans->count() !== count($ids)) throw new RuntimeException('Plan list mismatch');
|
||||
foreach ($plans as $p) if ($p->group_id !== null && $p->group_id !== $groupId) throw new RuntimeException('Plan already assigned elsewhere');
|
||||
$users = User::whereIn('plan_id', $ids)->whereNull('group_id')->get(['id','group_id','updated_at']);
|
||||
umask(0077);
|
||||
file_put_contents("$dir/before-promotion.json", json_encode([
|
||||
'plans' => $plans->map->getAttributes(), 'user_groups' => $users,
|
||||
'node_groups' => $node->group_ids, 'group' => ServerGroup::findOrFail($groupId)->getAttributes(),
|
||||
'subscribe_url' => admin_setting('subscribe_url'), 'qa_banned' => $qa->banned,
|
||||
], JSON_THROW_ON_ERROR));
|
||||
DB::transaction(function () use ($ids, $users, $node, $qa, $groupId) {
|
||||
$group = ServerGroup::findOrFail($groupId);
|
||||
if (str_starts_with($group->name, 'dpl 接入验收 ')) { $group->name = '基础节点'; $group->save(); }
|
||||
Plan::whereIn('id', $ids)->whereNull('group_id')->update(['group_id' => $groupId]);
|
||||
User::whereIn('id', $users->pluck('id'))->whereNull('group_id')->update(['group_id' => $groupId]);
|
||||
$node->group_ids = [(string)$groupId]; $node->save();
|
||||
$qa->banned = 1; $qa->saveQuietly();
|
||||
admin_setting(['subscribe_url' => 'https://c2.deeplink-subscribe.com']);
|
||||
});
|
||||
$result = ['node_id' => $node->id, 'group_id' => $groupId, 'plan_ids' => $ids,
|
||||
'existing_users_assigned' => $users->count(), 'qa_disabled' => true];
|
||||
file_put_contents("$dir/promoted.json", json_encode($result, JSON_THROW_ON_ERROR));
|
||||
echo json_encode($result) . PHP_EOL;
|
||||
// Default is preview. Apply only in a separately authorized target environment.
|
||||
require __DIR__ . '/operations.php';
|
||||
use DplOnboarding\PanelChange;
|
||||
$opts = getopt('', ['backup:', 'mode:', 'plans:', 'group:', 'subscribe-url:', 'group-name:', 'verification:', 'apply', 'help']);
|
||||
if (isset($opts['help'])) {
|
||||
echo "--backup=DIR --mode=append --group=ID [--verification=QA_REPORT --apply]\n";
|
||||
echo "--backup=DIR --mode=initialize --plans=IDS --subscribe-url=HTTPS_URL [--group=ID] [--group-name=NAME] [--verification=QA_REPORT --apply]\n";
|
||||
exit;
|
||||
}
|
||||
try {
|
||||
$dir = $opts['backup'] ?? throw new RuntimeException('Missing --backup');
|
||||
$apply = array_key_exists('apply', $opts);
|
||||
unset($opts['backup'], $opts['apply']);
|
||||
if (!isset($opts['mode']) || !in_array($opts['mode'], ['append','initialize'], true)) throw new RuntimeException('Explicit --mode required');
|
||||
if (isset($opts['group'])) { DplOnboarding\ids($opts['group']); if (str_contains($opts['group'],',')) throw new RuntimeException('One group required'); }
|
||||
DplOnboarding\boot();
|
||||
$result = DplOnboarding\locked($dir, fn() => PanelChange::promote($dir, $opts, $apply));
|
||||
// Journal is field-scoped and contains no passwords/UUIDs/tokens.
|
||||
echo json_encode($result, JSON_THROW_ON_ERROR | JSON_UNESCAPED_UNICODE) . PHP_EOL;
|
||||
} catch (Throwable $e) {
|
||||
$reason=get_class($e)===RuntimeException::class ? $e->getMessage() : "Inspect protected journal/current state";
|
||||
fwrite(STDERR, "Promotion stopped: " . $reason . "\n");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
@ -1,16 +1,32 @@
|
||||
<?php
|
||||
require getcwd() . '/vendor/autoload.php';
|
||||
$app = require getcwd() . '/bootstrap/app.php';
|
||||
$app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
|
||||
$opts = getopt('', ['backup:']);
|
||||
$dir = $opts['backup'] ?? throw new RuntimeException('Missing backup');
|
||||
$m = json_decode(file_get_contents("$dir/manifest.json"), true, flags: JSON_THROW_ON_ERROR);
|
||||
$node = App\Models\Server::findOrFail($m['node_id']);
|
||||
$qa = App\Models\User::findOrFail($m['test_user_id']);
|
||||
echo json_encode([
|
||||
'node_id' => $node->id, 'name' => $node->name, 'is_online' => $node->is_online,
|
||||
'last_check_at' => $node->last_check_at, 'last_push_at' => $node->last_push_at,
|
||||
'qa_upload' => $qa->u, 'qa_download' => $qa->d, 'qa_banned' => $qa->banned,
|
||||
'plans' => App\Models\Plan::orderBy('id')->get(['id', 'group_id']),
|
||||
'existing_users_with_selected_plan_and_null_group' => App\Models\User::whereIn('plan_id', [1,6,7,8])->whereNull('group_id')->count(),
|
||||
], JSON_UNESCAPED_UNICODE) . PHP_EOL;
|
||||
require __DIR__ . '/operations.php';
|
||||
$opts=getopt('', ['backup:', 'plans:', 'group:', 'help']);
|
||||
if (isset($opts['help'])) { echo "--backup=DIR [--plans=ID,ID] [--group=ID]; no implicit plan IDs\n"; exit; }
|
||||
try {
|
||||
$dir=$opts['backup'] ?? throw new RuntimeException('Missing --backup');
|
||||
$ids=isset($opts['plans']) ? DplOnboarding\ids($opts['plans']) : [];
|
||||
$group=null;
|
||||
if (isset($opts['group'])) {
|
||||
$groups=DplOnboarding\ids($opts['group']);
|
||||
if (count($groups)!==1) throw new RuntimeException('One group required');
|
||||
$group=$groups[0];
|
||||
}
|
||||
$m=DplOnboarding\manifest($dir);
|
||||
DplOnboarding\boot();
|
||||
$node=App\Models\Server::findOrFail($m['node_id']);
|
||||
$qa=App\Models\User::findOrFail($m['test_user_id']);
|
||||
if ((int)$node->machine_id!==$m['machine_id']) throw new RuntimeException('Manifest machine mismatch');
|
||||
$plans=App\Models\Plan::whereIn('id',$ids)->orderBy('id')->get(['id','group_id']);
|
||||
if ($plans->count()!==count($ids)) throw new RuntimeException('Some selected plans missing');
|
||||
if ($group!==null) App\Models\ServerGroup::findOrFail($group);
|
||||
$result=[
|
||||
'node_id'=>$node->id, 'name'=>$node->name, 'is_online'=>(bool)$node->is_online,
|
||||
'last_check_at'=>$node->last_check_at, 'last_push_at'=>$node->last_push_at,
|
||||
'group_ids'=>$node->group_ids, 'qa_upload'=>(int)$qa->u, 'qa_download'=>(int)$qa->d,
|
||||
'qa_banned'=>(bool)$qa->banned, 'selected_plan_ids'=>$ids, 'plans'=>$plans->toArray(),
|
||||
'existing_users_with_selected_plan_and_null_group'=>$ids ? App\Models\User::whereIn('plan_id',$ids)->whereNull('group_id')->count() : null,
|
||||
'selected_plans_in_other_group'=>$group===null ? null : $plans->filter(fn($p) => $p->group_id!==null && (int)$p->group_id!==$group)->pluck('id')->all(),
|
||||
'scope'=>'Read-only snapshot; not subscription/traffic/GUI acceptance',
|
||||
];
|
||||
echo json_encode($result,JSON_THROW_ON_ERROR|JSON_UNESCAPED_UNICODE).PHP_EOL;
|
||||
} catch (Throwable $e) { fwrite(STDERR,"Readiness failed; verify parameters and manifest.\n"); exit(1); }
|
||||
|
||||
16
scripts/node-onboarding/rollback.php
Normal file
16
scripts/node-onboarding/rollback.php
Normal file
@ -0,0 +1,16 @@
|
||||
<?php
|
||||
// Field-scoped compare-before-restore. Preview unless explicitly --apply.
|
||||
require __DIR__ . '/operations.php';
|
||||
use DplOnboarding\PanelChange;
|
||||
$opts=getopt('', ['backup:', 'apply', 'help']);
|
||||
if (isset($opts['help'])) { echo "--backup=DIR [--apply]; restores new-format promotion journal only\n"; exit; }
|
||||
try {
|
||||
$dir=$opts['backup'] ?? throw new RuntimeException('Missing --backup');
|
||||
DplOnboarding\boot();
|
||||
$result=DplOnboarding\locked($dir, fn() => PanelChange::rollback($dir, array_key_exists('apply',$opts)));
|
||||
echo json_encode($result, JSON_THROW_ON_ERROR | JSON_UNESCAPED_UNICODE) . PHP_EOL;
|
||||
} catch (Throwable $e) {
|
||||
$reason=get_class($e)===RuntimeException::class ? $e->getMessage() : "Inspect journal/conflicts";
|
||||
fwrite(STDERR, "Rollback stopped: " . $reason . "\n");
|
||||
exit(1);
|
||||
}
|
||||
@ -1,82 +1,109 @@
|
||||
<?php
|
||||
// Run from the production application root. All credential output is private.
|
||||
require getcwd() . '/vendor/autoload.php';
|
||||
$app = require getcwd() . '/bootstrap/app.php';
|
||||
$app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
|
||||
|
||||
use App\Models\Server;
|
||||
use App\Models\ServerGroup;
|
||||
use App\Models\ServerMachine;
|
||||
use App\Models\User;
|
||||
// Run only in an authorized target environment. --help never boots the app.
|
||||
require __DIR__ . '/operations.php';
|
||||
use App\Models\{Server,ServerGroup,ServerMachine,User};
|
||||
use Illuminate\Support\Facades\DB;
|
||||
use Illuminate\Support\Str;
|
||||
|
||||
$opts = getopt('', ['key:', 'name:', 'relay-host:', 'relay-port:', 'listen-port:', 'tls-domain:', 'backup:']);
|
||||
foreach (['key', 'name', 'relay-host', 'relay-port', 'listen-port', 'tls-domain', 'backup'] as $key) {
|
||||
if (empty($opts[$key])) throw new RuntimeException("Missing --$key");
|
||||
}
|
||||
if (!preg_match('/^[a-z0-9-]+$/', $opts['key'])) throw new RuntimeException('Invalid key');
|
||||
foreach (['relay-port', 'listen-port'] as $key) {
|
||||
if (!ctype_digit((string)$opts[$key]) || $opts[$key] < 1024 || $opts[$key] > 65535) throw new RuntimeException('Invalid port');
|
||||
}
|
||||
$dir = rtrim($opts['backup'], '/');
|
||||
umask(0077);
|
||||
if (!is_dir($dir) && !mkdir($dir, 0700, true)) throw new RuntimeException('Backup creation failed');
|
||||
if (is_file("$dir/manifest.json")) {
|
||||
$manifest = json_decode(file_get_contents("$dir/manifest.json"), true, flags: JSON_THROW_ON_ERROR);
|
||||
if ($manifest['key'] !== $opts['key']) throw new RuntimeException('Backup belongs to another node');
|
||||
echo json_encode(['already_staged' => true, 'node_id' => $manifest['node_id']]) . PHP_EOL;
|
||||
$opts=getopt('', ['key:', 'name:', 'relay-host:', 'relay-port:', 'listen-port:', 'tls-domain:', 'panel-url:', 'backup:', 'help']);
|
||||
if (isset($opts['help'])) {
|
||||
echo "--key=KEY --name=NAME --relay-host=HOST --relay-port=PORT --listen-port=PORT --tls-domain=DOMAIN --panel-url=HTTPS_URL --backup=DIR\n";
|
||||
exit;
|
||||
}
|
||||
if (ServerMachine::where('name', 'dpl-' . $opts['key'])->exists() || Server::where('name', $opts['name'])->exists()) {
|
||||
throw new RuntimeException('Existing resources found without this backup manifest');
|
||||
try {
|
||||
foreach (['key','name','relay-host','relay-port','listen-port','tls-domain','panel-url','backup'] as $key) {
|
||||
if (!isset($opts[$key]) || !is_string($opts[$key]) || $opts[$key]==='') throw new RuntimeException('Missing required stage option');
|
||||
}
|
||||
if (!preg_match('/^[a-z0-9-]+$/D',$opts['key'])) throw new RuntimeException('Invalid deployment key');
|
||||
foreach (['relay-port','listen-port'] as $key) {
|
||||
if (!ctype_digit($opts[$key]) || (int)$opts[$key]<1024 || (int)$opts[$key]>65535) throw new RuntimeException('Invalid port');
|
||||
$opts[$key]=(int)$opts[$key];
|
||||
}
|
||||
foreach (['relay-host','tls-domain'] as $key) {
|
||||
if (!preg_match('/^[a-z0-9][a-z0-9.-]*\.[a-z]{2,}$/Di',$opts[$key])) throw new RuntimeException('Explicit hostname required');
|
||||
}
|
||||
$url=parse_url($opts['panel-url']);
|
||||
if (!is_array($url) || ($url['scheme']??'')!=='https' || empty($url['host']) || isset($url['user']) || isset($url['pass']) || isset($url['query']) || isset($url['fragment'])) {
|
||||
throw new RuntimeException('Explicit HTTPS panel URL without credentials/query required');
|
||||
}
|
||||
$dir=rtrim($opts['backup'],'/');
|
||||
if (!str_starts_with($dir,'/') || is_link($dir)) throw new RuntimeException('Absolute private backup directory required');
|
||||
umask(0077);
|
||||
if (!is_dir($dir) && !mkdir($dir,0700,true)) throw new RuntimeException('Cannot create backup directory');
|
||||
if ((fileperms($dir)&0077)!==0) throw new RuntimeException('Backup directory must be private');
|
||||
$parameters=$opts;unset($parameters['backup']);ksort($parameters);
|
||||
DplOnboarding\boot();
|
||||
$m=DplOnboarding\locked($dir,function () use ($dir,$parameters) {
|
||||
$path=$dir.'/manifest.json';
|
||||
if (is_file($path)) {
|
||||
$m=DplOnboarding\manifest($dir);
|
||||
if (($m['parameters']??null)!==$parameters) throw new RuntimeException('Existing manifest has different/legacy parameters; no automatic adoption');
|
||||
$node=Server::findOrFail($m['node_id']);$qa=User::findOrFail($m['test_user_id']);
|
||||
$machine=ServerMachine::findOrFail($m['machine_id']);
|
||||
if ((int)$node->machine_id!==$machine->id || $machine->name!=='dpl-'.$parameters['key'] ||
|
||||
$node->host!==$parameters['relay-host'] || (int)$node->port!==$parameters['relay-port'] ||
|
||||
(int)$node->server_port!==$parameters['listen-port'] || $node->type!=='anytls' ||
|
||||
($node->protocol_settings['tls']['server_name']??null)!==$parameters['tls-domain'] ||
|
||||
(int)$qa->group_id!==$m['group_id']) throw new RuntimeException('Staged resources drifted');
|
||||
if (is_file($dir.'/promotion-journal.json') || is_file($dir.'/promoted.json')) throw new RuntimeException('Node already promoted; staging cannot regenerate credentials');
|
||||
return $m;
|
||||
}
|
||||
return DB::transaction(function () use ($parameters,$path,$dir) {
|
||||
if (ServerMachine::where('name','dpl-'.$parameters['key'])->exists() || Server::where('name',$parameters['name'])->exists() ||
|
||||
Server::where('host',$parameters['relay-host'])->where('port',(string)$parameters['relay-port'])->exists() ||
|
||||
ServerGroup::where('name','dpl 接入验收 '.$parameters['key'])->exists()) {
|
||||
throw new RuntimeException('Existing resource/relay tuple without matching manifest; refusing takeover');
|
||||
}
|
||||
DplOnboarding\writeJson($dir.'/before.json',[
|
||||
'server_ids'=>Server::orderBy('id')->pluck('id')->all(),
|
||||
'machine_ids'=>ServerMachine::orderBy('id')->pluck('id')->all(),
|
||||
'group_ids'=>ServerGroup::orderBy('id')->pluck('id')->all(),
|
||||
]);
|
||||
$group=new ServerGroup();$group->name='dpl 接入验收 '.$parameters['key'];$group->save();
|
||||
$machine=ServerMachine::create(['name'=>'dpl-'.$parameters['key'],
|
||||
'notes'=>'Independent AnyTLS service; existing XrayR preserved','is_active'=>true,'token'=>ServerMachine::generateToken()]);
|
||||
$node=Server::create([
|
||||
'name'=>$parameters['name'],'type'=>'anytls','host'=>$parameters['relay-host'],
|
||||
'port'=>(string)$parameters['relay-port'],'server_port'=>(string)$parameters['listen-port'],
|
||||
'group_ids'=>[(string)$group->id],'route_ids'=>[],'tags'=>[],
|
||||
'rate'=>1,'show'=>true,'enabled'=>true,'machine_id'=>$machine->id,'sort'=>1,
|
||||
'protocol_settings'=>['tls'=>['server_name'=>$parameters['tls-domain'],'allow_insecure'=>false]],
|
||||
'cert_config'=>['cert_mode'=>'file','domain'=>$parameters['tls-domain'],
|
||||
'cert_file'=>'/etc/letsencrypt/live/'.$parameters['tls-domain'].'/fullchain.pem',
|
||||
'key_file'=>'/etc/letsencrypt/live/'.$parameters['tls-domain'].'/privkey.pem'],
|
||||
]);
|
||||
$qa=new User([
|
||||
'email'=>'node-qa-'.$parameters['key'].'-'.Str::lower(Str::random(8)).'@example.invalid',
|
||||
'password'=>password_hash(Str::random(40),PASSWORD_BCRYPT),'uuid'=>(string)Str::uuid(),'token'=>Str::random(32),
|
||||
'group_id'=>$group->id,'transfer_enable'=>104857600,'expired_at'=>time()+86400,'banned'=>0,'is_admin'=>0,
|
||||
'remind_expire'=>0,'remind_traffic'=>0,'device_limit'=>2,
|
||||
]);$qa->saveQuietly();
|
||||
$m=['key'=>$parameters['key'],'node_id'=>$node->id,'machine_id'=>$machine->id,'group_id'=>$group->id,
|
||||
'test_user_id'=>$qa->id,'parameters'=>$parameters];
|
||||
// Save identities before commit. A failed/unknown commit must be inspected,
|
||||
// never repeated into new resources merely because a response was lost.
|
||||
DplOnboarding\writeJson($path,$m);
|
||||
return $m;
|
||||
},1);
|
||||
});
|
||||
// Recovery can regenerate missing transport files from the same staged objects.
|
||||
$machine=ServerMachine::findOrFail($m['machine_id']);$qa=User::findOrFail($m['test_user_id']);
|
||||
$panel=json_encode($parameters['panel-url'],JSON_THROW_ON_ERROR|JSON_UNESCAPED_SLASHES);
|
||||
$config="panel:\n url: $panel\nmachine:\n machine_id: {$machine->id}\n token_env: DPL_MACHINE_TOKEN\nkernel:\n type: singbox\n config_dir: /var/lib/dpl-node/kernel\nlog:\n level: info\nruntime:\n gomemlimit: 384MiB\n gogc: 75\nnode:\n push_interval: 30\n pull_interval: 30\n";
|
||||
foreach (['node-config.yml'=>$config,'node-secret.env'=>'DPL_MACHINE_TOKEN='.$machine->token."\n"] as $file=>$body) {
|
||||
$path=$dir.'/'.$file;
|
||||
if (is_link($path)) throw new RuntimeException('Refusing private transport symlink');
|
||||
$tmp=tempnam($dir,'.private-');if ($tmp===false) throw new RuntimeException('Private file failed');
|
||||
chmod($tmp,0600);
|
||||
if (file_put_contents($tmp,$body)!==strlen($body) || !rename($tmp,$path)) {
|
||||
if (is_file($tmp)) unlink($tmp);
|
||||
throw new RuntimeException('Private file failed');
|
||||
}
|
||||
}
|
||||
DplOnboarding\writeJson($dir.'/qa-private.json',['user_id'=>$qa->id,'uuid'=>$qa->uuid,'subscribe_token'=>$qa->token,
|
||||
'host'=>$parameters['relay-host'],'port'=>$parameters['relay-port'],'sni'=>$parameters['tls-domain']]);
|
||||
echo json_encode(['key'=>$m['key'],'node_id'=>$m['node_id'],'machine_id'=>$m['machine_id'],
|
||||
'group_id'=>$m['group_id'],'test_user_id'=>$m['test_user_id']],JSON_THROW_ON_ERROR).PHP_EOL;
|
||||
} catch (Throwable $e) {
|
||||
$reason=get_class($e)===RuntimeException::class ? $e->getMessage() : 'Inspect manifest/DB state before retrying';
|
||||
fwrite(STDERR,'Staging stopped: '.$reason.PHP_EOL);exit(1);
|
||||
}
|
||||
file_put_contents("$dir/before.json", json_encode([
|
||||
'servers' => DB::table('v2_server')->get(),
|
||||
'groups' => DB::table('v2_server_group')->get(),
|
||||
'machines' => DB::table('v2_server_machine')->get(),
|
||||
], JSON_THROW_ON_ERROR));
|
||||
|
||||
$manifest = DB::transaction(function () use ($opts) {
|
||||
$group = new ServerGroup();
|
||||
$group->name = 'dpl 接入验收 ' . $opts['key'];
|
||||
$group->save();
|
||||
$machine = ServerMachine::create([
|
||||
'name' => 'dpl-' . $opts['key'], 'notes' => 'Independent AnyTLS service; existing XrayR preserved',
|
||||
'is_active' => true, 'token' => ServerMachine::generateToken(),
|
||||
]);
|
||||
$node = Server::create([
|
||||
'name' => $opts['name'], 'type' => 'anytls', 'host' => $opts['relay-host'],
|
||||
'port' => (string)$opts['relay-port'], 'server_port' => (string)$opts['listen-port'],
|
||||
'group_ids' => [(string)$group->id], 'route_ids' => [], 'tags' => [],
|
||||
'rate' => 1, 'show' => true, 'enabled' => true, 'machine_id' => $machine->id, 'sort' => 1,
|
||||
'protocol_settings' => ['tls' => ['server_name' => $opts['tls-domain'], 'allow_insecure' => false]],
|
||||
'cert_config' => [
|
||||
'cert_mode' => 'file', 'domain' => $opts['tls-domain'],
|
||||
'cert_file' => '/etc/letsencrypt/live/' . $opts['tls-domain'] . '/fullchain.pem',
|
||||
'key_file' => '/etc/letsencrypt/live/' . $opts['tls-domain'] . '/privkey.pem',
|
||||
],
|
||||
]);
|
||||
$user = new User([
|
||||
'email' => 'node-qa-' . $opts['key'] . '-' . Str::lower(Str::random(8)) . '@example.invalid',
|
||||
'password' => password_hash(Str::random(40), PASSWORD_BCRYPT),
|
||||
'uuid' => (string)Str::uuid(), 'token' => Str::random(32),
|
||||
'group_id' => $group->id, 'transfer_enable' => 104857600,
|
||||
'expired_at' => time() + 86400, 'banned' => 0, 'is_admin' => 0,
|
||||
'remind_expire' => 0, 'remind_traffic' => 0, 'device_limit' => 2,
|
||||
]);
|
||||
$user->saveQuietly();
|
||||
return ['key' => $opts['key'], 'node_id' => $node->id, 'machine_id' => $machine->id,
|
||||
'group_id' => $group->id, 'test_user_id' => $user->id];
|
||||
});
|
||||
file_put_contents("$dir/manifest.json", json_encode($manifest, JSON_THROW_ON_ERROR));
|
||||
$machine = ServerMachine::findOrFail($manifest['machine_id']);
|
||||
$user = User::findOrFail($manifest['test_user_id']);
|
||||
$config = "panel:\n url: \"https://dp-link.net\"\nmachine:\n machine_id: {$machine->id}\n token_env: DPL_MACHINE_TOKEN\nkernel:\n type: singbox\n config_dir: /var/lib/dpl-node/kernel\nlog:\n level: info\nruntime:\n gomemlimit: 384MiB\n gogc: 75\nnode:\n push_interval: 30\n pull_interval: 30\n";
|
||||
file_put_contents("$dir/node-config.yml", $config);
|
||||
file_put_contents("$dir/node-secret.env", 'DPL_MACHINE_TOKEN=' . $machine->token . "\n");
|
||||
file_put_contents("$dir/qa-private.json", json_encode([
|
||||
'user_id' => $user->id, 'uuid' => $user->uuid, 'subscribe_token' => $user->token,
|
||||
'host' => $opts['relay-host'], 'port' => (int)$opts['relay-port'], 'sni' => $opts['tls-domain'],
|
||||
], JSON_THROW_ON_ERROR));
|
||||
echo json_encode($manifest, JSON_UNESCAPED_UNICODE) . PHP_EOL;
|
||||
|
||||
@ -1,72 +1,240 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run an isolated SOCKS client with strict TLS; never print QA credentials."""
|
||||
"""Validate private subscriptions; network execution requires explicit --run.
|
||||
|
||||
For a formal multi-node subscription use a private --expectations JSON array:
|
||||
[{"tag": "exact outbound tag", "host": "relay.example", "port": 12345,
|
||||
"sni": "node.example", "uuid": "private password", "expected_exit": "192.0.2.1",
|
||||
"expected_country": "JP"}]. Never print credentials or the subscription.
|
||||
"""
|
||||
import argparse
|
||||
import copy
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--qa', required=True)
|
||||
p.add_argument('--binary', required=True)
|
||||
p.add_argument('--expected-exit', required=True)
|
||||
p.add_argument('--port', type=int, default=19080)
|
||||
p.add_argument('--subscription')
|
||||
a = p.parse_args()
|
||||
q = json.loads(pathlib.Path(a.qa).read_text())
|
||||
c = {'log': {'level': 'error'}, 'dns': {'servers': [{'type': 'local', 'tag': 'local'}]},
|
||||
'inbounds': [{'type': 'mixed', 'listen': '127.0.0.1', 'listen_port': a.port}],
|
||||
'outbounds': [{'type': 'anytls', 'tag': 'qa', 'server': q['host'], 'server_port': q['port'],
|
||||
'password': q['uuid'], 'tls': {'enabled': True, 'server_name': q['sni'], 'insecure': False}}],
|
||||
'route': {'default_domain_resolver': 'local', 'final': 'qa'}}
|
||||
if a.subscription:
|
||||
c = json.loads(pathlib.Path(a.subscription).read_text())
|
||||
nodes = [x for x in c['outbounds'] if x.get('type') == 'anytls']
|
||||
assert len(nodes) == 1 and nodes[0]['tls'].get('insecure') is False
|
||||
assert nodes[0]['server'] == q['host'] and nodes[0]['server_port'] == q['port']
|
||||
assert nodes[0]['password'] == q['uuid']
|
||||
c['inbounds'] = [{'type': 'mixed', 'listen': '127.0.0.1', 'listen_port': a.port}]
|
||||
c['log'] = {'level': 'error'}
|
||||
os.umask(0o077)
|
||||
with tempfile.TemporaryDirectory(prefix='dpl-client-') as d:
|
||||
config = pathlib.Path(d) / 'client.json'
|
||||
config.write_text(json.dumps(c))
|
||||
env = dict(os.environ, GOMEMLIMIT='64MiB', GOGC='75')
|
||||
with (pathlib.Path(d) / 'client.log').open('wb') as log:
|
||||
proc = subprocess.Popen([a.binary, 'run', '-c', str(config)], cwd=d, env=env, stdout=log, stderr=log)
|
||||
|
||||
class VerificationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise VerificationError(message)
|
||||
|
||||
|
||||
def read_private(path):
|
||||
p = Path(path)
|
||||
require(p.is_file() and not p.is_symlink(), 'Private input must be a regular file')
|
||||
require(p.stat().st_mode & 0o077 == 0, 'Private input must have no group/other permissions')
|
||||
return json.loads(p.read_text())
|
||||
|
||||
|
||||
def validate(config, expectations, exact=True):
|
||||
require(isinstance(config, dict) and isinstance(config.get('outbounds'), list), 'Missing outbounds')
|
||||
all_tags = [o.get('tag') for o in config['outbounds'] if isinstance(o, dict)]
|
||||
require(len(all_tags) == len(config['outbounds']) and all(isinstance(t, str) and t for t in all_tags), 'Each outbound needs a tag')
|
||||
require(len(set(all_tags)) == len(all_tags), 'Duplicate outbound tags')
|
||||
nodes = {o['tag']: o for o in config['outbounds'] if o.get('type') == 'anytls'}
|
||||
require(isinstance(expectations, list) and expectations, 'Nonempty expectations required')
|
||||
selected = []
|
||||
for spec in expectations:
|
||||
require(isinstance(spec, dict), 'Invalid expectation')
|
||||
for key in ['tag', 'host', 'sni', 'uuid', 'expected_exit', 'expected_country']:
|
||||
require(isinstance(spec.get(key), str) and bool(spec[key]), 'Expectation missing required field')
|
||||
require(isinstance(spec.get('port'), int) and not isinstance(spec['port'], bool) and 1 <= spec['port'] <= 65535, 'Invalid relay port')
|
||||
try:
|
||||
for _ in range(50):
|
||||
assert proc.poll() is None, 'Client exited during startup'
|
||||
try:
|
||||
with socket.create_connection(('127.0.0.1', a.port), 0.1):
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(0.1)
|
||||
command = ['curl', '-q', '--fail', '--silent', '--show-error', '--max-time', '30',
|
||||
'--socks5-hostname', f'127.0.0.1:{a.port}']
|
||||
result = subprocess.run(command + ['https://api.ipify.org'], capture_output=True, text=True, check=True)
|
||||
ip = result.stdout.strip()
|
||||
assert ip == a.expected_exit, 'Unexpected exit IP'
|
||||
result = subprocess.run(command + ['https://www.cloudflare.com/cdn-cgi/trace'], capture_output=True, text=True, check=True)
|
||||
trace = dict(line.split('=', 1) for line in result.stdout.splitlines() if '=' in line)
|
||||
assert trace.get('ip') == a.expected_exit, 'Second exit check mismatch'
|
||||
print(json.dumps({'anytls': True, 'relay': q['host'], 'port': q['port'], 'strict_tls': True,
|
||||
'exit_ip': ip, 'country': trace.get('loc'), 'https_tls': trace.get('tls')}, ensure_ascii=False))
|
||||
ipaddress.ip_address(spec['expected_exit'])
|
||||
except ValueError as exc:
|
||||
raise VerificationError('Invalid expected exit IP') from exc
|
||||
require(len(spec['expected_country']) == 2 and spec['expected_country'].isalpha() and spec['expected_country'].isupper(), 'Expected country must be two uppercase letters')
|
||||
tag = spec['tag']
|
||||
require(tag in nodes and tag not in selected, 'Missing/duplicate selected node')
|
||||
node = nodes[tag]
|
||||
tls = node.get('tls') or {}
|
||||
require(tls.get('enabled') is True and tls.get('insecure') is False, 'Strict TLS must be explicit')
|
||||
require(node.get('server') == spec['host'] and node.get('server_port') == spec['port'], 'Relay endpoint mismatch')
|
||||
require(tls.get('server_name') == spec['sni'], 'SNI mismatch')
|
||||
require(node.get('password') == spec['uuid'], 'Subscription password mismatch')
|
||||
selected.append(tag)
|
||||
if exact:
|
||||
require(set(selected) == set(nodes), 'Expectations must cover every AnyTLS node exactly')
|
||||
return selected
|
||||
|
||||
|
||||
def probe_config(config, spec, port):
|
||||
result = copy.deepcopy(config)
|
||||
result['inbounds'] = [{'type': 'mixed', 'listen': '127.0.0.1', 'listen_port': port}]
|
||||
result['log'] = {'level': 'error'}
|
||||
# Preserve DNS, remote rulesets and all original outbounds. Only the two
|
||||
# diagnostic destinations are pinned; ordinary subscription routing remains.
|
||||
route = result.setdefault('route', {})
|
||||
rules = route.setdefault('rules', [])
|
||||
require(isinstance(rules, list), 'Invalid route rules')
|
||||
rules.insert(0, {'domain': ['api.ipify.org', 'www.cloudflare.com'],
|
||||
'action': 'route', 'outbound': spec['tag']})
|
||||
return result
|
||||
|
||||
|
||||
def stop(proc):
|
||||
if proc is None or proc.poll() is not None:
|
||||
return
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait(timeout=5)
|
||||
|
||||
|
||||
def run_node(config, spec, binary, port, diagnostic_dir):
|
||||
require(1024 <= port <= 65535, 'Invalid local SOCKS port')
|
||||
with socket.socket() as free:
|
||||
try:
|
||||
free.bind(('127.0.0.1', port))
|
||||
except OSError as exc:
|
||||
raise VerificationError('Local SOCKS port is occupied') from exc
|
||||
with tempfile.TemporaryDirectory(prefix='dpl-client-', dir=diagnostic_dir) as tmp:
|
||||
temporary = Path(tmp)
|
||||
path = temporary / 'client.json'
|
||||
path.write_text(json.dumps(probe_config(config, spec, port)))
|
||||
path.chmod(0o600)
|
||||
env = dict(os.environ, GOMEMLIMIT='64MiB', GOGC='75')
|
||||
proc = None
|
||||
log_path = temporary / 'client.log'
|
||||
try:
|
||||
with log_path.open('wb') as log:
|
||||
subprocess.run([binary, 'check', '-c', str(path)], env=env,
|
||||
cwd=tmp, stdout=log, stderr=log, check=True, timeout=30)
|
||||
proc = subprocess.Popen([binary, 'run', '-c', str(path)], env=env,
|
||||
cwd=tmp, stdout=log, stderr=log)
|
||||
ready = False
|
||||
for _ in range(100):
|
||||
require(proc.poll() is None, 'Client exited during startup')
|
||||
try:
|
||||
with socket.create_connection(('127.0.0.1', port), 0.1):
|
||||
ready = True
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(0.1)
|
||||
require(ready, 'Client startup timed out')
|
||||
command = ['curl', '-q', '--fail', '--silent', '--show-error', '--max-time', '30',
|
||||
'--socks5-hostname', f'127.0.0.1:{port}', '--noproxy', '']
|
||||
ip = subprocess.run(command + ['https://api.ipify.org'], capture_output=True,
|
||||
text=True, check=True, timeout=35).stdout.strip()
|
||||
require(ipaddress.ip_address(ip) == ipaddress.ip_address(spec['expected_exit']), 'Unexpected exit IP')
|
||||
response = subprocess.run(command + ['https://www.cloudflare.com/cdn-cgi/trace'],
|
||||
capture_output=True, text=True, check=True, timeout=35)
|
||||
trace = dict(line.split('=', 1) for line in response.stdout.splitlines() if '=' in line)
|
||||
require(trace.get('ip') == ip, 'Second exit check mismatch')
|
||||
require(trace.get('loc') == spec['expected_country'], 'Exit country mismatch')
|
||||
require(trace.get('tls', '').startswith('TLSv'), 'Target HTTPS TLS not observed')
|
||||
return {'tag': spec['tag'], 'strict_tls': True, 'exit_ip': ip,
|
||||
'country': trace['loc'], 'https_tls': trace['tls'], 'live': True,
|
||||
'relay': spec['host'], 'relay_port': spec['port'], 'sni': spec['sni']}
|
||||
except Exception:
|
||||
log.flush()
|
||||
failure = pathlib.Path(a.qa).parent / 'client-failure.log'
|
||||
shutil.copyfile(pathlib.Path(d) / 'client.log', failure)
|
||||
failure.chmod(0o600)
|
||||
print(f'Client failed; protected diagnostic log: {failure}')
|
||||
# Logs may contain private config context; retain only in private task directory.
|
||||
stop(proc)
|
||||
if log_path.exists():
|
||||
fd, name = tempfile.mkstemp(prefix='client-failure-', suffix='.log', dir=diagnostic_dir)
|
||||
os.close(fd)
|
||||
failure = Path(name)
|
||||
shutil.copyfile(log_path, failure)
|
||||
failure.chmod(0o600)
|
||||
raise
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
stop(proc)
|
||||
|
||||
|
||||
def write_report(path, report):
|
||||
destination = Path(path)
|
||||
require(destination.parent.is_dir() and destination.parent.stat().st_mode & 0o077 == 0,
|
||||
'Report directory must be private')
|
||||
require(not destination.is_symlink(), 'Refusing report symlink')
|
||||
with tempfile.NamedTemporaryFile(mode='w', dir=destination.parent, delete=False) as stream:
|
||||
temporary = Path(stream.name)
|
||||
try:
|
||||
os.chmod(temporary, 0o600)
|
||||
json.dump(report, stream, ensure_ascii=False)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, destination)
|
||||
finally:
|
||||
if temporary.exists():
|
||||
temporary.unlink()
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sources = parser.add_mutually_exclusive_group(required=True)
|
||||
sources.add_argument('--qa')
|
||||
sources.add_argument('--expectations')
|
||||
parser.add_argument('--subscription', required=True)
|
||||
parser.add_argument('--binary')
|
||||
parser.add_argument('--expected-exit')
|
||||
parser.add_argument('--expected-country')
|
||||
parser.add_argument('--target-tag')
|
||||
parser.add_argument('--report', help='Private JSON result file; required for --run')
|
||||
parser.add_argument('--port', type=int, default=19080)
|
||||
execution = parser.add_mutually_exclusive_group()
|
||||
execution.add_argument('--run', action='store_true')
|
||||
execution.add_argument('--validate-only', action='store_true')
|
||||
args = parser.parse_args()
|
||||
os.umask(0o077)
|
||||
try:
|
||||
require(not args.run or args.report, '--run requires a private --report destination')
|
||||
if args.report:
|
||||
destination = Path(args.report)
|
||||
inputs = [args.subscription, args.qa, args.expectations]
|
||||
require(all(destination.resolve() != Path(item).resolve() for item in inputs if item), 'Report must not overwrite a private input')
|
||||
require(destination.parent.is_dir() and destination.parent.stat().st_mode & 0o077 == 0 and not destination.is_symlink(), 'Private regular report destination required')
|
||||
if args.report:
|
||||
write_report(args.report, {'state': 'verification_pending', 'verified_at': None})
|
||||
config = read_private(args.subscription)
|
||||
qa_user_id = None
|
||||
source = args.expectations or args.qa
|
||||
if args.expectations:
|
||||
require(not any([args.expected_exit, args.expected_country, args.target_tag]), 'Put each node expectation in the private expectations file')
|
||||
specs = read_private(args.expectations)
|
||||
else:
|
||||
qa = read_private(args.qa)
|
||||
qa_user_id = qa.get('user_id')
|
||||
require(args.expected_exit and args.expected_country, 'QA requires explicit expected-exit and expected-country')
|
||||
candidates = [n for n in config.get('outbounds', []) if n.get('type') == 'anytls'
|
||||
and n.get('server') == qa.get('host') and n.get('server_port') == qa.get('port')
|
||||
and (not args.target_tag or n.get('tag') == args.target_tag)]
|
||||
require(len(candidates) == 1, 'QA target ambiguous/missing; supply exact target-tag')
|
||||
specs = [{'tag': candidates[0]['tag'], 'host': qa.get('host'), 'port': qa.get('port'),
|
||||
'sni': qa.get('sni'), 'uuid': qa.get('uuid'), 'expected_exit': args.expected_exit,
|
||||
'expected_country': args.expected_country}]
|
||||
tags = validate(config, specs, exact=bool(args.expectations))
|
||||
results = []
|
||||
if args.run:
|
||||
require(args.binary and Path(args.binary).is_file(), 'Explicit existing client binary required')
|
||||
binary = str(Path(args.binary).resolve())
|
||||
folder = Path(source).resolve().parent
|
||||
require(folder.stat().st_mode & 0o077 == 0, 'Live diagnostics directory must be private')
|
||||
for spec in specs:
|
||||
results.append(run_node(config, spec, binary, args.port, str(folder)))
|
||||
report = {'state': 'live_verified' if args.run else 'structure_validated',
|
||||
'verified_at': int(time.time()) if args.run else None,
|
||||
'qa_user_id': qa_user_id, 'selected_nodes': tags, 'results': results,
|
||||
'scope': 'Full subscription retained; only diagnostic destination routes pinned. No traffic/account/GUI acceptance implied.'}
|
||||
if args.report:
|
||||
write_report(args.report, report)
|
||||
print(json.dumps(report, ensure_ascii=False))
|
||||
return 0
|
||||
except (Exception, KeyboardInterrupt) as exc:
|
||||
# No traceback or subprocess stderr: either can contain credentials.
|
||||
message = str(exc) if isinstance(exc, VerificationError) else 'Verification failed; inspect protected diagnostics'
|
||||
print(json.dumps({'state': 'failed', 'reason': message}), file=__import__('sys').stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
|
||||
116
tests/node-onboarding/PanelChangeIsolatedTest.php
Normal file
116
tests/node-onboarding/PanelChangeIsolatedTest.php
Normal file
@ -0,0 +1,116 @@
|
||||
<?php
|
||||
/** Standalone: php vendor/bin/phpunit --no-configuration tests/node-onboarding/PanelChangeIsolatedTest.php
|
||||
* Real Eloquent + SQLite :memory:, no application bootstrap or dotenv.
|
||||
* NodeSync is a test double; no Redis, queue, HTTP or production connections.
|
||||
*/
|
||||
namespace App\Services {
|
||||
final class NodeSyncService {
|
||||
public static array $calls=[];
|
||||
public static bool $fail=false;
|
||||
public static function notifyFullSync(int $id): void {
|
||||
self::$calls[]=['node',$id];
|
||||
if (self::$fail) throw new \RuntimeException('Injected post-commit notification failure');
|
||||
}
|
||||
public static function notifyMachineNodesChanged(int $id): void { self::$calls[]=['machine',$id]; }
|
||||
}
|
||||
}
|
||||
namespace {
|
||||
require_once dirname(__DIR__,2).'/vendor/autoload.php';
|
||||
require_once dirname(__DIR__,2).'/scripts/node-onboarding/operations.php';
|
||||
|
||||
use App\Models\{Server,User};
|
||||
use App\Services\NodeSyncService;
|
||||
use DplOnboarding\PanelChange;
|
||||
use Illuminate\Container\Container;
|
||||
use Illuminate\Database\Capsule\Manager as Capsule;
|
||||
use Illuminate\Events\Dispatcher;
|
||||
use Illuminate\Support\Facades\Facade;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
final class PanelChangeIsolatedTest extends TestCase {
|
||||
private Capsule $db;
|
||||
private string $dir;
|
||||
private array $request;
|
||||
protected function setUp(): void {
|
||||
$app=new Container(); Container::setInstance($app); Facade::clearResolvedInstances(); Facade::setFacadeApplication($app);
|
||||
$app->instance('config',new \Illuminate\Config\Repository(['cache'=>['default'=>'array','stores'=>['array'=>['driver'=>'array']]],'app'=>['key'=>'test']]));
|
||||
$events=new Dispatcher($app); $app->instance('events',$events);
|
||||
$this->db=new Capsule($app); $this->db->addConnection(['driver'=>'sqlite','database'=>':memory:','foreign_key_constraints'=>true]);
|
||||
$this->db->setEventDispatcher($events); $this->db->setAsGlobal(); $this->db->bootEloquent();
|
||||
$app->instance('db',$this->db->getDatabaseManager());
|
||||
$app->singleton('cache',fn($app)=>new \Illuminate\Cache\CacheManager($app));
|
||||
$schema=$this->db->schema();
|
||||
$schema->create('v2_server',function($t){$t->increments('id');$t->string('name');$t->string('type');$t->integer('machine_id');$t->text('group_ids');$t->text('protocol_settings');$t->string('host');$t->string('port');$t->boolean('enabled');$t->boolean('show');$t->timestamps();});
|
||||
$schema->create('v2_user',function($t){$t->increments('id');$t->string('email');$t->boolean('is_admin')->default(false);$t->integer('group_id')->nullable();$t->integer('plan_id')->nullable();$t->integer('u')->default(0);$t->integer('d')->default(0);$t->boolean('banned')->default(false);$t->integer('created_at')->nullable();$t->integer('updated_at')->nullable();});
|
||||
$schema->create('v2_server_group',function($t){$t->increments('id');$t->string('name');$t->integer('created_at')->nullable();$t->integer('updated_at')->nullable();});
|
||||
$schema->create('v2_plan',function($t){$t->increments('id');$t->integer('group_id')->nullable();$t->string('name');$t->integer('price');$t->integer('created_at')->nullable();$t->integer('updated_at')->nullable();});
|
||||
$schema->create('v2_settings',function($t){$t->increments('id');$t->string('name')->unique();$t->text('value')->nullable();$t->timestamps();});
|
||||
Capsule::table('v2_server_group')->insert([['id'=>1,'name'=>'formal'],['id'=>2,'name'=>'qa']]);
|
||||
Capsule::table('v2_plan')->insert(['id'=>1,'name'=>'protected-plan','price'=>1234,'group_id'=>1]);
|
||||
Capsule::table('v2_user')->insert([['id'=>1,'email'=>'node-qa-test-key-a@example.invalid','group_id'=>2,'u'=>10,'d'=>20],['id'=>2,'email'=>'formal@example.invalid','group_id'=>1,'u'=>200,'d'=>300]]);
|
||||
Capsule::table('v2_settings')->insert(['name'=>'subscribe_url','value'=>'https://original.example.invalid']);
|
||||
foreach ([1,2] as $id) Server::create(['id'=>$id,'name'=>'node-'.$id,'type'=>'anytls','machine_id'=>$id,'group_ids'=>[(string)$id], 'host'=>'relay.example.invalid','port'=>(string)(47000+$id),'enabled'=>true,'show'=>true,'protocol_settings'=>['tls'=>['server_name'=>'node.example.invalid','allow_insecure'=>false]]]);
|
||||
// Guarded IDs auto-increment to 1/2, matching the manifest.
|
||||
\Illuminate\Support\Facades\Cache::put(\App\Utils\CacheKey::get('SERVER_ANYTLS_LAST_CHECK_AT',2),time());
|
||||
$this->dir=sys_get_temp_dir().'/dpl-isolated-'.bin2hex(random_bytes(8)); mkdir($this->dir,0700);
|
||||
\DplOnboarding\writeJson($this->dir.'/manifest.json',['key'=>'test-key','node_id'=>2,'machine_id'=>2,'group_id'=>2,'test_user_id'=>1]);
|
||||
$this->request=['mode'=>'append','group'=>'1','verification'=>$this->dir.'/report.json'];
|
||||
$this->evidence(); NodeSyncService::$calls=[]; NodeSyncService::$fail=false;
|
||||
}
|
||||
private function evidence(array $overrides=[]): void {
|
||||
\DplOnboarding\writeJson($this->dir.'/report.json',array_replace(['state'=>'live_verified','qa_user_id'=>1,'verified_at'=>time(),'results'=>[['live'=>true,'strict_tls'=>true,'relay'=>'relay.example.invalid','relay_port'=>47002,'sni'=>'node.example.invalid']]],$overrides));
|
||||
}
|
||||
protected function tearDown(): void { foreach(glob($this->dir.'/*') as $p) unlink($p);rmdir($this->dir);$this->db->getDatabaseManager()->disconnect();NodeSyncService::$fail=false; }
|
||||
private function snapshot(): array { $all=[];foreach(['v2_server','v2_user','v2_plan','v2_settings','v2_server_group'] as $t)$all[$t]=Capsule::table($t)->orderBy('id')->get()->toJson();return $all; }
|
||||
private function protectedSnapshot(): array { return [Capsule::table('v2_server')->where('id',1)->first(),Capsule::table('v2_user')->where('id',2)->first(),Capsule::table('v2_plan')->get()->toJson(),Capsule::table('v2_settings')->get()->toJson(),Capsule::table('v2_server_group')->get()->toJson()]; }
|
||||
private function rejects(callable $f): void { $rejected=false;try{$f();}catch(\RuntimeException $e){$rejected=true;}self::assertTrue($rejected,'Expected safe refusal'); }
|
||||
public function testPreviewAppendIdempotenceAndRollback(): void {
|
||||
$before=$this->snapshot();$protected=$this->protectedSnapshot();
|
||||
self::assertSame('preview',PanelChange::promote($this->dir,$this->request,false)['state']);self::assertSame($before,$this->snapshot());self::assertFileDoesNotExist($this->dir.'/promotion-journal.json');
|
||||
self::assertSame('applied',PanelChange::promote($this->dir,$this->request,true)['state']);self::assertSame(['1'],Server::find(2)->group_ids);self::assertTrue((bool)User::find(1)->banned);self::assertEquals($protected,$this->protectedSnapshot());
|
||||
$after=$this->snapshot();PanelChange::promote($this->dir,$this->request,true);self::assertSame($after,$this->snapshot());
|
||||
self::assertSame('preview',PanelChange::rollback($this->dir,false)['state']);self::assertSame($after,$this->snapshot());
|
||||
PanelChange::rollback($this->dir,true);self::assertSame(['2'],Server::find(2)->group_ids);self::assertFalse(Server::find(2)->enabled);self::assertFalse(Server::find(2)->show);self::assertTrue((bool)User::find(1)->banned);self::assertEquals($protected,$this->protectedSnapshot());
|
||||
$rolled=$this->snapshot();PanelChange::rollback($this->dir,true);self::assertSame($rolled,$this->snapshot());
|
||||
}
|
||||
public function testAppendForbidsSharedWritesAndBadEvidence(): void {
|
||||
$before=$this->snapshot();foreach(['plans'=>'1','subscribe-url'=>'https://bad.example.invalid','group-name'=>'bad'] as $key=>$value){$r=$this->request;$r[$key]=$value;$this->rejects(fn()=>PanelChange::promote($this->dir,$r,true));}
|
||||
foreach([['state'=>'structure_validated'],['qa_user_id'=>2],['verified_at'=>time()-901],['verified_at'=>time()+60],['results'=>[]],['results'=>[['live'=>true,'strict_tls'=>true,'relay'=>'wrong.example.invalid','relay_port'=>47002,'sni'=>'node.example.invalid']]],['results'=>[['live'=>true,'strict_tls'=>false,'relay'=>'relay.example.invalid','relay_port'=>47002,'sni'=>'node.example.invalid']]]] as $bad){$this->evidence($bad);$this->rejects(fn()=>PanelChange::promote($this->dir,$this->request,true));self::assertSame($before,$this->snapshot());}
|
||||
}
|
||||
public function testTransactionFailurePreparedBeforeRecovers(): void {
|
||||
$before=$this->snapshot();$once=true;
|
||||
Server::updating(function()use(&$once){if($once){$once=false;throw new \RuntimeException('Injected transaction failure');}});
|
||||
$this->rejects(fn()=>PanelChange::promote($this->dir,$this->request,true));self::assertSame($before,$this->snapshot());self::assertSame('prepared',\DplOnboarding\readJson($this->dir.'/promotion-journal.json')['state']);
|
||||
PanelChange::promote($this->dir,$this->request,true);self::assertTrue((bool)User::find(1)->banned);
|
||||
}
|
||||
public function testCommittedNotificationFailureRecoversWithoutReactivation(): void {
|
||||
NodeSyncService::$fail=true;$this->rejects(fn()=>PanelChange::promote($this->dir,$this->request,true));self::assertTrue((bool)User::find(1)->banned);self::assertSame(['1'],Server::find(2)->group_ids);
|
||||
self::assertSame('prepared',\DplOnboarding\readJson($this->dir.'/promotion-journal.json')['state']);NodeSyncService::$fail=false;
|
||||
$after=$this->snapshot();PanelChange::promote($this->dir,$this->request,true);self::assertSame($after,$this->snapshot());self::assertSame('applied',\DplOnboarding\readJson($this->dir.'/promotion-journal.json')['state']);
|
||||
}
|
||||
public function testRollbackConflictDoesNotOverwriteLaterChange(): void {
|
||||
PanelChange::promote($this->dir,$this->request,true);Capsule::table('v2_server')->where('id',2)->update(['group_ids'=>'["3"]']);$before=$this->snapshot();
|
||||
$this->rejects(fn()=>PanelChange::rollback($this->dir,true));self::assertSame($before,$this->snapshot());self::assertTrue((bool)User::find(1)->banned);
|
||||
}
|
||||
public function testPreparedMixedStateCannotResume(): void {
|
||||
NodeSyncService::$fail=true;$this->rejects(fn()=>PanelChange::promote($this->dir,$this->request,true));NodeSyncService::$fail=false;
|
||||
Capsule::table('v2_user')->where('id',1)->update(['banned'=>0]);$before=$this->snapshot();$this->rejects(fn()=>PanelChange::promote($this->dir,$this->request,true));self::assertSame($before,$this->snapshot());
|
||||
}
|
||||
public function testDifferentRequestAndRollbackPreventRepromotion(): void {
|
||||
PanelChange::promote($this->dir,$this->request,true);$before=$this->snapshot();$r=$this->request;$r['group']='3';
|
||||
$this->rejects(fn()=>PanelChange::promote($this->dir,$r,true));self::assertSame($before,$this->snapshot());
|
||||
PanelChange::rollback($this->dir,true);$rolled=$this->snapshot();$this->rejects(fn()=>PanelChange::promote($this->dir,$this->request,true));self::assertSame($rolled,$this->snapshot());self::assertTrue((bool)User::find(1)->banned);
|
||||
}
|
||||
public function testRollbackNotificationFailureRecoversAfterState(): void {
|
||||
PanelChange::promote($this->dir,$this->request,true);NodeSyncService::$fail=true;
|
||||
$this->rejects(fn()=>PanelChange::rollback($this->dir,true));self::assertSame('prepared',\DplOnboarding\readJson($this->dir.'/rollback-journal.json')['state']);self::assertFalse(Server::find(2)->enabled);self::assertTrue((bool)User::find(1)->banned);
|
||||
$rolled=$this->snapshot();NodeSyncService::$fail=false;PanelChange::rollback($this->dir,true);self::assertSame($rolled,$this->snapshot());self::assertSame('rolled_back',\DplOnboarding\readJson($this->dir.'/rollback-journal.json')['state']);
|
||||
}
|
||||
public function testQaReactivationAndContainmentChangeBlockRollback(): void {
|
||||
PanelChange::promote($this->dir,$this->request,true);Capsule::table('v2_user')->where('id',1)->update(['banned'=>0]);$before=$this->snapshot();
|
||||
$this->rejects(fn()=>PanelChange::rollback($this->dir,true));self::assertSame($before,$this->snapshot());
|
||||
Capsule::table('v2_user')->where('id',1)->update(['banned'=>1]);Capsule::table('v2_server')->where('id',2)->update(['enabled'=>false]);$before=$this->snapshot();
|
||||
$this->rejects(fn()=>PanelChange::rollback($this->dir,true));self::assertSame($before,$this->snapshot());self::assertTrue((bool)User::find(1)->banned);
|
||||
}
|
||||
}
|
||||
}
|
||||
202
tests/node-onboarding/test_batch.py
Normal file
202
tests/node-onboarding/test_batch.py
Normal file
@ -0,0 +1,202 @@
|
||||
"""Offline process-level tests. No Laravel bootstrap, .env, SSH or network."""
|
||||
import copy
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
RUNNER = ROOT / 'scripts/node-onboarding/batch.py'
|
||||
ADAPTER = ROOT / 'scripts/node-onboarding/command-adapter.py'
|
||||
spec = importlib.util.spec_from_file_location('dpl_batch', RUNNER)
|
||||
batch = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(batch)
|
||||
|
||||
FAKE = r'''
|
||||
import argparse, json, os, sys, time
|
||||
from pathlib import Path
|
||||
p=argparse.ArgumentParser()
|
||||
for x in ('plan','node','phase','action','state-dir'): p.add_argument('--'+x,required=True)
|
||||
a=p.parse_args(); n=json.loads(Path(a.node).read_text()); plan=json.loads(Path(a.plan).read_text())
|
||||
root=Path(plan['fixture_root']); root.mkdir(exist_ok=True)
|
||||
identity=dict(key=n['key'], phase=a.phase, request_sha256=n['request_sha256'])
|
||||
def emit(status, **kw): print(json.dumps(dict(identity,status=status,**kw)))
|
||||
if plan.get('bad_receipt'): identity['request_sha256']='wrong'
|
||||
if a.phase=='preflight':
|
||||
emit('waiting' if plan.get('block_preflight') and n['key']=='jp2' else 'ready'); sys.exit()
|
||||
if a.phase in ('verify','audit'): emit('ready'); sys.exit()
|
||||
marker=root/(n['key']+'-'+a.phase)
|
||||
if a.action=='probe':
|
||||
emit('ready' if marker.exists() else 'absent',retry_safe=True); sys.exit()
|
||||
def log(event):
|
||||
fd=os.open(root/'events',os.O_CREAT|os.O_WRONLY|os.O_APPEND,0o600)
|
||||
os.write(fd,(json.dumps(dict(key=n['key'],phase=a.phase,event=event,t=time.monotonic()))+'\n').encode());os.close(fd)
|
||||
log('start'); time.sleep(1.2 if a.phase=='provision' else .12); marker.write_text('owned'); log('end')
|
||||
if plan.get('crash') and a.phase=='provision' and not (root/'crashed').exists():
|
||||
(root/'crashed').write_text('yes'); sys.exit(8)
|
||||
emit('ready')
|
||||
'''
|
||||
|
||||
|
||||
class BatchTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.tmp.name)
|
||||
self.fake = self.root / 'fake.py'
|
||||
self.fake.write_text(FAKE)
|
||||
self.state = self.root / 'state'
|
||||
self.plan = {
|
||||
'version': 1, 'mode': 'append', 'batch_id': 'offline-test',
|
||||
'context': {'panel_vps': 'VPS-001', 'gateway_vps': 'VPS-002',
|
||||
'jump_vps': 'VPS-003', 'app_root': '/app',
|
||||
'panel_url': 'https://panel.example.invalid',
|
||||
'subscription_url': 'https://sub.example.invalid',
|
||||
'provision_commit': 'a' * 40, 'bundle': '/private/bundle',
|
||||
'client_binary': '/private/sing-box'},
|
||||
'adapter': [sys.executable, str(self.fake)],
|
||||
'fixture_root': str(self.root / 'remote'), 'nodes': [self.node(1)],
|
||||
}
|
||||
self.path = self.root / 'plan.json'
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def node(self, i):
|
||||
return dict(key=f'jp{i}', country='JP', sequence=i, flag='🇯🇵', country_name='日本',
|
||||
name=f'🇯🇵 日本 JP - {i:02d}', vps_id=f'VPS-{10+i:03d}',
|
||||
node_address=f'192.0.2.{i}', listen_port=24443, relay_host='relay.example.invalid',
|
||||
relay_port=47000+i, tls_domain=f'jp{i}.example.invalid', formal_group_id=1,
|
||||
expected_exit=f'192.0.2.{i}', socks_port=19000+i)
|
||||
|
||||
def argv(self, command='run'):
|
||||
self.path.write_text(json.dumps(self.plan, ensure_ascii=False))
|
||||
return [sys.executable, str(RUNNER), command, '--plan', str(self.path),
|
||||
'--state-root', str(self.state), '--jobs', '3'] + (['--execute'] if command == 'run' else [])
|
||||
|
||||
def run_cli(self, command='run', expected=0):
|
||||
p = subprocess.run(self.argv(command), capture_output=True, text=True, timeout=30)
|
||||
self.assertEqual(p.returncode, expected, p.stdout + p.stderr)
|
||||
return json.loads(p.stdout)
|
||||
|
||||
def events(self):
|
||||
path = self.root / 'remote/events'
|
||||
return [json.loads(x) for x in path.read_text().splitlines()] if path.exists() else []
|
||||
|
||||
def test_plan_and_status_are_read_only(self):
|
||||
self.run_cli('plan'); self.run_cli('status')
|
||||
self.assertFalse(self.state.exists())
|
||||
self.assertFalse((self.root/'remote').exists())
|
||||
|
||||
def test_duplicate_parameters_and_naming_rejected(self):
|
||||
for field in ('key','name','vps_id','node_address','tls_domain','relay_port','socks_port','sequence'):
|
||||
with self.subTest(field=field):
|
||||
other = self.node(2); other[field] = self.plan['nodes'][0][field]
|
||||
p = copy.deepcopy(self.plan); p['nodes'].append(other)
|
||||
with self.assertRaises(ValueError): batch.validate(p)
|
||||
p = copy.deepcopy(self.plan); p['nodes'][0]['name'] = '🇯🇵 日本 JP - 11'
|
||||
with self.assertRaises(ValueError): batch.validate(p)
|
||||
|
||||
def test_preflight_barrier_blocks_every_write(self):
|
||||
self.plan['nodes'].append(self.node(2)); self.plan['block_preflight'] = True
|
||||
self.run_cli(expected=2)
|
||||
self.assertEqual(self.events(), [])
|
||||
|
||||
def test_parallel_nodes_serial_panel_and_resume(self):
|
||||
self.plan['nodes'] = [self.node(i) for i in (1,2,3)]
|
||||
self.run_cli()
|
||||
events = self.events()
|
||||
for phase, expected_peak in [('stage',1), ('publish',1), ('provision',None)]:
|
||||
active = peak = 0
|
||||
for e in sorted((e for e in events if e['phase']==phase), key=lambda e:e['t']):
|
||||
active += 1 if e['event']=='start' else -1; peak=max(peak,active)
|
||||
if expected_peak is None:
|
||||
self.assertGreater(peak,1,'Independent nodes should overlap')
|
||||
self.assertLessEqual(peak,3)
|
||||
else:
|
||||
self.assertEqual(peak, expected_peak, phase)
|
||||
self.run_cli()
|
||||
self.assertEqual(self.events(), events, 'A completed retry must not write again')
|
||||
status = self.run_cli('status')
|
||||
self.assertTrue(all(n['audit']=='done' for n in status.values()))
|
||||
self.assertEqual(self.state.stat().st_mode & 0o777, 0o700)
|
||||
self.assertEqual((self.state/'registry.sqlite3').stat().st_mode & 0o777, 0o600)
|
||||
|
||||
def test_lost_response_recovers_by_probe_without_duplicate(self):
|
||||
self.plan['crash'] = True
|
||||
self.run_cli(expected=2)
|
||||
self.run_cli()
|
||||
writes = [e for e in self.events() if e['phase']=='provision' and e['event']=='start']
|
||||
self.assertEqual(len(writes),1)
|
||||
self.assertEqual(self.run_cli('status')['jp1']['audit'],'done')
|
||||
|
||||
def test_cross_batch_and_plan_drift_rejected_before_commands(self):
|
||||
self.run_cli(); events = self.events()
|
||||
self.plan['nodes'][0]['expected_exit'] = '192.0.2.99'
|
||||
self.run_cli(expected=2)
|
||||
self.plan['batch_id'] = 'other-batch'
|
||||
self.run_cli(expected=2)
|
||||
self.assertEqual(self.events(), events)
|
||||
|
||||
def test_two_processes_share_locks_and_do_not_duplicate(self):
|
||||
argv = self.argv()
|
||||
a = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
b = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
for process in (a,b):
|
||||
out, err = process.communicate(timeout=30)
|
||||
self.assertEqual(process.returncode,0,out+err)
|
||||
stages = [e for e in self.events() if e['phase']=='stage' and e['event']=='start']
|
||||
self.assertEqual(len(stages),1)
|
||||
|
||||
def test_invalid_receipt_cannot_authorize_write(self):
|
||||
self.plan['bad_receipt'] = True
|
||||
self.run_cli(expected=2)
|
||||
self.assertEqual(self.events(), [])
|
||||
|
||||
def test_interrupted_apply_records_unknown_and_can_resume(self):
|
||||
p = subprocess.Popen(self.argv(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
try:
|
||||
deadline = time.monotonic() + 10
|
||||
while not any(e['phase']=='provision' and e['event']=='start' for e in self.events()):
|
||||
self.assertIsNone(p.poll())
|
||||
self.assertLess(time.monotonic(), deadline)
|
||||
time.sleep(.02)
|
||||
p.send_signal(signal.SIGTERM)
|
||||
out, err = p.communicate(timeout=10)
|
||||
self.assertEqual(p.returncode,2,out+err)
|
||||
self.assertEqual(self.run_cli('status')['jp1']['provision'],'unknown')
|
||||
self.run_cli()
|
||||
self.assertEqual(self.run_cli('status')['jp1']['audit'],'done')
|
||||
finally:
|
||||
if p.poll() is None:
|
||||
p.kill(); p.communicate()
|
||||
|
||||
def test_configured_command_adapter_runs_full_protocol(self):
|
||||
self.plan['adapter'] = [sys.executable,str(ADAPTER)]
|
||||
command = [sys.executable,str(self.fake),'--plan','{plan}','--node','{node}',
|
||||
'--phase','{phase}','--action','{action}','--state-dir','{state_dir}']
|
||||
self.plan['command_profile'] = {
|
||||
'files': {str(self.fake): hashlib.sha256(self.fake.read_bytes()).hexdigest()},
|
||||
'phases': {phase: {'probe': command, 'apply': command}
|
||||
for phase in ('preflight',) + batch.PHASES}}
|
||||
self.run_cli()
|
||||
self.assertEqual(self.run_cli('status')['jp1']['audit'],'done')
|
||||
|
||||
def test_command_adapter_pins_handlers_and_missing_steps_wait(self):
|
||||
self.plan['adapter'] = [sys.executable,str(ADAPTER)]
|
||||
self.plan['command_profile'] = {'files': {str(self.fake): hashlib.sha256(self.fake.read_bytes()).hexdigest()},
|
||||
'phases': {}}
|
||||
self.run_cli(expected=2)
|
||||
self.assertEqual(self.run_cli('status')['jp1']['preflight'],'waiting')
|
||||
self.fake.write_text('# changed')
|
||||
self.run_cli(expected=2)
|
||||
self.assertEqual(self.run_cli('status')['jp1']['preflight'],'unknown')
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
Loading…
Reference in New Issue
Block a user