graphify --update 与 --cluster-only：增量更新与重新聚类的完整工作原理

graphify --update 与 --cluster-only增量更新与重新聚类的完整工作原理【免费下载链接】graphifyTurn any codebase, with its docs, SQL schemas, configs, and PDFs, into a queryable knowledge graph. A /graphify skill for Claude Code, Cursor, Codex, and Gemini CLI: local deterministic AST parsing, every edge explained, no vector store.项目地址: https://gitcode.com/GitHub_Trending/graph/graphifygraphify 的知识图谱graphify-out/graph.json一旦构建完成后续维护几乎都不需要从头重建--update只重新抽取新增/修改的文件--cluster-only则在已有图上重跑社区发现并刷新全部产物。本文以 graphify 技能包中 update.md 这份增量更新参考文档为主线完整拆解两个子命令的每一步命令、中间文件状态机并结合 graphify/detect.py、graphify/build.py 的源码说明只重抽变更文件背后可靠的变更检测、替换式合并与清单manifest机制。定位update.md 是 /graphify 技能的增量参考文档在 skill-agents.md 定义的/graphify流水线中首次全量构建依次执行 Step 1确认解释器到 Step 9保存 manifest、清理、汇报而--update和--cluster-only被明确定义为非默认子命令/graphify path --update # incremental - re-extract only new/changed files /graphify path --cluster-only # rerun clustering on existing graph主技能文档只保留一句话指引——Seereferences/update.mdfor both flows完整的增量运行手册就是本文拆解的 update.md。该文件开头即声明加载时机只有用户传了--update或--cluster-only时才读取它首次全量构建永远不会读它。两个分支的定位分别是--update增量重抽取上次运行之后有文件新增或修改时使用。只重抽变更文件节省 token 与时间。--cluster-only仅重聚类跳过 Step 1–3 的抽取直接在现有图上重跑聚类、命名社区并重新生成GRAPH_REPORT.md、graph.json、graph.html。两个分支中所有 bash 块都通过$(cat graphify-out/.graphify_python)调用 Python 解释器——这是主技能 Step 1 写入的解释器路径文件技能包对--update、query等子命令都要求先做这个解释器守卫缺失时重新解析保证增量流程使用与首次构建相同的graphify包环境。文中INPUT_PATH占位符替换为用户实际路径IS_DIRECTED在给了--directed时替换为True、否则False。--update 流程第一步detect_incremental 检测变更首次运行增量流程时先调用detect_incremental与上次运行的 manifest 对比产出变更集并落盘为graphify-out/.graphify_incremental.json$(cat graphify-out/.graphify_python) -c import sys, json from graphify.detect import detect_incremental, save_manifest from pathlib import Path result detect_incremental(Path(INPUT_PATH)) new_total result.get(new_total, 0) print(json.dumps(result, indent2, ensure_asciiFalse)) Path(graphify-out/.graphify_incremental.json).write_text(json.dumps(result, ensure_asciiFalse), encoding\utf-8\) deleted list(result.get(deleted_files, [])) if new_total 0 and not deleted: print(No files changed since last run. Nothing to update.) raise SystemExit(0) if deleted: print(f{len(deleted)} deleted file(s) to prune.) if new_total 0: print(f{new_total} new/changed file(s) to re-extract.) 如果既无新文件也无删除流程在此安全终止。返回值的关键字段可以结合 detect.py 中detect_incremental的实现 理解new_files/unchanged_files/new_total按文件类型code/document/paper/image/video分桶的新增或修改文件。deleted_files与excluded_files的严格区分#1908manifest 中消失的行要按磁盘上是否还存在分流——文件从磁盘删掉了才是真删除其缓存节点成了幽灵文件仍在磁盘但不在本次扫描结果里只是被排除ignore 规则或--exclude变更绝不能报成删除。这与 watch.py 中_reconcile_existing_graph的 fail-closed 驱逐逻辑 是同一套原则。快速路径与慢速路径mtime未变且 hash 匹配则直接判定未变更只花一次stat的代价mtime变了才走慢速路径用 MD5 内容与 manifest 里对应的ast_hash/semantic_hash对比后才决定是否重抽见 detect.py 的 docstring。用!而非比较 mtime是为了让 git checkout 旧提交、tarball 恢复这类 mtime 回退也能触发重抽#1859。kindast与kindsemantic双 hashmanifest 条目同时记录ast_hash和semantic_hash。AST 层是确定性解析、免费语义层要消耗 LLM token所以两者分开记账semantic_hash缺失意味着该文件还没做过语义抽取下次extract必须补上。无 manifest 的首次运行detect_incremental把全部文件视为new_files等价于全量。第二步填充 .graphify_detect.json 供后续步骤消费--update复用了全量构建的 Step 3A–6而这些步骤无条件读取graphify-out/.graphify_detect.json。因此要把增量结果改写成 detect 的格式files字段只装变更子集驱动 Step 3A 的 AST 抽取和 Step 3B0 的缓存检查只作用于变更文件all_files字段装全量语料供需要全局上下文的步骤使用$(cat graphify-out/.graphify_python) -c import json from pathlib import Path r json.loads(Path(graphify-out/.graphify_incremental.json).read_text(encoding\utf-8\)) Path(graphify-out/.graphify_detect.json).write_text(json.dumps({ files: r.get(new_files, {}), all_files: r.get(files, {}), total_files: r.get(new_total, 0), total_words: r.get(total_words, 0), skipped_sensitive: r.get(skipped_sensitive, []), needs_graph: True, }, ensure_asciiFalse), encoding\utf-8\) 第三步code_only 快速路径判断如果存在新文件先判断所有变更文件是否都是代码文件$(cat graphify-out/.graphify_python) -c import json from pathlib import Path result json.loads(open(graphify-out/.graphify_incremental.json, encodingutf-8).read()) if Path(graphify-out/.graphify_incremental.json).exists() else {} code_exts {.py,.ts,.js,.go,.rs,.java,.cpp,.c,.rb,.swift,.kt,.cs,.scala,.php,.cc,.cxx,.hpp,.h,.kts,.lua,.toc,.f,.F,.f90,.F90,.f95,.F95,.f03,.F03,.f08,.F08} new_files result.get(new_files, {}) all_changed [f for files in new_files.values() for f in files] code_only all(Path(f).suffix.lower() in code_exts for f in all_changed) print(code_only:, code_only) 这个分支与主技能 Step 3 的设计一致代码走确定性 AST 抽取Part A完全不需要 LLM语义抽取Part B子代理读文档/论文/图片才花 token。因此code_only为 True打印[graphify update] Code-only changes detected - skipping semantic extraction (no LLM needed)只对变更文件跑 Step 3AAST完全跳过 Step 3B不派子代理然后直接进入合并与 Step 4–8。code_only为 False任一变更文件是文档/论文/图片/视频先检查变更集里是否有new_files[video]。若有必须先按 transcribe.mdStep 2.5把这些视频/音频转写成文本然后重写.graphify_detect.json把转写产物路径挪进files[document]并删掉files[video]——否则原始.mp4/.mp3路径会被当作不可读的媒体直接喂给语义子代理#1392。之后正常跑完整的 Step 3A–3C 管线。只有删除时的空抽取让合并步骤可以裁剪若本次没有新增文件、只有删除要人为造一个空抽取让后续的 merge 步骤有输入可裁剪if [ ! -f graphify-out/.graphify_extract.json ]; then echo [graphify update] Only deletions -- creating empty extraction for merge. $(cat graphify-out/.graphify_python) -c import json from pathlib import Path Path(graphify-out/.graphify_extract.json).write_text(json.dumps({nodes:[],edges:[],hyperedges:[],input_tokens:0,output_tokens:0}), encodingutf-8) fi核心合并build_merge 的三个关键参数合并步骤是整个--update的心脏完整代码原文档原样保留注意其中的注释解释了三个历史坑位$(cat graphify-out/.graphify_python) -c import json from pathlib import Path from graphify.build import build_merge from graphify.detect import save_manifest # Load new extraction and incremental state new_extraction json.loads(Path(graphify-out/.graphify_extract.json).read_text(encoding\utf-8\)) incremental json.loads(Path(graphify-out/.graphify_incremental.json).read_text(encoding\utf-8\)) deleted list(incremental.get(deleted_files, [])) # prune_sources is ONLY for genuinely DELETED files. Changed/re-extracted files are # handled by build_merges replace-on-re-extract (#1344): every source_file in # new_chunks is dropped from the base before merge, so old/stale nodes dont survive. # Do NOT add changed here: with root passed, prune_set relativizes to the same base # as the freshly merged nodes and would DELETE the re-extracted content (#1178 is moot # now that replace — not the dedup pass — reconciles changed files). prune list(deleted) or None # Use build_merge() — reads graph.json directly without NetworkX round-trip # so edge direction (calls, implements, imports) is always preserved (#801). # Pass root so prune_sources (absolute paths from detect_incremental) are # relativized to match the graphs relative source_file values; without it # nothing is pruned and stale nodes accumulate on every update (#1361). # directedIS_DIRECTED: replace IS_DIRECTED with True if --directed was given, else # False. Without it a --directed --update silently rebuilds undirected and collapses # reciprocal A-B edges (#1392). G build_merge( [new_extraction], graph_pathgraphify-out/graph.json, prune_sourcesprune, rootINPUT_PATH, directedIS_DIRECTED, ) print(f[graphify update] Merged: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges) # Write merged result back to .graphify_extract.json so Step 4 sees the full graph merged_out { nodes: [{id: n, **d} for n, d in G.nodes(dataTrue)], edges: [ # Explicit source/target last so they win over any stale attrs in d. {**{k: val for k, val in d.items() if k not in (_src, _tgt, source, target)}, source: d.get(_src, u), target: d.get(_tgt, v)} for u, v, d in G.edges(dataTrue) ], # G.graph[hyperedges] holds hyperedges from both existing graph.json # and new_extraction (build_merge combines them). Falling back to # new_extraction only would silently drop prior-run hyperedges (#801). hyperedges: list(G.graph.get(hyperedges, [])), input_tokens: new_extraction.get(input_tokens, 0), output_tokens: new_extraction.get(output_tokens, 0), } Path(graphify-out/.graphify_extract.json).write_text(json.dumps(merged_out, ensure_asciiFalse), encoding\utf-8\) print(f[graphify update] Merged extraction written ({len(merged_out[\nodes\])} nodes, {len(merged_out[\edges\])} edges)) # Save manifest so next --update diffs against todays state, not the # prior runs baseline (prevents ghost-node reports on subsequent updates). # root matches the build_merge call above so the manifest keys stay relative to # the scan root — portable across clones/machines, so --update keeps matching # cached files instead of missing every one after a move (#1417). # # Only stamp semantic files (docs/papers/images) that ACTUALLY produced output # THIS run (new_extraction is this runs fresh extraction, read above before the # merge overwrote the file): a changed doc whose chunk failed must stay unstamped # so the next --update re-queues it, otherwise it is marked done and its content # is lost forever (#2015). Mirrors the library extract path # (cli._stamped_manifest_files clear_semantic scan_corpus). from graphify.cli import _stamped_manifest_files _manifest_files _stamped_manifest_files(incremental[files], new_extraction, Path(INPUT_PATH)) # Changed semantic files dispatched this run but NOT stamped had their chunk fail # or be omitted; clear any stale semantic_hash so they are re-queued (#1948). _sem_types (document, paper, image) _dispatched {f for t, fl in incremental.get(new_files, {}).items() if t in _sem_types for f in fl} _stamped {f for fl in _manifest_files.values() for f in fl} _cleared _dispatched - _stamped # scan_corpus the RAW full corpus so in-root files newly excluded since last run # are dropped rather than masquerading as deletions; untouched rows preserved (#1908). _scan {f for fl in incremental[files].values() for f in fl} save_manifest(_manifest_files, rootINPUT_PATH, scan_corpus_scan, clear_semantic_cleared or None) print([graphify update] Manifest saved.) 这段代码值得逐点对照源码理解它把三个曾经真实发生过的缺陷固化成了调用约定1. 替换式重抽而不是追加式合并build_merge 的 docstring 说明其核心语义new_chunks中出现的每个source_file其既有贡献会在合并前从旧图里整体丢弃#1344 的 replace-on-re-extract。否则变更文件里已经删掉的函数节点、改掉的边会永久残留增量会退化成只增不减。实现上这一替换是按层级tier作用域的#2333/#2336同一文件有两类生产者——确定性的 AST 路径与语义/LLM 路径二者节点在图中并存一次只重抽 AST 层时只替换该文件的 AST 层贡献语义层完好无损。源码在 build_merge 内部 把new_chunks的节点按_is_ast_tier分成new_ast_sources/new_sem_sources只对同层级的旧条目执行丢弃。这直接推出注释里那句告诫prune_sources只能传真删除的文件绝不能把changed文件塞进去——被重抽的文件是替换不是删除两者冲突时替换必须赢。2. root 决定路径能否对上detect_incremental给出的删除文件是绝对路径而图里存的source_file是相对路径。build_merge传root后会把 prune 集合同新 chunk 的路径都相对化到同一基准#1361不传root时什么都剪不掉陈旧节点每次 update 都累积。build_merge 中_eff_root的注释 还记录了 #1571技能运行手册曾不传root导致绝对路径的删除文件永远匹配不上相对键的节点。若调用方省略root实现会回退到从graph.json记录的扫描根推断两种形态都能对上。3. directed 标志不能丢directedIS_DIRECTED必须显式传--directed --update若不传会静默重建为无向图把双向 A↔B 边坍缩成一条#1392。build_merge 的 directed 参数语义 是None时继承磁盘上已有图的directed标志#2342显式 True/False 则永远覆盖它——运行手册选择显式传保证与用户原始构建的取向一致。另外两个细节也来自同一处注释build_merge直接读graph.json而不做 NetworkX 的 node-link 往返所以calls/implements/imports等边方向始终保真#801超边hyperedges要从G.graph[hyperedges]取它已合并了旧图与新抽取只回退到new_extraction会静默丢掉前几轮的超边同为 #801。manifest 保存的四个参数合并后写回 manifest 时调用了save_manifest四个参数各解决一个持久化正确性问题与 save_manifest 的 docstring 一一对应rootmanifest 键相对化到扫描根磁盘格式跨克隆、跨机器可移植--update迁移后依然能命中缓存文件而不是全部未命中#1417。相对键在 load_manifest 读回时再锚回绝对路径且做 NFC 归一化以兼容 macOS 的 NFD 文件名#2221。只给真正产出了输出的语义文件盖戳_stamped_manifest_filescli.py按本次新鲜抽取过滤某个变更文档的 chunk 若失败它保持未盖章下次--update自动重新排队——否则会被标记完成、内容永久丢失#2015。代码文件恒盖章因为 AST 是确定性的。clear_semantic本次派发了语义抽取但没盖戳的文件chunk 失败或被 LLM 遗漏清掉其陈旧的semantic_hash防止detect_incremental下次把它读成未变更#1948。scan_corpus传入原始全量语料而非过滤后的子集使上次运行后新被排除的 in-root 文件被从 manifest 中丢弃而不是伪装成删除未触碰文件的既有行保留#1908。合并之后Step 4–8 与图谱 diff合并完成后对合并后的图正常跑 Step 4–8构建、聚类、分析、报告、导出、清理。在 Step 4 之后额外展示图谱差异对比更新前的备份图$(cat graphify-out/.graphify_python) -c import json from graphify.analyze import graph_diff from graphify.build import build_from_json from networkx.readwrite import json_graph import networkx as nx from pathlib import Path # Load old graph (before update) from backup written before merge old_data json.loads(Path(graphify-out/.graphify_old.json).read_text(encoding\utf-8\)) if Path(graphify-out/.graphify_old.json).exists() else None new_extract json.loads(Path(graphify-out/.graphify_extract.json).read_text(encoding\utf-8\)) G_new build_from_json(new_extract, directedIS_DIRECTED) if old_data: G_old json_graph.node_link_graph(old_data, edgeslinks) diff graph_diff(G_old, G_new) print(diff[summary]) if diff[new_nodes]: print(New nodes:, , .join(n[label] for n in diff[new_nodes][:5])) if diff[new_edges]: print(New edges:, len(diff[new_edges])) 配套的备份与清理动作在运行手册中各占一行合并前cp graphify-out/graph.json graphify-out/.graphify_old.json保存旧图流程结束后rm -f graphify-out/.graphify_old.json清理。graph_diff的实现在 analyze.py返回包含summary、new_nodes、new_edges等键的对比结果让用户能直观看到这次增量到底改变了什么。--cluster-only自包含的重新聚类--cluster-only分支极简——跳过 Step 1–3只跑一条命令graphify cluster-only .graphify cluster-only .是自包含的它基于现有graph.json重新聚类、命名社区并重新生成GRAPH_REPORT.md、graph.json与graph.html。运行手册特别警告不要重跑 Step 5–9。原因很具体——这些步骤读取的中间文件.graphify_extract.json、.graphify_detect.json、.graphify_analysis.json早在上一次构建的 Step 9 清理阶段就被删掉了重跑会抛FileNotFoundError#1392。命令跑完后照常向用户展示刷新后的GRAPH_REPORT.md摘要即可。适用场景是图本身没变但你想要新的社区划分例如换了聚类参数后或对报告的社区命名不满意只想重生成报告和可视化而完全避开昂贵的抽取阶段。与 --watch / 提交钩子的关系同一套替换式合并--update的手动流程并非孤立存在。watch.py 中的自动重建路径--watch监视目录、post-commit 钩子复用同一套语义_reconcile_existing_graph里重抽文件按层级替换 AST 贡献、语义层保留、删除与排除严格区分、远程/虚拟 source 永不驱逐等规则与本文--update手册中prune_sources/replace-on-re-extract的原则完全同构。可以推断手动--update与自动 watch 只是触发方式不同底层的图一致性规则是共享的。watch 路径还额外处理并发安全fcntl.flock的 .rebuild.lock 顾问锁与无法拿锁时的 .pending_changes 排队机制这些属于 add-watch.md 参考文档的主题此处不展开。小结增量更新的状态机一览把--update的中间文件串起来就是一条清晰的状态链阶段输入输出关键函数变更检测graphify-out/manifest.json 磁盘.graphify_incremental.jsondetect_incrementaldetect.py状态回填.graphify_incremental.json.graphify_detect.jsonfiles变更集all_files全量—分支判断变更集扩展名code-only 走纯 AST有视频先转写—抽取变更文件.graphify_extract.json含空抽取的删除场景Step 3A/3C合并graph.json 新抽取合并图写回.graphify_extract.jsonbuild_mergebuild.py记账本次新鲜抽取manifest相对键、选择性盖章、清除失败 hashsave_manifestdetect.py报告差异.graphify_old.json备份节点/边增删摘要graph_diffanalyze.py理解了这条链就能回答增量场景下的绝大多数问题为什么变更文件的旧节点不会残留替换式重抽按层级作用域为什么删除的文件会被正确裁剪prune_sourcesroot相对化为什么失败的语义抽取下次会自动重试选择性盖章 clear_semantic以及为什么迁移目录后增量仍然生效manifest 相对键。--cluster-only则是这条链之外的旁路不碰抽取只重算社区与报告。【免费下载链接】graphifyTurn any codebase, with its docs, SQL schemas, configs, and PDFs, into a queryable knowledge graph. A /graphify skill for Claude Code, Cursor, Codex, and Gemini CLI: local deterministic AST parsing, every edge explained, no vector store.项目地址: https://gitcode.com/GitHub_Trending/graph/graphify创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考