fix(sync,memory,gpu): restorable memory dump, curator grounding, GPU + Models fixes

Ported from downstream development. Four independent defects.

1. The memory dump was unrestorable. iterdump() serializes sqlite_vec virtual
   tables as a raw INSERT INTO sqlite_master(...) followed by inserts into a
   table the replaying connection cannot see, so replaying memory.db.sql died
   on "no such table: vec_messages" and left ZERO tables behind. dump_db() now
   loads the vec0 extension and filters the derived vec tables out of the
   iterdump stream, matched on each statement's target table rather than as a
   substring - a chat message whose text mentions vec_messages is an
   INSERT INTO "messages" and has to survive.

   compare() reported an unreadable dump as "diverged", which read like a real
   verdict and made both guards refuse backup AND restore, locking the machine
   out of syncing in either direction. Unreadable is now its own verdict.

   _extra() compared updated_at against a "" default, but the column is REAL,
   so the comparison raises TypeError on the first conversation the other side
   lacks - exactly the case it counts. It tests membership first now. The
   direction test declared updated_at TEXT, which is why this survived: the
   test compared str to str while the field compared str to float.

2. The memory curator invented facts. It attributed the ASSISTANT's words to
   the user, wrote absence claims read off the existing-memory block, and added
   judgements ("favorite") the user never used. The prompt now scopes the USER
   line as the only source, and two deterministic guards drop absence claims
   and facts whose distinctive tokens appear nowhere in the user's message -
   prompt wording alone did not hold on a 7B curator.

3. _best_vulkan_device scored Mesa's llvmpipe above an integrated GPU, pinning
   Ollama to a software rasterizer advertising 31 GiB of "VRAM" - CPU inference
   with Vulkan overhead on top. Software rasterizers are dropped.

4. Models.jsx compared catalog names to installed names literally, but Ollama
   resolves a bare name to ":latest", so an untagged entry (nomic-embed-text)
   read as missing forever and the Required gate never opened. Chatbot.jsx
   fetched the model list once on mount although App keeps the page mounted
   behind display:none, so a newly pulled model never appeared in the picker
   until a full browser reload.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
janvanwan
2026-08-14 17:33:43 -05:00
co-authored by Claude Opus 5
parent b5541d1c48
commit 8691a67803
13 changed files with 443 additions and 47 deletions
+18 -6
View File
@@ -33,10 +33,6 @@ cp -f "$NEXUS/management/autostart/nm-tray-autostart.desktop" "$AUTOSTART/nm-tra
cp -f "$NEXUS/management/autostart/blueman.desktop" "$AUTOSTART/blueman.desktop"
cp -f "$NEXUS/management/autostart/blueman-applet.desktop" "$AUTOSTART/blueman-applet.desktop"
# Genmon configs (regular files — panel writes back to them)
cp -f "$NEXUS/management/panel/genmon-13.rc" "$PANEL_CFG/genmon-13.rc"
cp -f "$NEXUS/management/panel/genmon-15.rc" "$PANEL_CFG/genmon-15.rc"
cp -f "$NEXUS/management/panel/genmon-16.rc" "$PANEL_CFG/genmon-16.rc"
# ── Register the genmon applets into the XFCE panel ────────────────────────
# The network applet's genmon (plugin-13) was added by hand once; the Nexus
@@ -71,8 +67,24 @@ register_genmon() {
register_genmon 15 before # Nexus applet, left of the network applet
register_genmon 16 after # Bluetooth applet, right of the network applet
echo "Reloading panel…"
xfce4-panel -r >/dev/null 2>&1 || true
# Genmon configs. These MUST be written with the panel down: genmon keeps its
# config in memory and rewrites genmon-N.rc when the panel exits, so a copy made
# while it's running is overwritten by the very reload meant to pick it up — a
# newly registered plugin has an empty in-memory config, which is how all three
# applets came back blank (Command=, UseLabel=1, "(genmon)" label).
if command -v xfce4-panel >/dev/null 2>&1; then
echo "Restarting panel with applet configs…"
xfce4-panel -q >/dev/null 2>&1 || true
sleep 1
fi
for id in 13 15 16; do
cp -f "$NEXUS/management/panel/genmon-$id.rc" "$PANEL_CFG/genmon-$id.rc"
done
if command -v xfce4-panel >/dev/null 2>&1; then
setsid xfce4-panel >/dev/null 2>&1 < /dev/null &
sleep 1
fi
# Start the popup daemon now so the first click works without a re-login.
# Pin to the system python3 explicitly: PyGObject (gi) is a system package, and
+81 -4
View File
@@ -19,6 +19,7 @@ A fresh machine clones first (git clone <repo> nexus-core), then runs this.
"""
import argparse
import os
import re
import shutil
import sqlite3
import subprocess
@@ -118,6 +119,42 @@ def requirements() -> str:
# -- memory DB -----------------------------------------------------------------
def _vec0_extension() -> "Path | None":
"""Path to the sqlite_vec native extension inside the venv, if built. Found
by glob rather than importing sqlite_vec - this script runs stdlib-only,
before the venv necessarily exists."""
matches = list((ROOT / "Promethean").glob("**/sqlite_vec/vec0.*"))
return matches[0] if matches else None
_DERIVED = re.compile(
r'^\s*(?:CREATE\s+(?:VIRTUAL\s+)?TABLE|INSERT\s+INTO)\s+"?vec_(?:messages|documents)',
re.IGNORECASE,
)
def _is_derived_stmt(stmt: str) -> bool:
"""True for a dump statement that builds or fills a sqlite_vec table.
Matched on the statement's TARGET, never as a substring: a chat message
whose text happens to mention vec_messages is an INSERT INTO "messages" and
has to survive. (It didn't, the first time - the test caught it.)
"""
if _DERIVED.match(stmt):
return True
# iterdump writes a virtual table straight into the schema table, and parks
# its shadow tables' AUTOINCREMENT counters in sqlite_sequence. Both name the
# vec table as a quoted VALUE rather than as the statement's target.
head = stmt.lstrip().upper()
if (head.startswith("INSERT INTO SQLITE_MASTER")
or head.startswith('INSERT INTO "SQLITE_SEQUENCE"')) and "'vec_" in stmt:
return True
# Only meaningful when replaying over an existing DB, and sqlite_sequence
# exists only if some AUTOINCREMENT table survives the filter - today true
# by luck alone. Restore deletes the DB file first, so drop it.
return stmt.lstrip().upper().startswith('DELETE FROM "SQLITE_SEQUENCE"')
def dump_db() -> bool:
"""Dump the (gitignored, binary, WAL) memory DB to a diff-friendly SQL file
so git backs up the assistant's memory + conversation history. sqlite3 reads
@@ -126,7 +163,27 @@ def dump_db() -> bool:
return False
try:
with sqlite3.connect(f"file:{DB}?mode=ro", uri=True) as conn:
DB_SQL.write_text("\n".join(conn.iterdump()) + "\n", encoding="utf-8")
ext = _vec0_extension()
if ext:
try:
conn.enable_load_extension(True)
conn.load_extension(str(ext))
conn.enable_load_extension(False)
except (sqlite3.OperationalError, AttributeError):
pass # iterdump below will fail with a clear "no such module" if this was needed
# Drop the sqlite_vec virtual tables and their shadow tables.
# iterdump() serializes a virtual table as a raw
# INSERT INTO sqlite_master(...) followed by inserts into a table
# the connection can't see yet, so replaying the dump dies with
# "no such table: vec_messages" and leaves ZERO tables behind -
# the whole backup was unrestorable. They're derived data anyway:
# _backfill_vec_msgs()/_backfill_vec() rebuild both indexes from
# message_vectors and documents on the next search.
# Filter the generator, not the joined text - each yield is one
# complete statement, while splitting the text on ";\n" tears
# apart INSERTs whose content contains newlines.
stmts = [s for s in conn.iterdump() if not _is_derived_stmt(s)]
DB_SQL.write_text("\n".join(stmts) + "\n", encoding="utf-8")
except sqlite3.Error as exc:
print(f"Warning: could not dump {DB} ({exc}) - DB not captured")
return False
@@ -158,13 +215,17 @@ def _extra(a, b) -> int:
'one box is simply ahead' from a real divergence."""
a_conv, a_mem = a
b_conv, b_mem = b
newer = sum(1 for cid, ts in a_conv.items() if b_conv.get(cid, "") < ts)
# Membership test first, never a placeholder default: updated_at is a REAL
# column, so ANY typed default is a cross-type comparison against the one
# case this function exists to count - a conversation the other side lacks.
# ("" < 1778553309.83 raises TypeError; 0 only worked by accident.)
newer = sum(1 for cid, ts in a_conv.items() if cid not in b_conv or b_conv[cid] < ts)
return newer + len(a_mem - b_mem)
def compare(db: Path = DB, dump: Path = DB_SQL) -> str:
"""Which way the sync should go. One of: same, local-ahead, local-behind,
diverged, no-live, no-dump.
diverged, no-live, no-dump, unreadable.
ponytail: detects direction, does not merge. Diverged is reported, not resolved."""
if not dump.exists() or not dump.stat().st_size:
return "no-dump"
@@ -177,7 +238,11 @@ def compare(db: Path = DB, dump: Path = DB_SQL) -> str:
dump_conn.executescript(dump.read_text(encoding="utf-8"))
backup = _state(dump_conn)
except (sqlite3.Error, OSError):
return "diverged" # can't tell: fail safe, refuse both directions
# A dump that won't replay is NOT a divergence. Reporting it as one
# blocked backup and restore alike with a verdict that looked like a
# real answer - dumps written before the vec-table filter above land
# here every time. Still refuses both directions, but says why.
return "unreadable"
ahead, behind = _extra(live, backup), _extra(backup, live)
if ahead and behind:
return "diverged"
@@ -208,6 +273,12 @@ def restore_db() -> None:
if state == "no-dump":
print("No memory dump in the backup - skipping DB restore.")
return
if state == "unreadable":
print("WARNING: the backup's memory dump will not replay (written before the")
print(" vec-table fix). Not applying it - it would leave an empty DB.")
print(" Run `backup --force-db` on the machine with the good history")
print(" to publish a clean dump, then restore here.")
return
print("Restoring memory DB from backup...")
rollback = DB.with_suffix(".db.pre-restore")
@@ -318,6 +389,12 @@ def check_db_direction(tmp_dir: Path) -> bool:
print(" the other lacks. Nothing merges these automatically.")
print(" Force this box's history to win: python bin/sync.py backup --force-db")
return False
if state == "unreadable":
print("REFUSING TO BACK UP: the remote memory dump will not replay, so there is")
print(" no way to tell whether it holds history this machine lacks.")
print(" If this box has the good history: python bin/sync.py backup --force-db")
print(" (that publishes a clean dump and clears this for good).")
return False
return True