forked from enderofwings/NexusOS
fix(sync,memory,gpu): restorable memory dump, curator grounding, GPU + Models fixes
Ported from downstream development. Four independent defects.
1. The memory dump was unrestorable. iterdump() serializes sqlite_vec virtual
tables as a raw INSERT INTO sqlite_master(...) followed by inserts into a
table the replaying connection cannot see, so replaying memory.db.sql died
on "no such table: vec_messages" and left ZERO tables behind. dump_db() now
loads the vec0 extension and filters the derived vec tables out of the
iterdump stream, matched on each statement's target table rather than as a
substring - a chat message whose text mentions vec_messages is an
INSERT INTO "messages" and has to survive.
compare() reported an unreadable dump as "diverged", which read like a real
verdict and made both guards refuse backup AND restore, locking the machine
out of syncing in either direction. Unreadable is now its own verdict.
_extra() compared updated_at against a "" default, but the column is REAL,
so the comparison raises TypeError on the first conversation the other side
lacks - exactly the case it counts. It tests membership first now. The
direction test declared updated_at TEXT, which is why this survived: the
test compared str to str while the field compared str to float.
2. The memory curator invented facts. It attributed the ASSISTANT's words to
the user, wrote absence claims read off the existing-memory block, and added
judgements ("favorite") the user never used. The prompt now scopes the USER
line as the only source, and two deterministic guards drop absence claims
and facts whose distinctive tokens appear nowhere in the user's message -
prompt wording alone did not hold on a 7B curator.
3. _best_vulkan_device scored Mesa's llvmpipe above an integrated GPU, pinning
Ollama to a software rasterizer advertising 31 GiB of "VRAM" - CPU inference
with Vulkan overhead on top. Software rasterizers are dropped.
4. Models.jsx compared catalog names to installed names literally, but Ollama
resolves a bare name to ":latest", so an untagged entry (nomic-embed-text)
read as missing forever and the Required gate never opened. Chatbot.jsx
fetched the model list once on mount although App keeps the page mounted
behind display:none, so a newly pulled model never appeared in the picker
until a full browser reload.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
b5541d1c48
commit
8691a67803
+18
-6
@@ -33,10 +33,6 @@ cp -f "$NEXUS/management/autostart/nm-tray-autostart.desktop" "$AUTOSTART/nm-tra
|
||||
cp -f "$NEXUS/management/autostart/blueman.desktop" "$AUTOSTART/blueman.desktop"
|
||||
cp -f "$NEXUS/management/autostart/blueman-applet.desktop" "$AUTOSTART/blueman-applet.desktop"
|
||||
|
||||
# Genmon configs (regular files — panel writes back to them)
|
||||
cp -f "$NEXUS/management/panel/genmon-13.rc" "$PANEL_CFG/genmon-13.rc"
|
||||
cp -f "$NEXUS/management/panel/genmon-15.rc" "$PANEL_CFG/genmon-15.rc"
|
||||
cp -f "$NEXUS/management/panel/genmon-16.rc" "$PANEL_CFG/genmon-16.rc"
|
||||
|
||||
# ── Register the genmon applets into the XFCE panel ────────────────────────
|
||||
# The network applet's genmon (plugin-13) was added by hand once; the Nexus
|
||||
@@ -71,8 +67,24 @@ register_genmon() {
|
||||
|
||||
register_genmon 15 before # Nexus applet, left of the network applet
|
||||
register_genmon 16 after # Bluetooth applet, right of the network applet
|
||||
echo "Reloading panel…"
|
||||
xfce4-panel -r >/dev/null 2>&1 || true
|
||||
|
||||
# Genmon configs. These MUST be written with the panel down: genmon keeps its
|
||||
# config in memory and rewrites genmon-N.rc when the panel exits, so a copy made
|
||||
# while it's running is overwritten by the very reload meant to pick it up — a
|
||||
# newly registered plugin has an empty in-memory config, which is how all three
|
||||
# applets came back blank (Command=, UseLabel=1, "(genmon)" label).
|
||||
if command -v xfce4-panel >/dev/null 2>&1; then
|
||||
echo "Restarting panel with applet configs…"
|
||||
xfce4-panel -q >/dev/null 2>&1 || true
|
||||
sleep 1
|
||||
fi
|
||||
for id in 13 15 16; do
|
||||
cp -f "$NEXUS/management/panel/genmon-$id.rc" "$PANEL_CFG/genmon-$id.rc"
|
||||
done
|
||||
if command -v xfce4-panel >/dev/null 2>&1; then
|
||||
setsid xfce4-panel >/dev/null 2>&1 < /dev/null &
|
||||
sleep 1
|
||||
fi
|
||||
|
||||
# Start the popup daemon now so the first click works without a re-login.
|
||||
# Pin to the system python3 explicitly: PyGObject (gi) is a system package, and
|
||||
|
||||
+81
-4
@@ -19,6 +19,7 @@ A fresh machine clones first (git clone <repo> nexus-core), then runs this.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
@@ -118,6 +119,42 @@ def requirements() -> str:
|
||||
|
||||
# -- memory DB -----------------------------------------------------------------
|
||||
|
||||
def _vec0_extension() -> "Path | None":
|
||||
"""Path to the sqlite_vec native extension inside the venv, if built. Found
|
||||
by glob rather than importing sqlite_vec - this script runs stdlib-only,
|
||||
before the venv necessarily exists."""
|
||||
matches = list((ROOT / "Promethean").glob("**/sqlite_vec/vec0.*"))
|
||||
return matches[0] if matches else None
|
||||
|
||||
|
||||
_DERIVED = re.compile(
|
||||
r'^\s*(?:CREATE\s+(?:VIRTUAL\s+)?TABLE|INSERT\s+INTO)\s+"?vec_(?:messages|documents)',
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _is_derived_stmt(stmt: str) -> bool:
|
||||
"""True for a dump statement that builds or fills a sqlite_vec table.
|
||||
|
||||
Matched on the statement's TARGET, never as a substring: a chat message
|
||||
whose text happens to mention vec_messages is an INSERT INTO "messages" and
|
||||
has to survive. (It didn't, the first time - the test caught it.)
|
||||
"""
|
||||
if _DERIVED.match(stmt):
|
||||
return True
|
||||
# iterdump writes a virtual table straight into the schema table, and parks
|
||||
# its shadow tables' AUTOINCREMENT counters in sqlite_sequence. Both name the
|
||||
# vec table as a quoted VALUE rather than as the statement's target.
|
||||
head = stmt.lstrip().upper()
|
||||
if (head.startswith("INSERT INTO SQLITE_MASTER")
|
||||
or head.startswith('INSERT INTO "SQLITE_SEQUENCE"')) and "'vec_" in stmt:
|
||||
return True
|
||||
# Only meaningful when replaying over an existing DB, and sqlite_sequence
|
||||
# exists only if some AUTOINCREMENT table survives the filter - today true
|
||||
# by luck alone. Restore deletes the DB file first, so drop it.
|
||||
return stmt.lstrip().upper().startswith('DELETE FROM "SQLITE_SEQUENCE"')
|
||||
|
||||
|
||||
def dump_db() -> bool:
|
||||
"""Dump the (gitignored, binary, WAL) memory DB to a diff-friendly SQL file
|
||||
so git backs up the assistant's memory + conversation history. sqlite3 reads
|
||||
@@ -126,7 +163,27 @@ def dump_db() -> bool:
|
||||
return False
|
||||
try:
|
||||
with sqlite3.connect(f"file:{DB}?mode=ro", uri=True) as conn:
|
||||
DB_SQL.write_text("\n".join(conn.iterdump()) + "\n", encoding="utf-8")
|
||||
ext = _vec0_extension()
|
||||
if ext:
|
||||
try:
|
||||
conn.enable_load_extension(True)
|
||||
conn.load_extension(str(ext))
|
||||
conn.enable_load_extension(False)
|
||||
except (sqlite3.OperationalError, AttributeError):
|
||||
pass # iterdump below will fail with a clear "no such module" if this was needed
|
||||
# Drop the sqlite_vec virtual tables and their shadow tables.
|
||||
# iterdump() serializes a virtual table as a raw
|
||||
# INSERT INTO sqlite_master(...) followed by inserts into a table
|
||||
# the connection can't see yet, so replaying the dump dies with
|
||||
# "no such table: vec_messages" and leaves ZERO tables behind -
|
||||
# the whole backup was unrestorable. They're derived data anyway:
|
||||
# _backfill_vec_msgs()/_backfill_vec() rebuild both indexes from
|
||||
# message_vectors and documents on the next search.
|
||||
# Filter the generator, not the joined text - each yield is one
|
||||
# complete statement, while splitting the text on ";\n" tears
|
||||
# apart INSERTs whose content contains newlines.
|
||||
stmts = [s for s in conn.iterdump() if not _is_derived_stmt(s)]
|
||||
DB_SQL.write_text("\n".join(stmts) + "\n", encoding="utf-8")
|
||||
except sqlite3.Error as exc:
|
||||
print(f"Warning: could not dump {DB} ({exc}) - DB not captured")
|
||||
return False
|
||||
@@ -158,13 +215,17 @@ def _extra(a, b) -> int:
|
||||
'one box is simply ahead' from a real divergence."""
|
||||
a_conv, a_mem = a
|
||||
b_conv, b_mem = b
|
||||
newer = sum(1 for cid, ts in a_conv.items() if b_conv.get(cid, "") < ts)
|
||||
# Membership test first, never a placeholder default: updated_at is a REAL
|
||||
# column, so ANY typed default is a cross-type comparison against the one
|
||||
# case this function exists to count - a conversation the other side lacks.
|
||||
# ("" < 1778553309.83 raises TypeError; 0 only worked by accident.)
|
||||
newer = sum(1 for cid, ts in a_conv.items() if cid not in b_conv or b_conv[cid] < ts)
|
||||
return newer + len(a_mem - b_mem)
|
||||
|
||||
|
||||
def compare(db: Path = DB, dump: Path = DB_SQL) -> str:
|
||||
"""Which way the sync should go. One of: same, local-ahead, local-behind,
|
||||
diverged, no-live, no-dump.
|
||||
diverged, no-live, no-dump, unreadable.
|
||||
ponytail: detects direction, does not merge. Diverged is reported, not resolved."""
|
||||
if not dump.exists() or not dump.stat().st_size:
|
||||
return "no-dump"
|
||||
@@ -177,7 +238,11 @@ def compare(db: Path = DB, dump: Path = DB_SQL) -> str:
|
||||
dump_conn.executescript(dump.read_text(encoding="utf-8"))
|
||||
backup = _state(dump_conn)
|
||||
except (sqlite3.Error, OSError):
|
||||
return "diverged" # can't tell: fail safe, refuse both directions
|
||||
# A dump that won't replay is NOT a divergence. Reporting it as one
|
||||
# blocked backup and restore alike with a verdict that looked like a
|
||||
# real answer - dumps written before the vec-table filter above land
|
||||
# here every time. Still refuses both directions, but says why.
|
||||
return "unreadable"
|
||||
ahead, behind = _extra(live, backup), _extra(backup, live)
|
||||
if ahead and behind:
|
||||
return "diverged"
|
||||
@@ -208,6 +273,12 @@ def restore_db() -> None:
|
||||
if state == "no-dump":
|
||||
print("No memory dump in the backup - skipping DB restore.")
|
||||
return
|
||||
if state == "unreadable":
|
||||
print("WARNING: the backup's memory dump will not replay (written before the")
|
||||
print(" vec-table fix). Not applying it - it would leave an empty DB.")
|
||||
print(" Run `backup --force-db` on the machine with the good history")
|
||||
print(" to publish a clean dump, then restore here.")
|
||||
return
|
||||
|
||||
print("Restoring memory DB from backup...")
|
||||
rollback = DB.with_suffix(".db.pre-restore")
|
||||
@@ -318,6 +389,12 @@ def check_db_direction(tmp_dir: Path) -> bool:
|
||||
print(" the other lacks. Nothing merges these automatically.")
|
||||
print(" Force this box's history to win: python bin/sync.py backup --force-db")
|
||||
return False
|
||||
if state == "unreadable":
|
||||
print("REFUSING TO BACK UP: the remote memory dump will not replay, so there is")
|
||||
print(" no way to tell whether it holds history this machine lacks.")
|
||||
print(" If this box has the good history: python bin/sync.py backup --force-db")
|
||||
print(" (that publishes a clean dump and clears this for good).")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user