diff --git a/AGENTS.md b/AGENTS.md index a916734..177b437 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -493,6 +493,38 @@ substitutes a different source turns a hard error into a plausible wrong answer* byte that is off by four is invisible until something that matters lives in those four columns. Report the source, and test that the preferred path is actually taken. +### The screen buffers are found, not hardcoded + +The flag was `--frame-addr 0x200012BE --status-addr 0x2000163E` -- one build's +addresses, in the launcher, as a default. Pointed at another firmware that reads +somewhere else, the picture is plausible and wrong: measured, the build the user +actually flashed keeps its buffers at `0x2000129E` / `0x2000161E`, exactly 32 bytes +earlier, so every line landed 32 bytes off. That is what "the other firmware looks +shifted" was. + +Nothing needs to be assumed. The firmware images here are minimal ELFs -- one program +header, no section headers, no symbol table (tools/bin2elf.py writes them) -- so there +are no `gFrameBuffer` symbols to read, but there is behaviour: the firmware's own +buffers hold the same bytes the controller holds, because that is where the driver +copied them from. `tools/uvk5_buffers.py` slides the controller's memory through SRAM +and keeps the offset that agrees; it reported 1024/1024 bytes and the right pair of +addresses for the exact file the user flashed. + +So `--frame-addr` and `--status-addr` are optional now, `work/run-webui.ps1` no +longer passes them (or any machine-specific path), and the page reports what it found: + + buffers: {"frame": 0x2000129E, "status": 0x2000161E, "how": "sram search", + "score": 1024, "total": 1024} + +Two habits from this, both already in this file in other words: **a default that names +one machine's or one build's value is a bug waiting for a second build**, and **when +there are no symbols to read, ask the thing itself** -- the bytes in the buffers are +the answer, and they can be found by matching rather than guessed. + +The panel path needs none of this, and is what the page draws from: the controller's +memory is the screen for every firmware. The addresses only serve the guest-RAM +fallback, which is why a failed search is reported and does not stop anything. + ## The keypad: two real bugs, both fixed The old note here said "keys reach the firmware but the UI does not react" and diff --git a/AGENTS.zh-CN.md b/AGENTS.zh-CN.md index 3a4e4cf..524f368 100644 --- a/AGENTS.zh-CN.md +++ b/AGENTS.zh-CN.md @@ -400,6 +400,31 @@ PTT+SIDE1/SIDE2 与 MENU(那是应用的几个特殊模式)、开机窗口 错答案**;而一个偏了四列的字节,在"重要的东西恰好住在那四列里"之前,是看不出来的。要报出来源, 并且要测"该走的那条优选路径确实被走了"。 +### 屏幕缓冲是**找出来**的,不是写死的 + +原来是个默认值:`--frame-addr 0x200012BE --status-addr 0x2000163E` —— 那是**某一份构建**的地址, +被写进启动脚本当默认值。换成把缓冲放在别处的固件,画面就"貌似合理但错":实测用户真正刷进去的那份, +缓冲在 `0x2000129E` / `0x2000161E`,正好**早 32 字节**,于是**每一行都偏 32 字节**。这就是「换个 +固件就偏移」的本来面目。 + +其实什么都不用假设。这里的固件镜像是**最小 ELF** —— 一个程序头、没有节头、没有符号表 +(`tools/bin2elf.py` 就是这么写的)—— 所以**没有** `gFrameBuffer` 符号可读;但它有**行为**: +固件自己的缓冲里存着与控制器**相同的字节**,因为驱动就是从那拷贝过去的。 +`tools/uvk5_buffers.py` 把控制器显存沿着 SRAM 滑动,取吻合度最高的那个偏移;对用户刷的那份文件, +它给出 1024/1024 字节吻合和**正确**的一对地址。 + +于是 `--frame-addr` / `--status-addr` 现在都是可选的 ✓,`work/run-webui.ps1` 不再传它们 +(也不再有任何本机路径 ✓),页面会报出自己找到了什么: + + buffers: {"frame": 0x2000129E, "status": 0x2000161E, "how": "sram search", + "score": 1024, "total": 1024} + +两条习惯,这份文件里用别的话说过:**默认值里写进"某台机器/某份构建"的具体值,就是在等第二份构建来踩**; +以及 **没有符号可读时,就直接问它本人** —— 缓冲里的字节就是答案,靠**匹配**找出来,而不是猜。 + +面板路径完全不需要这些,而页面画的正是它:对任何固件,控制器的显存就是屏幕。这些地址只服务 +guest RAM 回落路径 —— 所以搜索失败会被报出来,而不会挡住任何东西。 + ## 键盘:两个真 bug,都已修复 这里原来的笔记写的是"按键到达了固件但界面不反应",并且归咎于机器模型。结果发现有**两个 diff --git a/tools/test_uvk5_buffers.py b/tools/test_uvk5_buffers.py new file mode 100644 index 0000000..a68f36f --- /dev/null +++ b/tools/test_uvk5_buffers.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""The buffer search must count *matches*, not just bytes compared. + +Its first version summed one per zipped pair and forgot `if a == b`, so every +offset scored the full total and the search always answered "the first offset" -- +which looked like a working discovery and pointed the fallback at whatever happened +to sit at the start of SRAM. These tests use synthetic memory, so they need no +emulator and fail on that code. +""" +import struct +import unittest + +import uvk5_buffers +from uvk5_lcd import FRAME_BYTES, STATUS_BYTES + + +def panel_memory(seed=1): + """A controller memory with a recognisable, non-repeating-ish pattern.""" + frame = bytes(((i * 13) + seed) & 0xFF for i in range(FRAME_BYTES)) + status = bytes(((i * 29) + seed) & 0xFF for i in range(STATUS_BYTES)) + return status + frame # the panel's order: the status page comes first + + +class TestLocate(unittest.TestCase): + def test_it_finds_the_buffers_where_they_are(self): + gram = panel_memory() + pad = 0x1234 + # In SRAM the frame comes first and the status line follows it. + sram = bytes(pad) + gram[STATUS_BYTES:] + gram[:STATUS_BYTES] + bytes(0x100) + + frame, status, best, total = uvk5_buffers.locate(sram, gram) + + self.assertEqual(frame, uvk5_buffers.SRAM_BASE + pad) + self.assertEqual(status, uvk5_buffers.SRAM_BASE + pad + FRAME_BYTES) + self.assertEqual(best, total, "a perfect match must score the whole window") + self.assertEqual(total, FRAME_BYTES + STATUS_BYTES) + + def test_a_single_match_does_not_win_by_being_first(self): + """The bug this guards: scoring pairs instead of matches picked offset 0.""" + gram = panel_memory() + pad = 0x0800 + sram = bytes(pad) + gram[STATUS_BYTES:] + gram[:STATUS_BYTES] + bytes(0x40) + _, _, best, total = uvk5_buffers.locate(sram, gram) + self.assertGreater(best, total * 0.9) + self.assertNotEqual(uvk5_buffers.score(sram, 0, gram), total, + "offset 0 is blank here and must not score full marks") + + def test_a_partly_stale_buffer_still_wins(self): + """The panel can be a frame ahead of the buffer it was copied from.""" + gram = panel_memory() + pad = 0x0400 + frame = bytearray(gram[STATUS_BYTES:]) + for i in range(0, 60): + frame[i] ^= 0x5A + sram = bytes(pad) + bytes(frame) + gram[:STATUS_BYTES] + bytes(0x80) + found, _, best, total = uvk5_buffers.locate(sram, gram) + self.assertEqual(found, uvk5_buffers.SRAM_BASE + pad) + # What matters is that the right offset wins, not that it is perfect: the + # panel can be a frame ahead of the buffer it was copied from. + runner_up = max(uvk5_buffers.score(sram, off, gram) + for off in range(0, len(sram) - total + 1) + if off != pad) + self.assertGreater(best, runner_up) + + +class TestSymbols(unittest.TestCase): + def test_a_minimal_elf_has_no_symbols_to_offer(self): + """tools/bin2elf.py writes program headers only, which is the usual case.""" + elf = bytearray(0x34 + 32 + 0x40) + elf[0:4] = b"\x7fELF" + elf[4] = 1 # 32-bit + elf[5] = 1 # little endian + struct.pack_into(" int: + """How many bytes at @offset match the controller's memory, frame then status.""" + if offset < 0 or offset + FRAME_BYTES + STATUS_BYTES > len(sram): + return -1 + frame = sram[offset:offset + FRAME_BYTES] + status = sram[offset + FRAME_BYTES:offset + FRAME_BYTES + STATUS_BYTES] + return (sum(1 for a, b in zip(frame, gram[STATUS_BYTES:]) if a == b) + + sum(1 for a, b in zip(status, gram[:STATUS_BYTES]) if a == b)) + + +def locate(sram: bytes, gram: bytes): + """(frame address, status address, matching bytes, total) for the best offset.""" + total = FRAME_BYTES + STATUS_BYTES + best, best_at = -1, None + for offset in range(0, len(sram) - total + 1): + value = score(sram, offset, gram) + if value > best: + best, best_at = value, offset + if best_at is None: + return None, None, 0, total + return SRAM_BASE + best_at, SRAM_BASE + best_at + FRAME_BYTES, best, total + + +def from_symbols(path: str): + """(frame, status) from a real ELF symbol table, or (None, None). + + Kept because it is authoritative when it works -- a fully linked ELF (the CW + timing build, for instance) does name these buffers. The images this project + usually runs do not. + """ + try: + data = open(path, "rb").read() + except OSError: + return None, None + if len(data) < 52 or data[0] != 0x7F or data[1:4] != b"ELF" or data[4] != 1: + return None, None + e_shoff, = struct.unpack_from(" bytes: + """Read the whole of SRAM through QMP memsave.""" + path = os.path.join(tempfile.mkdtemp(prefix="uvk5-buffers-"), "sram.bin") + client.command("memsave", val=SRAM_BASE, size=SRAM_SIZE, filename=path) + with open(path, "rb") as fh: + return fh.read() + + +def discover(client, gram: bytes = None, image_path: str = None) -> dict: + """{frame, status, how, score, total} -- symbols first, then the search.""" + if image_path: + frame, status = from_symbols(image_path) + if frame: + return dict(frame=frame, status=status, how="elf symbols", + score=None, total=None) + if client is None: + return dict(frame=None, status=None, how="no emulator", score=0, total=0) + gram = gram if gram is not None else uvk5_lcd.FrameGrabber(client, 0, 0).panel_gram() + frame, status, best, total = locate(sram_from(client), gram) + how = "sram search" if best >= CONFIDENT * total else "sram search (weak match)" + return dict(frame=frame, status=status, how=how, score=best, total=total) + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--qmp", default="127.0.0.1:4444") + ap.add_argument("--image", help="firmware file, to try its symbols first") + args = ap.parse_args(argv) + + from uvk5_qmp import QmpClient + try: + client = QmpClient(args.qmp, timeout=20) + except Exception as exc: + print("cannot reach QMP at %s: %s" % (args.qmp, str(exc).splitlines()[0]), file=sys.stderr) + print("if the web UI is running it holds the one QMP client", file=sys.stderr) + return 2 + found = discover(client, image_path=args.image) + print("frame 0x%08X" % found["frame"] if found["frame"] else "frame (not found)") + print("status 0x%08X" % found["status"] if found["status"] else "status (not found)") + print("how %s" % found["how"]) + if found["score"] is not None: + print("match %d/%d bytes" % (found["score"], found["total"])) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/uvk5_stream.py b/tools/uvk5_stream.py index 9d1bcaa..42a9c28 100644 --- a/tools/uvk5_stream.py +++ b/tools/uvk5_stream.py @@ -82,6 +82,21 @@ class FramePump: self._raw = None self._generation += 1 + def set_buffers(self, frame_addr: int, status_addr: int): + """Point the guest-RAM fallback at this firmware's buffers. + + The addresses move between builds, so they are discovered from the firmware + itself (tools/uvk5_buffers.py) rather than hardcoded. Only the fallback uses + them: the panel path needs none. + """ + with self._lock: + self._frame_addr = frame_addr + self._status_addr = status_addr + grabber = self._grabber + if grabber is not None: + self._grabber = FrameGrabber(grabber._client, frame_addr, status_addr, + self._spool_dir) + def _run(self): while not self._stop.is_set(): started = time.monotonic() diff --git a/tools/uvk5_testenv.py b/tools/uvk5_testenv.py index a15553a..ab4d92c 100644 --- a/tools/uvk5_testenv.py +++ b/tools/uvk5_testenv.py @@ -36,7 +36,16 @@ def qemu(): if value and os.path.exists(value): return pathlib.Path(value) found = shutil.which("qemu-system-arm") - return pathlib.Path(found) if found else None + if found: + return pathlib.Path(found) + # The sibling tree tools/setup_qemu.sh builds into, which is also what run_tests.sh + # assumes. Not a machine-specific path: it is this checkout's own convention, so a + # fresh clone that followed the README finds its QEMU without being told. + sibling = pathlib.Path(__file__).resolve().parent.parent.parent / "qemu-7.2" / "build" + for name in ("qemu-system-arm", "qemu-system-arm.exe"): + if (sibling / name).exists(): + return sibling / name + return None def gdb(): diff --git a/tools/webui.py b/tools/webui.py index 4a99f1b..e01b3d7 100644 --- a/tools/webui.py +++ b/tools/webui.py @@ -144,7 +144,7 @@ def image_has_multiboot(path): return any(marker in blob for marker in MULTIBOOT_MARKERS) -def create_app(client, frame_addr: int, status_addr: int, scale: int = 4, +def create_app(client, frame_addr: int = None, status_addr: int = None, scale: int = 4, supervisor=None, log=None, image=None, boot_key=None, flash=None): app = Flask(__name__) @@ -154,6 +154,46 @@ def create_app(client, frame_addr: int, status_addr: int, scale: int = 4, # One background grabber for every client. client may be None: the emulator # can be powered off, and the page still has to load. + told_addresses = frame_addr is not None and status_addr is not None + frame_addr = frame_addr or 0 + status_addr = status_addr or 0 + buffers = {"info": None} + + def ensure_buffers(): + """Ask the firmware where it keeps its screen, instead of being told. + + The addresses move between builds, and passing one build's for another is how + the page ended up drawing a picture that was plausible and offset. The panel + path needs no addresses at all, so this concerns only the guest-RAM fallback: + it stays unset, and says so, rather than guessing. A --frame-addr on the + command line skips the search and is trusted. + """ + if told_addresses: + return {"frame": frame_addr, "status": status_addr, "how": "command line"} + if buffers["info"] is not None: + return buffers["info"] + target = active_client() + if target is None: + return None + try: + import uvk5_buffers + path = image.current.path if image is not None and image.current else None + info = uvk5_buffers.discover(target, image_path=path) + except Exception as exc: + info = {"frame": None, "status": None, + "how": "discovery failed: %s" % exc, "score": 0, "total": 0} + buffers["info"] = info + if info.get("frame"): + pump.set_buffers(info["frame"], info["status"]) + log.add("qemu", "screen buffers read from the firmware: frame 0x%08X, " + "status 0x%08X (%s; %s/%s bytes agree)" + % (info["frame"], info["status"], info["how"], + info["score"], info["total"])) + else: + log.add("qemu", "no screen buffers found for this firmware (%s); the panel " + "path does not need them" % info["how"]) + return info + pump = FramePump(client, frame_addr, status_addr, fps=TARGET_FPS, scale=scale, on_fallback=lambda note: log.add( "qemu", "panel unavailable, drawing from guest RAM at " @@ -274,7 +314,7 @@ def create_app(client, frame_addr: int, status_addr: int, scale: int = 4, return jsonify(powered=False, status="unreachable", error=str(exc)) return jsonify(powered=True, speaker=speaker_on(), panel=panel_state(), firmware=firmware_info(), - frame_source=pump.source()[0], **info) + frame_source=pump.source()[0], buffers=ensure_buffers(), **info) @app.get("/api/panel") def api_panel(): @@ -288,11 +328,11 @@ def create_app(client, frame_addr: int, status_addr: int, scale: int = 4, can be compared without guessing. """ source, note = pump.source() - body = {"source": source, "note": note, + body = {"source": source, "note": note, "buffers": ensure_buffers(), "frame_addr": frame_addr, "status_addr": status_addr} target = active_client() if target is None: - return jsonify(body, powered=False) + return jsonify(dict(body, powered=False)) try: body["gram"] = target.command("qom-get", path=PANEL_PATH, property="gram") body["invert"] = bool(target.command("qom-get", path=PANEL_PATH, @@ -1220,10 +1260,14 @@ def _default_firmware(): def main() -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("--qmp", default="/tmp/uvk5-qmp.sock") - ap.add_argument("--frame-addr", type=lambda v: int(v, 0), required=True, - help="address of gFrameBuffer (moves between builds)") - ap.add_argument("--status-addr", type=lambda v: int(v, 0), required=True, - help="address of gStatusLine") + # Optional, and normally omitted: the addresses move between builds, so the page + # finds them by asking the firmware (tools/uvk5_buffers.py). They are only needed + # by the guest-RAM fallback -- the panel path, which is what the page uses, needs + # no addresses at all. Passing one skips discovery and trusts the value. + ap.add_argument("--frame-addr", type=lambda v: int(v, 0), default=None, + help="address of gFrameBuffer; omit to find it in the firmware") + ap.add_argument("--status-addr", type=lambda v: int(v, 0), default=None, + help="address of gStatusLine; omit to find it in the firmware") ap.add_argument("--host", default="127.0.0.1") ap.add_argument("--port", type=int, default=8080) ap.add_argument("--scale", type=int, default=4) diff --git a/work/run-webui.ps1 b/work/run-webui.ps1 index b0b5ab6..29b33f6 100644 --- a/work/run-webui.ps1 +++ b/work/run-webui.ps1 @@ -3,6 +3,11 @@ # powershell -File work\run-webui.ps1 # the page starts the emulator # powershell -File work\run-webui.ps1 -Attach # attach to run-emulator.ps1 # +# Nothing here is machine-specific. The QEMU binary comes from QEMU or PATH, the +# interpreter is the one on PATH, the firmware is whatever the checkout has (or one +# uploaded from the page), the flash image is the last one a session used, and the +# screen-buffer addresses are read out of the running firmware rather than passed in. +# # Owning the process is what puts the firmware's serial output in the page's log # pane: the model prints it to stderr as "SERIAL ..." and the server reads QEMU's # stderr. With -Attach the server never sees that stream, so the pane stays empty. @@ -10,28 +15,52 @@ param( [int]$Port = 8080, [int]$QmpPort = 4444, - [string]$Frame = '0x200012BE', # gFrameBuffer, proven against the firmware source - [string]$Status = '0x2000163E', # gStatusLine - [string]$Kernel = "$PSScriptRoot\f4hwn\EGZUMER+F4HWN-v5.9.0.CN.elf", - [string]$Flash = [System.IO.Path]::GetFullPath("$PSScriptRoot\..\assets\flash.img"), - [string]$Qemu = 'F:\dsh-build\qemu-7.2.0\build\qemu-system-arm.exe', + [string]$Frame = '', # only to override the discovered gFrameBuffer + [string]$Status = '', # only to override the discovered gStatusLine + [string]$Kernel = '', # empty: whatever the checkout has, or upload one + [string]$Flash = '', # empty: the last image a session used + [string]$Qemu = '', # empty: QEMU or PATH [switch]$Attach ) $ErrorActionPreference = 'Stop' +Set-Location (Join-Path $PSScriptRoot '..') + # The QEMU this server spawns is a native Windows build, so the MSYS2 mingw64 DLLs # have to be on the child's PATH -- inherited from here, since the child gets this # environment. Without it QEMU dies at load and "power on" reports only that the QMP # port never appeared. -$env:PATH = 'F:\msys64\mingw64\bin;' + $env:PATH -$py = 'C:\Users\Administrator\.dsh\dsh-runtimes\dsh-primary-runtime\dependencies\python\python.exe' -Set-Location (Join-Path $PSScriptRoot '..') +if (Test-Path 'F:\msys64\mingw64\bin') { + $env:PATH = 'F:\msys64\mingw64\bin;' + $env:PATH +} + +$py = if ($env:PYTHON) { $env:PYTHON } else { + $found = Get-Command python -ErrorAction SilentlyContinue + if (-not $found) { $found = Get-Command py -ErrorAction SilentlyContinue } + if (-not $found) { throw 'no python on PATH; set PYTHON to one with flask installed' } + $found.Source +} + +$common = @('tools\webui.py', '--qmp', "127.0.0.1:$QmpPort", '--port', $Port) +if ($Frame) { $common += @('--frame-addr', $Frame) } +if ($Status) { $common += @('--status-addr', $Status) } + +if (-not $Kernel -and $env:ELF) { $Kernel = $env:ELF } +if (-not $Flash) { + $candidates = @($env:UVK5_FLASH_IMAGE, 'work\user-flash.img', 'assets\flash.img') + foreach ($candidate in $candidates) { + if ($candidate -and (Test-Path $candidate)) { $Flash = $candidate; break } + } +} +if (-not $Qemu -and $env:QEMU) { $Qemu = $env:QEMU } -$common = @('tools\webui.py', '--qmp', "127.0.0.1:$QmpPort", - '--frame-addr', $Frame, '--status-addr', $Status, '--port', $Port) if ($Attach) { Write-Host "attaching to an emulator already listening on 127.0.0.1:$QmpPort" & $py @common --attach } else { - Write-Host "the page will start: $Qemu" - & $py @common --qemu $Qemu --elf $Kernel --flash $Flash + if ($Qemu) { $common += @('--qemu', $Qemu) } + if ($Kernel) { $common += @('--elf', $Kernel) } + if ($Flash) { $common += @('--flash', $Flash) } + Write-Host "the page will start: $(if ($Qemu) { $Qemu } else { 'qemu-system-arm from PATH' })" + if ($Flash) { Write-Host "flash image: $Flash" } + & $py @common }