Overlay apps: the region, the format, and a tool that writes them

The Labs edition runs overlay apps (Tetris, Breakout, Plasma, Cube3D, Beam, Beacon, FoxHunt, BroadcastFM) that upstream UVStudio installs over WebSerial. This page owns the flash image, so the same bytes go to the same offsets with no serial protocol and no browser permission: APP_REGION_BASE 0x102000, APP_SLOT_STRIDE 0x2000, APP_CODE_OFFSET 0x1000, 16 slots, taken from the firmware's own App/apps/app_overlay.h rather than inferred.

tools/uvk5_apps.py parses and validates the 64-byte FAP1 header (zlib CRC-32 over the code, vma 0x20000280, name, version, capabilities), lists, installs and erases slots, and refuses what the firmware would show as APP ERROR. test_uvk5_apps covers those refusals plus install/erase/list round trips, and parses a real upstream Beam.app when one has been downloaded. The header struct was 60 bytes at first -- a missing vma field -- which the real file's bytes showed at once.

AGENTS.md

@@ -550,6 +550,27 @@ and still starts before anything waits on QEMU. **A rewrite that preserves the p
 were fixing while losing another is the expensive kind**, and this one hid because the log
 still "worked" for the binary screen stream.
 
+### Overlay apps are a flash region, and the page can write it
+
+The Labs edition runs small overlay apps (Tetris, Breakout, Plasma, Cube3D, Beam, Beacon,
+FoxHunt, BroadcastFM). Upstream installs them from UVStudio over WebSerial; here they are
+bytes in the external flash image, which the page already owns -- so no serial protocol and
+no browser permission are involved, just the same bytes at the same offsets.
+
+The layout is the firmware's own, read out of the header it compiles rather than inferred
+(`App/apps/app_overlay.h`): `APP_REGION_BASE 0x00102000` -- right behind the two state
+markers -- `APP_SLOT_STRIDE 0x2000`, `APP_CODE_OFFSET 0x1000`, 16 slots. A slot holds a
+64-byte `app_header_t` (magic `FAP1`, zlib CRC-32 over the code, name, version, vma
+0x20000280, capabilities) and its code one 4 KiB sector later. `tools/uvk5_apps.py` parses,
+validates, lists, installs and erases them; `tools/test_uvk5_apps.py` covers the refusals.
+
+Two things are worth knowing. **The 64-byte header is shared with the multiboot slots**:
+`FMB1` means firmware, `FAP1` means app, which is why "install app to slot 1" and "put a
+firmware in slot 1" touch the same external flash and the power-on menu lists both. And the
+region is found the same way the screen buffers were: from the firmware's own constants,
+not from a guess -- the first version of the header here was 60 bytes because a field
+(`vma`) was missing, and the real `Beam.app` bytes said so immediately.
+
 ## The keypad: two real bugs, both fixed
 
 The old note here said "keys reach the firmware but the UI does not react" and

AGENTS.zh-CN.md

@@ -444,6 +444,24 @@ matches_uploaded, note}`，页面显示 `device reports: ...`，**只在"设备
 QEMU 之前就开始排空**。**一次既保住你要修的性质、又悄悄丢掉另一种性质的改写，是最贵的那种** ——
 而这次它藏得住，是因为对那条二进制屏幕流来说，日志"看起来还是好的"。
 
+### 叠加应用就是外部 flash 里的一块区域，页面可以直接写
+
+Labs 版能跑小的叠加应用（Tetris、Breakout、Plasma、Cube3D、Beam、Beacon、FoxHunt、
+BroadcastFM）。上游是用 UVStudio 通过 WebSerial 装进去的；在我们这里，它们就是**外部 flash 镜像
+里的字节** —— 而镜像本来就在页面手里，所以既不需要串口协议，也不需要浏览器授权，只是**同样的字节
+写在同样的偏移**。
+
+布局是固件自己的，从它编译的头文件读出（`App/apps/app_overlay.h`），不是猜的：
+`APP_REGION_BASE 0x00102000`（紧跟在两个状态标记之后）、`APP_SLOT_STRIDE 0x2000`、
+`APP_CODE_OFFSET 0x1000`、16 个槽。每个槽放一个 64 字节的 `app_header_t`（magic `FAP1`、
+对代码做 zlib CRC-32、名字、版本、vma 0x20000280、capabilities），代码在**下一个 4 KiB 扇区**。
+`tools/uvk5_apps.py` 负责解析、校验、列出、安装、擦除；`tools/test_uvk5_apps.py` 管各种拒绝。
+
+两点值得知道。**这 64 字节头和多系统槽位是共用的**：`FMB1` 是固件、`FAP1` 是应用 ——
+这就是为什么"把应用装进槽 1"和"把固件放进槽 1"动的是同一块外部 flash，开机菜单也把两者列在一起。
+另外，这块区域和屏幕缓冲一样，是**从固件自己的常量里读出来的**而非猜的：这里头结构的第一版是 60 字节，
+因为漏了一个字段（`vma`），而真实的 `Beam.app` 字节当场就指出了这一点。
+
 ## 键盘：两个真 bug，都已修复
 
 这里原来的笔记写的是"按键到达了固件但界面不反应"，并且归咎于机器模型。结果发现有**两个

tools/test_uvk5_apps.py

@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+"""The app installer must refuse anything the firmware would show as APP ERROR.
+
+The format is the firmware's own (App/apps/app_overlay.h): a 64-byte FAP1 header with a
+zlib CRC-32 over the code, code at slot_base + 0x1000, 16 slots of 8 KiB from 0x102000.
+These tests build blobs here, so they need no emulator and no third-party binary.
+"""
+import os
+import struct
+import unittest
+import zlib
+
+import uvk5_apps as A
+
+
+def image(size=2 * 1024 * 1024, fill=0xFF):
+    return bytearray([fill]) * size
+
+
+class TestFormat(unittest.TestCase):
+    def test_a_built_blob_round_trips(self):
+        blob = A.build(b"\x00\x01\x02\x03" * 8, "Beam", "1.0", shortcut="beam")
+        info = A.parse(blob)
+        self.assertEqual(info["name"], "Beam")
+        self.assertEqual(info["version"], "1.0")
+        self.assertEqual(info["code_size"], 32)
+        self.assertEqual(info["shortcut"], "beam")
+        self.assertTrue(info["committed"])
+        self.assertEqual(info["vma"], A.OVERLAY_VMA)      # 0x20000280, in SRAM
+        self.assertEqual(info["capabilities"], 0)        # build() sets none by default
+
+    def test_the_magic_is_the_one_the_firmware_defines(self):
+        self.assertEqual(A.MAGIC.to_bytes(4, "little"), b"FAP1")
+        self.assertEqual(A.REGION_BASE, 0x00102000)
+        self.assertEqual(A.SLOT_STRIDE, 0x2000)
+        self.assertEqual(A.SLOT_COUNT, 16)
+        self.assertEqual(A.build(b"x", "A")[:4], b"FAP1")
+        self.assertEqual(A.HDR_SIZE, 64, "the header is 64 bytes in app_overlay.h")
+        self.assertEqual(A.OVERLAY_VMA, 0x20000280, "the overlay runs from SRAM")
+
+    def test_a_bad_magic_is_refused(self):
+        blob = bytearray(A.build(b"code", "A"))
+        blob[0:4] = b"FMB1"
+        with self.assertRaises(A.AppError):
+            A.parse(bytes(blob))
+
+    def test_a_crc_that_does_not_match_the_code_is_refused(self):
+        blob = bytearray(A.build(b"code", "A"))
+        blob[A.HDR_SIZE] ^= 0xFF          # change the code, leave the CRC
+        with self.assertRaises(A.AppError) as caught:
+            A.parse(bytes(blob))
+        self.assertIn("CRC-32", str(caught.exception))
+
+    def test_code_over_the_overlay_budget_is_refused(self):
+        with self.assertRaises(A.AppError):
+            A.build(b"x" * (A.APP_OVERLAY_MAX + 1), "Big")
+
+    def test_a_truncated_blob_is_refused(self):
+        blob = A.build(b"x" * 100, "A")
+        with self.assertRaises(A.AppError):
+            A.parse(blob[:A.HDR_SIZE + 50])
+
+    def test_capabilities_round_trip(self):
+        """An app may require a resident facility; it lives in the header's spare bytes."""
+        blob = A.build(b"code", "FM", capabilities=0x01)
+        self.assertEqual(A.parse(blob)["capabilities"], 0x01)
+
+    def test_an_unknown_shortcut_is_refused(self):
+        with self.assertRaises(A.AppError):
+            A.build(b"x", "A", shortcut="solitaire")
+
+
+class TestInstall(unittest.TestCase):
+    def test_install_puts_the_header_and_the_code_where_the_firmware_looks(self):
+        img = image()
+        blob = A.build(b"\xAA" * 200, "Tetris", "1.0")
+        info = A.install(img, 1, blob)
+        base = A.REGION_BASE + A.SLOT_STRIDE
+        self.assertEqual(bytes(img[base:base + 4]), b"FAP1")
+        self.assertEqual(bytes(img[base + A.CODE_OFFSET:base + A.CODE_OFFSET + 4]), b"\xAA" * 4)
+        # the gap a real erase leaves is 0xFF, not zeroes
+        self.assertEqual(bytes(img[base + 64:base + A.CODE_OFFSET]), b"\xFF" * (A.CODE_OFFSET - 64))
+        self.assertEqual(info["name"], "Tetris")
+
+    def test_install_erases_what_was_there_before(self):
+        img = image()
+        A.install(img, 3, A.build(b"\x11" * 400, "Old", "1.0"))
+        A.install(img, 3, A.build(b"\x22" * 100, "New", "1.0"))
+        base = A.REGION_BASE + 3 * A.SLOT_STRIDE
+        self.assertNotIn(b"\x11", bytes(img[base:base + A.SLOT_STRIDE]))
+        self.assertEqual(A.read_slot(bytes(img), 3)["name"], "New")
+
+    def test_listing_reports_the_slot_and_the_name(self):
+        img = image()
+        A.install(img, 0, A.build(b"\x01" * 64, "Beam"))
+        A.install(img, 15, A.build(b"\x02" * 64, "Plasma"))
+        found = A.list_apps(bytes(img))
+        self.assertEqual([(a["slot"], a["name"]) for a in found], [(0, "Beam"), (15, "Plasma")])
+
+    def test_an_empty_slot_is_not_an_app(self):
+        self.assertEqual(A.list_apps(bytes(image())), [])
+
+    def test_a_firmware_in_the_same_slot_is_named_as_one(self):
+        """The 64-byte headers are shared: FMB1 is a firmware, FAP1 is an app."""
+        img = image()
+        base = A.REGION_BASE
+        img[base:base + 4] = b"FMB1"
+        img[base + 8:base + 12] = struct.pack("<I", 114 * 1024)
+        found = A.list_apps(bytes(img))
+        self.assertEqual(found[0]["kind"], "firmware")
+
+    def test_erase_clears_the_slot(self):
+        img = image()
+        A.install(img, 2, A.build(b"\x33" * 64, "Gone"))
+        A.erase(img, 2)
+        self.assertIsNone(A.read_slot(bytes(img), 2))
+
+    def test_a_slot_outside_the_region_is_refused(self):
+        with self.assertRaises(A.AppError):
+            A.install(image(), 16, A.build(b"x", "A"))
+
+
+class TestRealFile(unittest.TestCase):
+    """Runs against a downloaded Beam.app when one is present; skipped otherwise."""
+
+    def test_a_real_upstream_app_parses(self):
+        path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "work", "apps", "Beam.app")
+        if not os.path.exists(path):
+            self.skipTest("no work/apps/Beam.app; download one from the upstream archive")
+        with open(path, "rb") as fh:
+            info = A.parse(fh.read())
+        self.assertEqual(info["name"], "Beam")
+        self.assertEqual(info["shortcut"], "beam")
+        self.assertTrue(info["committed"])
+
+
+if __name__ == "__main__":
+    unittest.main()

tools/uvk5_apps.py

@@ -0,0 +1,262 @@
+#!/usr/bin/env python3
+"""Overlay apps: where they live in the external flash, and what a .app contains.
+
+The Labs edition of the F4HWN firmware runs small "overlay" apps -- Tetris, Breakout,
+Plasma, Cube3D, Beam, Beacon, FoxHunt, BroadcastFM -- which the upstream UVStudio page
+installs over WebSerial. This tool does the same thing to a flash **image**, which is
+what the browser page in this repository owns: no serial protocol, no browser permission,
+the same bytes at the same offsets.
+
+The layout is the firmware's own, read out of the header it compiles rather than inferred
+(`App/apps/app_overlay.h`):
+
+    APP_REGION_BASE  0x00102000   first app slot, right behind the two state markers
+    APP_SLOT_STRIDE  0x00002000   8 KiB per slot
+    APP_CODE_OFFSET  0x00001000   code starts after the slot's 4 KiB header sector
+    APP_SLOT_COUNT   16
+
+So slot *n* holds a 64-byte header at `base + n*0x2000` and its code at
+`base + n*0x2000 + 0x1000`. The header is `app_header_t`, little-endian and packed,
+with magic `FAP1` and a zlib CRC-32 over the code -- the same CRC the multiboot loader
+uses to validate a slot before offering RUN, which is why this tool checks it too.
+
+The 64-byte header is shared with the multiboot slots: a slot whose header says `FMB1`
+is a firmware, one that says `FAP1` is an app. That is how the power-on menu can list
+both in the same four rows, and why "install to slot N" in UVStudio and "put a firmware
+in slot N" in this page's slot table touch the same external flash.
+
+    tools/uvk5_apps.py list    work/user-flash.img
+    tools/uvk5_apps.py install work/user-flash.img 1 Beam.app
+    tools/uvk5_apps.py erase   work/user-flash.img 1
+"""
+import argparse
+import struct
+import sys
+import zlib
+
+MAGIC = 0x31504146           # "FAP1"
+HDR_VERSION = 1
+APP_OVERLAY_MAX = 0x1000     # 4 KiB of code per app
+NAME_LEN = 16
+VERSION_LEN = 16
+HDR_SIZE = 64
+
+REGION_BASE = 0x00102000
+SLOT_STRIDE = 0x0002000
+CODE_OFFSET = 0x00001000
+SLOT_COUNT = 16
+
+FLAG_COMMITTED = 0x0001
+FLAG_SCREEN_SAVER = 0x0002
+FLAG_SHORTCUT_MASK = 0x0F00
+FLAG_SHORTCUT_SHIFT = 8
+
+SHORTCUTS = {0x01: "fm", 0x02: "foxhunt", 0x04: "beacon", 0x08: "beam"}
+
+# magic | hdr_version | abi | api_min | code_size | crc32 | entry_off | flags | name | ver
+#   | vma | capabilities | reserved
+#
+# vma is where the overlay is loaded and run: 0x20000280, inside SRAM, which is what the
+# packer passes as --vma and what a real Beam.app carries at offset 52. Getting this field
+# wrong is how the header came out 60 bytes instead of 64 in the first version here.
+OVERLAY_VMA = 0x20000280
+_HEADER = struct.Struct("<IHBBIIHH16s16sIII")
+assert _HEADER.size == HDR_SIZE, _HEADER.size
+
+
+class AppError(Exception):
+    """A blob that is not an app this firmware could run."""
+
+
+def slot_base(slot: int) -> int:
+    if not 0 <= slot < SLOT_COUNT:
+        raise AppError("slot %s is outside 0..%d" % (slot, SLOT_COUNT - 1))
+    return REGION_BASE + slot * SLOT_STRIDE
+
+
+def _text(raw: bytes) -> str:
+    return raw.split(b"\x00", 1)[0].decode("ascii", "replace").strip()
+
+
+def parse(blob: bytes, strict: bool = True) -> dict:
+    """The 64-byte header of a .app blob, validated.
+
+    Strict by default: a bad magic, an oversized code section or a CRC that does not
+    match the code means the firmware would show APP ERROR, so it is refused here where
+    the reason can be said out loud.
+    """
+    if len(blob) < HDR_SIZE:
+        raise AppError("only %d bytes: too short to hold the 64-byte header" % len(blob))
+    (magic, hdr_version, abi, api_min, code_size, crc32, entry_off, flags,
+     name, version, vma, cap, reserved) = _HEADER.unpack_from(blob, 0)
+    if magic != MAGIC:
+        raise AppError("magic is %r, not FAP1 -- this is not an app blob"
+                       % blob[:4].decode("latin1", "replace"))
+    info = dict(magic=magic, hdr_version=hdr_version, abi=abi, api_min=api_min,
+                code_size=code_size, crc32=crc32, entry_off=entry_off, flags=flags,
+                name=_text(name), version=_text(version), vma=vma, capabilities=cap,
+                committed=bool(flags & FLAG_COMMITTED),
+                screen_saver=bool(flags & FLAG_SCREEN_SAVER),
+                shortcut=SHORTCUTS.get((flags & FLAG_SHORTCUT_MASK) >> FLAG_SHORTCUT_SHIFT,
+                                       "none"),
+                total=HDR_SIZE + code_size)
+    if not strict:
+        return info
+    if hdr_version != HDR_VERSION:
+        raise AppError("header version %d, this firmware writes %d" % (hdr_version, HDR_VERSION))
+    if code_size == 0 or code_size > APP_OVERLAY_MAX:
+        raise AppError("code_size %d is outside 1..%d" % (code_size, APP_OVERLAY_MAX))
+    if len(blob) < HDR_SIZE + code_size:
+        raise AppError("header says %d bytes of code but the file holds %d"
+                       % (code_size, len(blob) - HDR_SIZE))
+    actual = zlib.crc32(blob[HDR_SIZE:HDR_SIZE + code_size]) & 0xFFFFFFFF
+    if actual != crc32:
+        raise AppError("CRC-32 over the code is 0x%08X but the header says 0x%08X"
+                       % (actual, crc32))
+    if vma != OVERLAY_VMA:
+        raise AppError("vma is 0x%08X; this firmware loads overlays at 0x%08X"
+                       % (vma, OVERLAY_VMA))
+    return info
+
+
+def build(code: bytes, name: str, version: str = "1.0", abi: int = 1, api_min: int = 1,
+          shortcut: str = "none", flags: int = FLAG_COMMITTED,
+          capabilities: int = 0) -> bytes:
+    """A .app blob around @code. The packer's inverse, for tests and for repacking."""
+    if len(code) > APP_OVERLAY_MAX:
+        raise AppError("code is %d bytes; the overlay budget is %d" % (len(code), APP_OVERLAY_MAX))
+    for label, text in (("name", name), ("version", version)):
+        if not text or len(text) >= (NAME_LEN if label == "name" else VERSION_LEN):
+            raise AppError("%s must be 1..%d characters" % (label, NAME_LEN - 1))
+    if shortcut != "none":
+        matches = [k for k, v in SHORTCUTS.items() if v == shortcut]
+        if not matches:
+            raise AppError("unknown shortcut %r; known: %s"
+                           % (shortcut, ", ".join(sorted(SHORTCUTS.values()))))
+        flags |= matches[0] << FLAG_SHORTCUT_SHIFT
+    header = _HEADER.pack(MAGIC, HDR_VERSION, abi, api_min, len(code),
+                          zlib.crc32(code) & 0xFFFFFFFF, 0, flags,
+                          name.encode("ascii")[:NAME_LEN - 1].ljust(NAME_LEN, b"\x00"),
+                          version.encode("ascii")[:VERSION_LEN - 1].ljust(VERSION_LEN, b"\x00"),
+                          OVERLAY_VMA, capabilities, 0)
+    return header + code
+
+
+def read_slot(image: bytes, slot: int) -> dict:
+    """What slot @slot holds, or None when its header is not an app."""
+    base = slot_base(slot)
+    if base + SLOT_STRIDE > len(image):
+        return None
+    raw = image[base:base + SLOT_STRIDE]
+    if raw[:4] == b"\xff\xff\xff\xff" or not any(raw):
+        return None
+    if raw[:4] != b"FAP1":
+        return dict(slot=slot, base=base, magic=raw[:4].decode("latin1", "replace"),
+                    kind="firmware" if raw[:4] == b"FMB1" else "unknown")
+    try:
+        info = parse(raw[:HDR_SIZE] + raw[CODE_OFFSET:CODE_OFFSET + struct.unpack_from("<I", raw, 8)[0]],
+                     strict=False)
+    except AppError:
+        return dict(slot=slot, base=base, magic="FAP1", kind="app (unreadable header)")
+    info.update(slot=slot, base=base, kind="app")
+    return info
+
+
+def list_apps(image: bytes):
+    return [info for info in (read_slot(image, i) for i in range(SLOT_COUNT)) if info]
+
+
+def install(image: bytearray, slot: int, blob: bytes) -> dict:
+    """Write @blob into @slot: header at the base, code at +0x1000, rest erased.
+
+    The header sector is erased first, the way the flash would be: an install must not
+    leave a byte of the previous app behind for the loader to trip over.
+    """
+    info = parse(blob)
+    base = slot_base(slot)
+    if base + SLOT_STRIDE > len(image):
+        raise AppError("the image is too small for slot %d" % slot)
+    for i in range(base, base + SLOT_STRIDE):
+        image[i] = 0xFF
+    image[base:base + HDR_SIZE] = blob[:HDR_SIZE]
+    code = blob[HDR_SIZE:HDR_SIZE + info["code_size"]]
+    image[base + CODE_OFFSET:base + CODE_OFFSET + len(code)] = code
+    return info
+
+
+def erase(image: bytearray, slot: int) -> int:
+    base = slot_base(slot)
+    if base + SLOT_STRIDE > len(image):
+        raise AppError("the image is too small for slot %d" % slot)
+    for i in range(base, base + SLOT_STRIDE):
+        image[i] = 0xFF
+    return base
+
+
+def _read(path: str) -> bytearray:
+    with open(path, "rb") as fh:
+        return bytearray(fh.read())
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    p = sub.add_parser("list", help="what each app slot holds")
+    p.add_argument("image")
+    p = sub.add_parser("install", help="write a .app into a slot")
+    p.add_argument("image")
+    p.add_argument("slot", type=int)
+    p.add_argument("app")
+    p = sub.add_parser("erase", help="clear a slot")
+    p.add_argument("image")
+    p.add_argument("slot", type=int)
+    p = sub.add_parser("info", help="describe a .app file without installing it")
+    p.add_argument("app")
+    args = ap.parse_args(argv)
+
+    try:
+        if args.cmd == "info":
+            with open(args.app, "rb") as fh:
+                info = parse(fh.read())
+            print("%s %s  %d bytes of code (blob %d)  ABI %d  api>=%d  shortcut %s  CRC 0x%08X"
+                  % (info["name"], info["version"], info["code_size"], info["total"], info["abi"],
+                     info["api_min"], info["shortcut"], info["crc32"]))
+            return 0
+        image = _read(args.image)
+        if args.cmd == "list":
+            found = list_apps(image)
+            if not found:
+                print("no apps installed (%d slots at 0x%06X)" % (SLOT_COUNT, REGION_BASE))
+            for info in found:
+                if info.get("kind") == "app":
+                    print("slot %2d @0x%06X  %-16s %-6s %5d bytes  %s"
+                          % (info["slot"], info["base"], info["name"], info["version"],
+                             info["code_size"], info["shortcut"]))
+                else:
+                    print("slot %2d @0x%06X  %s" % (info["slot"], info["base"], info["kind"]))
+            return 0
+        with open(args.app, "rb") as fh:
+            blob = fh.read()
+        if args.cmd == "install":
+            info = install(image, args.slot, blob)
+            with open(args.image, "wb") as fh:
+                fh.write(image)
+            print("installed %s %s into slot %d at 0x%06X (%d bytes)"
+                  % (info["name"], info["version"], args.slot, slot_base(args.slot),
+                     info["code_size"]))
+            return 0
+        if args.cmd == "erase":
+            base = erase(image, args.slot)
+            with open(args.image, "wb") as fh:
+                fh.write(image)
+            print("erased slot %d at 0x%06X" % (args.slot, base))
+            return 0
+    except AppError as exc:
+        print("refused: %s" % exc, file=sys.stderr)
+        return 2
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())