Text as pictures, evidence bundle
Precisit, measurements recorded 24 September 2026

This archive contains the three recorded runs, their summaries, the rendering
check and the token-count script. Token counts can be recomputed from the pinned
model processor and the recorded first-turn responses, without generating new
answers. The counter currently loads the model as well as its processor, so allow
about 8 GB for the model download and sufficient unified memory.

--------------------------------------------------------------------------------
Setup and recomputing the counts (Apple Silicon, macOS)
--------------------------------------------------------------------------------
Run these commands from the directory containing this README and the extracted
scripts. Keep the bundled JSON and logs as the original evidence; the commands
below write new reports under reproduced/.

  git clone https://github.com/apple-aiml-research/ml-lensvlm
  git -C ml-lensvlm checkout 10709a7

  uv venv --python 3.12 /tmp/lensvlm
  uv pip install --python /tmp/lensvlm/bin/python \
      "mlx-vlm==0.7.2" "mlx==0.32.2" "pillow==12.3.0"

  mkdir -p reproduced
  /tmp/lensvlm/bin/python render_probe.py \
      --repo ./ml-lensvlm --out reproduced/render-check.json

  /tmp/lensvlm/bin/python measure-token-cost.py \
      --repo ./ml-lensvlm --runs runs.json --logs . \
      --model mlx-community/LensVLM-9B-OptiQ-4bit \
      --revision 9e3d1de1d65ed7a301cdeed4b58e70a461a02463 \
      --out reproduced/token-cost.json

The commands pin the direct packages and model revision used for the experiment;
they are not a complete lockfile for every transitive Python dependency.

--------------------------------------------------------------------------------
Running a new inference experiment
--------------------------------------------------------------------------------
Apple's reference demonstration uses vLLM. This archive adapts the loop to MLX
and imports Apple's renderer, prompts, tool parsing and answer extraction.

  /tmp/lensvlm/bin/python harness-lensvlm-mlx.py \
      --repo ./ml-lensvlm \
      --model mlx-community/LensVLM-9B-OptiQ-4bit \
      --revision 9e3d1de1d65ed7a301cdeed4b58e70a461a02463 \
      --compression 10x | tee reproduced/run-10x.log

Repeat with --compression 5x or 15x and a matching output filename if desired.
These are new runs, not replacements for the original logs. Timing depends on
the machine and its current load. The harness prints logs; it does not regenerate
runs.json or runs.csv automatically. Those files summarize the recorded logs.

--------------------------------------------------------------------------------
Files
--------------------------------------------------------------------------------
README.txt              setup, method and scope
harness-lensvlm-mlx.py   portable copy of the inference harness
render_probe.py         recomputes the renderer's page counts and dimensions
measure-token-cost.py   recomputes the three token-count views using the recorded
                        first responses and the harness's message-building helpers
token-cost.json         document-only, first-turn and second-turn input counts;
                        also separates image tokens, delimiters and other text
render-check.json       page counts, dimensions and first-page character counts
runs.json / runs.csv    summaries of the three original runs
run-5x.log              original 5x run output
run-10x.log             original 10x run output
run-15x.log             original 15x run output

The portable harness exposes model/revision arguments, saves its rendered images
and accepts a seed argument. These were added after the recorded runs. Both the
harness and token counter pin the model configuration to the same revision as the
weights and processor. The original measurements and logs are retained.

--------------------------------------------------------------------------------
Environment of record
--------------------------------------------------------------------------------
  Host            Apple M4, 24 GB unified memory, macOS 26.6.2
  Python          3.12
  mlx-vlm         0.7.2
  mlx             0.32.2
  pillow          12.3.0
  Model           mlx-community/LensVLM-9B-OptiQ-4bit,
                  revision 9e3d1de1d65ed7a301cdeed4b58e70a461a02463 (about 8 GB)
  Upstream code   apple-aiml-research/ml-lensvlm at commit 10709a7
  Document        examples/hotpotqa_demo.json: 32 033 characters, 5 136 words,
                  reference answer "Chief of Protocol"

--------------------------------------------------------------------------------
Rendering and generation
--------------------------------------------------------------------------------
The three named presets in the pinned upstream sample_render_config() fix the
font to bundled DejaVuSans, the background to white and the text to black. They
do not sample a colour palette. The seed arguments are retained but do not affect
these preset renderings. The final page can be shorter than the others, giving it
fewer visual tokens and lowering the average token count per page.

Generation was greedy, capped at 1 024 new tokens per turn, with no custom stop
sequence. Every recorded turn ended below the cap (the largest used 188 tokens).
Apple's reference demonstration stops at the closing tool-call tag; this harness
relies on the model's end-of-turn token instead. The loop permits up to six turns;
all three recorded runs completed in two turns with one page expansion.

The tool returns the requested page's real source text. expanded_page_text_chars
counts only that text; tool_response_chars also includes the tool wrapper.
Model loading is outside the recorded turn durations.

--------------------------------------------------------------------------------
Reading the token numbers
--------------------------------------------------------------------------------
View A compares representations with one identical short wrapper, then subtracts
that wrapper (27 tokens). The image representation is decomposed into image
tokens, vision delimiters and other text.

View B counts the first-turn message list, including Apple's system prompt, the
actual question and all page images. View C counts the second-turn message list,
including the recorded first response and the requested page's source text.
The image parts stay on the first user message in both views.

These are input lengths at the start of each generation, not cumulative work
across turns or measurements of peak KV-cache allocation. The full-text prompt
was counted but not run as an inference baseline.

--------------------------------------------------------------------------------
Scope
--------------------------------------------------------------------------------
This is one document, one question and one 4-bit conversion on one Mac. It does
not reproduce Apple's seven-benchmark evaluation or B200 efficiency experiment.
It does not measure how much quality the conversion preserves, isolate runtime
and quantisation effects, or establish performance on Swedish documents.

contains_gold is a literal reference-string check. In the 10x run, the model
named ambassadorships that the source page supports, rather than "Chief of
Protocol". A false value here is not sufficient to establish a false answer.
