Text as pictures, evidence bundle Precisit, measurements recorded 24 September 2026 This archive contains the three recorded runs, their summaries, the rendering check and the token-count script. Token counts can be recomputed from the pinned model processor and the recorded first-turn responses, without generating new answers. The counter currently loads the model as well as its processor, so allow about 8 GB for the model download and sufficient unified memory. -------------------------------------------------------------------------------- Setup and recomputing the counts (Apple Silicon, macOS) -------------------------------------------------------------------------------- Run these commands from the directory containing this README and the extracted scripts. Keep the bundled JSON and logs as the original evidence; the commands below write new reports under reproduced/. git clone https://github.com/apple-aiml-research/ml-lensvlm git -C ml-lensvlm checkout 10709a7 uv venv --python 3.12 /tmp/lensvlm uv pip install --python /tmp/lensvlm/bin/python \ "mlx-vlm==0.7.2" "mlx==0.32.2" "pillow==12.3.0" mkdir -p reproduced /tmp/lensvlm/bin/python render_probe.py \ --repo ./ml-lensvlm --out reproduced/render-check.json /tmp/lensvlm/bin/python measure-token-cost.py \ --repo ./ml-lensvlm --runs runs.json --logs . \ --model mlx-community/LensVLM-9B-OptiQ-4bit \ --revision 9e3d1de1d65ed7a301cdeed4b58e70a461a02463 \ --out reproduced/token-cost.json The commands pin the direct packages and model revision used for the experiment; they are not a complete lockfile for every transitive Python dependency. -------------------------------------------------------------------------------- Running a new inference experiment -------------------------------------------------------------------------------- Apple's reference demonstration uses vLLM. This archive adapts the loop to MLX and imports Apple's renderer, prompts, tool parsing and answer extraction. /tmp/lensvlm/bin/python harness-lensvlm-mlx.py \ --repo ./ml-lensvlm \ --model mlx-community/LensVLM-9B-OptiQ-4bit \ --revision 9e3d1de1d65ed7a301cdeed4b58e70a461a02463 \ --compression 10x | tee reproduced/run-10x.log Repeat with --compression 5x or 15x and a matching output filename if desired. These are new runs, not replacements for the original logs. Timing depends on the machine and its current load. The harness prints logs; it does not regenerate runs.json or runs.csv automatically. Those files summarize the recorded logs. -------------------------------------------------------------------------------- Files -------------------------------------------------------------------------------- README.txt setup, method and scope harness-lensvlm-mlx.py portable copy of the inference harness render_probe.py recomputes the renderer's page counts and dimensions measure-token-cost.py recomputes the three token-count views using the recorded first responses and the harness's message-building helpers token-cost.json document-only, first-turn and second-turn input counts; also separates image tokens, delimiters and other text render-check.json page counts, dimensions and first-page character counts runs.json / runs.csv summaries of the three original runs run-5x.log original 5x run output run-10x.log original 10x run output run-15x.log original 15x run output The portable harness exposes model/revision arguments, saves its rendered images and accepts a seed argument. These were added after the recorded runs. Both the harness and token counter pin the model configuration to the same revision as the weights and processor. The original measurements and logs are retained. -------------------------------------------------------------------------------- Environment of record -------------------------------------------------------------------------------- Host Apple M4, 24 GB unified memory, macOS 26.6.2 Python 3.12 mlx-vlm 0.7.2 mlx 0.32.2 pillow 12.3.0 Model mlx-community/LensVLM-9B-OptiQ-4bit, revision 9e3d1de1d65ed7a301cdeed4b58e70a461a02463 (about 8 GB) Upstream code apple-aiml-research/ml-lensvlm at commit 10709a7 Document examples/hotpotqa_demo.json: 32 033 characters, 5 136 words, reference answer "Chief of Protocol" -------------------------------------------------------------------------------- Rendering and generation -------------------------------------------------------------------------------- The three named presets in the pinned upstream sample_render_config() fix the font to bundled DejaVuSans, the background to white and the text to black. They do not sample a colour palette. The seed arguments are retained but do not affect these preset renderings. The final page can be shorter than the others, giving it fewer visual tokens and lowering the average token count per page. Generation was greedy, capped at 1 024 new tokens per turn, with no custom stop sequence. Every recorded turn ended below the cap (the largest used 188 tokens). Apple's reference demonstration stops at the closing tool-call tag; this harness relies on the model's end-of-turn token instead. The loop permits up to six turns; all three recorded runs completed in two turns with one page expansion. The tool returns the requested page's real source text. expanded_page_text_chars counts only that text; tool_response_chars also includes the tool wrapper. Model loading is outside the recorded turn durations. -------------------------------------------------------------------------------- Reading the token numbers -------------------------------------------------------------------------------- View A compares representations with one identical short wrapper, then subtracts that wrapper (27 tokens). The image representation is decomposed into image tokens, vision delimiters and other text. View B counts the first-turn message list, including Apple's system prompt, the actual question and all page images. View C counts the second-turn message list, including the recorded first response and the requested page's source text. The image parts stay on the first user message in both views. These are input lengths at the start of each generation, not cumulative work across turns or measurements of peak KV-cache allocation. The full-text prompt was counted but not run as an inference baseline. -------------------------------------------------------------------------------- Scope -------------------------------------------------------------------------------- This is one document, one question and one 4-bit conversion on one Mac. It does not reproduce Apple's seven-benchmark evaluation or B200 efficiency experiment. It does not measure how much quality the conversion preserves, isolate runtime and quantisation effects, or establish performance on Swedish documents. contains_gold is a literal reference-string check. In the 10x run, the model named ambassadorships that the source page supports, rather than "Chief of Protocol". A false value here is not sufficient to establish a false answer.