Run the localization loop¶
Boxes, keypoints, polygons, and masks share one procedure: the model looks at a gridded preview, proposes candidates, sees its own candidates drawn back onto the same view, corrects them by index, and commits — within a number of rounds agreed in the interview, three by default. Captioning has no such loop; localization does, because a model cannot judge a coordinate it has not seen rendered.
- Look —
preview_image_grid(source, columns=10, rows=10). The grid is the anchor: ten cells each way, so nine lines. Denser or sparser grids measure worse in practice. Tiny objects get acropand the loop runs on the crop, not a larger preview. - Propose — candidates in the model's native convention, with labels and confidence. Convert to normalized xyxy immediately, clamp to [0, 1], drop zero-area boxes.
- Verify —
preview_image_bboxes(source, objects=[...], grid={})with the same grid, each box carrying an index label. - Correct — ask for adjustments by index ("move box 2's right edge to the ear"), re-render, repeat until the model calls the overlay acceptable or the round limit is reached. Every round counts, including one that changes nothing.
- Commit — one write with the final list, the round count, and the model's self-assessment.
Intermediate rounds are never written as final; an item that exhausts its rounds is committed as
needs_reviewwith the last overlay saved.
Three ways to lose accuracy in this loop, all silent: rendering the verification overlay without the
grid the model proposed on, re-ordering the boxes between rounds so the indices move under the
model, and mixing index bases — the box labels drawn by the skill's asset are 0-based while
preview_image_polygons numbers vertices from 1, so the prompt has to say which.
Ask in the model's convention, store in yours¶
Storage is always normalized 0–1 relative to the uncropped source. Prompting is not: every model localizes best in the frame it was trained on, and Claude's documentation says so explicitly. Ask natively, then convert in code. Never ask a model to normalize.
| Model | Native convention (vendor statement, verified 2026-08-27) | Prompt wording | Convert with |
|---|---|---|---|
| Claude | Absolute pixels of the image as sent: "Claude works best with absolute pixel coordinates … does not work well when you ask for normalized coordinates, for example: 'Return bounding box coordinates between 0 and 1000'". Coordinates refer to the resized image if Claude had to resize; normalize by the resized, never the padded, size. | "Return [x1, y1, x2, y2] in pixel coordinates of the image as shown (width W, height H)" with W/H from output_width/output_height |
base = output_width, output_height, axis_order="xy" |
| GPT-5.4+ and Codex | The GPT-5.4 vision tips recommend "a strict coordinate contract like [x_min, y_min, x_max, y_max] and a fixed coordinate space such as 0..999 with the origin in the top-left corner" (plus code-interpreter access for localization). Pixels of the sent image also work but are not the documented recommendation. |
"Return [x_min, y_min, x_max, y_max] as integers in a fixed 0..999 space, origin top-left" |
base = 999, 999 (or 1000 if you prompt 0–1000), axis_order="xy" |
| Gemini | box_2d = [ymin, xmin, ymax, xmax] normalized to 0–1000 (y first); segmentation masks as [x, y] polygons in the same space |
"The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000" |
base = 1000, 1000, axis_order="yx" |
| Qwen2.5-VL | Absolute coordinates "using the actual size scale of the image, without performing traditional coordinate normalization" — pixels of the smart-resized input (multiples of 28) | pixel [x1, y1, x2, y2] of the image as shown; state W×H |
base = output_width, output_height (send a preview already sized to multiples of 28 so no resize happens) |
| Qwen3-VL | Relative 0–1000 [x1, y1, x2, y2] (top-left, bottom-right); some cookbook utilities divide by 999 |
"bbox_2d [x1, y1, x2, y2] in a 0-1000 space" |
base = 1000, 1000, axis_order="xy" |
| Unknown / other | Undocumented | Trial-label 3 images with the grid preview, asking for pixels of the shown image; assert the value range of the answer (≤ output_width → pixels; ≤ 1000 with values near 1000 on a small image → fixed space) before choosing |
per the observed range |
The conversion is one call, with base the frame the model answered in and crop taken from the
preview metadata:
from annotools import normalize_coordinates
# Gemini answers [ymin, xmin, ymax, xmax] in a 0-1000 space; the preview showed the full frame.
boxes = normalize_coordinates([[113, 21, 897, 494]], 1000, 1000, axis_order="yx")
# [[0.021, 0.113, 0.494, 0.897]]
Gemini's y-first order is the most common silent bug in this step: swapped axes still produce
plausible boxes. Assert the value range of the answer before converting — pixel answers stay under
output_width, fixed-space answers reach toward 1000 even on a small image — and a model that
answers in the wrong convention will be obvious rather than merely wrong.
Variants and quality gates¶
Keypoints use the same loop with preview_image_keypoints and a per-point visibility value;
polygons use preview_image_polygons with vertex indices on, or
preview_image_segmentation for masks; rotated boxes are converted with rotated_bbox_to_polygon
and rejected when is_rectangle fails; video keyframes are annotated, interpolated in code, and the
interpolated frames verified as images.
Before anything is rendered, reject boxes outside the image, below the spec's minimum area, duplicated at IoU > 0.9, or carrying a label outside the class list, and report the rejection back to the model. After the run, sample 5 % for a second pass with a different seed or model and log disagreement. Track rounds per item: a rising average means the prompt or the grid setting drifted.
What it measures¶
Both detection examples ran the loop on three COCO cat images at a 768 px preview with a 10×10 grid and a three-round limit on 2026-08-28:
object-detection-claude |
object-detection-codex |
|
|---|---|---|
| Coordinates asked for | pixels of the shown image | fixed 0..999 space |
| Mean rounds per image | 1.0 | 0.67 (1.0 over the two finished images) |
| Mean best IoU vs. the COCO boxes | 0.913 | 0.929 after run 1 (2 images); 0.866 after the retry (3 images) |
| Recall @ 0.5 IoU | 1.0 | 1.0 |
needs_review |
0 of 3 | 1 of 3 (no box committed; a retry finished it in one round) |
One round is the common case: the first proposal, seen rendered, is usually accepted. The IoU figures are informational — the COCO boxes are a reference, not the specification — and three images is a sanity check, not a benchmark.
Next: give the execution agent the tools this loop assumes.
Source: skills/localization-annotation-guide,
with the coordinate table from skills/mllm-multimodal-input
and the pipeline skeleton in skills/task-object-detection