MCP tools¶

Every tool the annotools MCP server exposes, in the order an annotation session usually needs them. Coordinates are normalized 0.0-1.0 relative to the uncropped source; shared parameter groups, metadata keys, and error rules are in the overview. The parameter tables come from the server's tool schemas; the text below each table is the tool's specification.

preview_image¶

Downscale an image (optionally zoomed to a normalized crop box) to fit a token budget.

Returns the image followed by one JSON metadata object (original_size, crop, output_size, scale, format). Coordinates are normalized 0-1 relative to the uncropped source.

Goal¶

Return an image at a size an MLLM can read within a bounded token budget, optionally zoomed into a region, so an agent never has to load a full-resolution asset to look at it.

Interface¶

Tool: preview_image (MCP) / annotools.image.preview.preview + encode (library)

Parameters: exactly PreviewOptions from .agents/knowledge/spec/mcp-overview.md.

Returns: [Image, metadata] with the base metadata keys.

Behavior¶

1. Load source through fsspec; apply EXIF orientation; record original_size.
2. If crop is given, validate it and crop to the pixel box (rounded outward so the region is never smaller than requested).
3. Compute the output size: start from the cropped size; scale down to fit max_width × max_height and, if given, target_pixels; scale up only when allow_upscale is true and the cropped image is smaller than the limits (the same fit rule, applied upward). Aspect ratio is preserved; sizes are rounded to integers ≥ 1.
4. Resize with Lanczos (down) / bicubic (up); encode as output_format (JPEG q90, alpha flattened on white); optionally write to save_to.
5. Error: crop outside [0, 1] or non-increasing → ValueError("crop: …").
6. Error: max_width, max_height, target_pixels < 1 → ValueError.
7. Error: missing source → FileNotFoundError; other read failures → OSError; undecodable content → ValueError; all name the URI.
8. EXIF orientation is applied on load (a rotated-by-tag portrait reports its display size).

Out of scope¶

Grids, overlays, video frames (own specs).

preview_image_grid¶

Preview an image with a semi-transparent grid (default 10x10 cells, 50% white) to anchor positions.

Returns the image and one JSON metadata object including grid step sizes in normalized coordinates of the cropped view. Coordinates are normalized 0-1 relative to the uncropped source.

Goal¶

Overlay a semi-transparent grid on a preview so an MLLM can anchor positions to cells; 10×10 by default (9 lines each way), which evidence suggests improves localization without occluding content.

Interface¶

Tool: preview_image_grid (MCP) / annotools.image.grid.draw_grid (library)

Parameters: PreviewOptions plus GridOptions (flattened: columns, rows, mode, column_width, row_width, color, opacity, line_width) from .agents/knowledge/spec/mcp-overview.md.

Returns: [Image, metadata] with the base keys plus grid: {"columns": int, "rows": int, "step_x": float, "step_y": float, "cell_width": float, "cell_height": float} where step_* is the cell size in normalized coordinates of the cropped view (so 1 / columns in ratio mode) and cell_* is the same cell size in output pixels (output_width / columns, or the configured widths in fixed mode). When a later step re-fits the image (the preview_image_segmentation legend), cell_* is rescaled with it and refers to the returned pixels.

Behavior¶

1. Render the preview exactly as preview_image.
2. Compute line positions in output pixels: ratio mode → i × width / columns for i = 1..columns−1 (same for rows); fixed mode → multiples of column_width / row_width inside the image; the resulting cell counts are reported as columns / rows.
3. Draw each line line_width px wide, centred on the position, blended with opacity: white and black blend toward that color; invert blends toward the per-pixel inverse of the image.
4. Encode as usual.
5. Error: columns/rows < 1, opacity outside [0, 1], line_width < 1 → ValueError.
6. Error: mode="fixed" without both widths, or a width < 1 → ValueError("column_width/row_width …").

Out of scope¶

Row/column index labels; overlays (own specs).

preview_image_bboxes¶

Preview an image with bounding boxes (normalized xyxy, optional labels), optionally over a grid.

Use it to check candidate detections before writing them to the database. Returns the image and one JSON metadata object (base keys, grid when used, objects count).

Goal¶

Let an agent see its candidate bounding boxes on the preview (optionally over the grid) before writing them to the database, so it can correct positions in a short verify-and-adjust loop.

Interface¶

Tool: preview_image_bboxes (MCP) / annotools.image.overlay.draw_bboxes (library)

Parameters: PreviewOptions, grid: GridOptions | null (nested object; null = no grid), objects: list[BBoxObject] (≥ 1), line_width: int = 2 (≥ 1).

BBoxObject: bbox: [x_min, y_min, x_max, y_max] normalized to the uncropped source, label: str | null, color: str = "blue" (CSS/PIL name or #RRGGBB).

Returns: [Image, metadata] with the base keys, grid when a grid was drawn, and objects: int.

Behavior¶

1. Render the preview (and the grid, if given) exactly as preview_image_grid.
2. Map each box from source-normalized coordinates into output pixels through the crop: parts outside the crop are clipped; a box entirely outside is skipped (still counted in objects).
3. Draw the rectangle outline line_width px wide in color. With a label, draw a filled tag in color with the label text (PIL default font) just above the box's top-left corner, moved inside the image when it would leave the frame; text is white or black depending on the color's luminance.
4. Encode as usual.
5. Error: empty objects, invalid box (range/order), unknown color, line_width < 1 → ValueError naming the object index (objects[2].bbox: …).

Out of scope¶

Keypoints, polygons, segmentation, filled boxes.

preview_image_keypoints¶

Preview an image with keypoints (normalized xy, optional labels) drawn as dots, optionally over a grid.

Returns the image and one JSON metadata object (base keys, grid when used, objects count).

Goal¶

Show keypoint annotations as dots (with optional labels) on the preview, optionally over the grid, so an agent can verify point placement before storing it.

Interface¶

Tool: preview_image_keypoints (MCP) / annotools.image.overlay.draw_keypoints (library)

Parameters: PreviewOptions, grid: GridOptions | null, objects: list[KeypointObject] (≥ 1), point_diameter: int = 3 (≥ 1, output pixels).

KeypointObject: point: [x, y] normalized to the uncropped source, label: str | null, color: str = "blue".

Returns: [Image, metadata] with the base keys, grid when drawn, and objects: int.

Behavior¶

1. Render the preview (and grid) as preview_image_grid.
2. Map each point through the crop; points outside the output image are skipped (still counted).
3. Draw a filled circle of point_diameter px centred on the mapped point. With a label, draw the same filled tag as bounding boxes to the right of the dot, vertically centred on it (moved inside the frame when needed).
4. Error: empty objects, point outside [0, 1], unknown color, point_diameter < 1 → ValueError naming the object index.

Out of scope¶

Skeleton edges between keypoints; visibility flags.

preview_image_polygons¶

Preview an image with polygons (flat normalized [x1, y1, x2, y2, ...]) with vertex dots and indices.

Works for COCO-style outlines and DOTA-style rotated boxes (4 corners). Returns the image and one JSON metadata object (base keys, grid when used, objects count).

Goal¶

Preview polygons — COCO-style segmentation outlines or DOTA-style rotated boxes given as 4 corners — with vertex dots and 1-based vertex indices, so an agent can verify both shape and vertex order.

Interface¶

Tool: preview_image_polygons (MCP) / annotools.image.overlay.draw_polygons (library)

Parameters: PreviewOptions, grid: GridOptions | null, objects: list[PolygonObject] (≥ 1), line_width: int = 2, point_diameter: int = 3, show_point_index: bool = true.

PolygonObject: points: [x1, y1, x2, y2, …] (flat, even count, ≥ 3 points, normalized to the uncropped source), label: str | null, color: str = "blue".

Returns: [Image, metadata] with the base keys, grid when drawn, and objects: int.

Behavior¶

1. Render the preview (and grid) as preview_image_grid.
2. Map every vertex through the crop; draw the closed outline line_width px wide (PIL clips segments outside the frame). A polygon with every vertex outside the frame is skipped (still counted).
3. Draw a dot of point_diameter at each vertex; when show_point_index is true, draw the 1-based index as a tag next to the vertex, offset away from the polygon centroid.
4. With a label, draw the label tag at the first vertex.
5. Error: empty objects, odd number of values or fewer than 3 points, a value outside [0, 1], unknown color, line_width/point_diameter < 1 → ValueError naming the object index.

Out of scope¶

Filled polygons; segmentation masks (own spec); rotated-box conversion (rotated_bbox_to_polygon, P2).

preview_image_segmentation¶

Preview an image with an ID mask (instance/panoptic/semantic) blended on top, labelled or with a legend.

Regions are coloured with color_from_text(str(id)). Returns the image and one JSON metadata object (base keys, grid when used, ids, and legend in legend mode).

Goal¶

Show an instance / panoptic / semantic mask on top of the preview (optionally over the grid) so an agent can check region extents and which ID is which, at preview size.

Interface¶

Tool: preview_image_segmentation (MCP) / annotools.image.segmentation.load_mask + overlay_mask

Parameters: PreviewOptions, grid: GridOptions | null, plus

Returns: [Image, metadata] with the base keys, grid when drawn, ids (number of non-zero IDs present), and in legend mode legend: [{"id", "name", "color"}] plus image_size: [w, h] — the image area at the top of the composite. In legend mode output_size (and output_width/output_height) is the whole composite and grid.cell_*, if a grid was drawn, is rescaled to it; the inverse mapping of mcp-overview.md applies to image_size (and scale refers to it).

Behavior¶

1. Load the mask; it must decode to mode L, P (palette indices), I, or I;16 (any byte order) with one channel — anything else raises ValueError("mask_source: …").
2. Resize the mask to the source size (nearest neighbour) if it differs, apply exactly the pixel crop the image received (carried on the preview result, not re-derived from the normalized box), and resize to the output size (nearest). RGBA sources are flattened onto white first.
3. Colour every non-zero ID with color_from_text(str(id)) and blend it over the image at alpha.
4. If line_width > 0, draw region boundaries (pixels whose ID differs from a 4-neighbour) in the region colour at full opacity, line_width px.
5. label: draw the ID (or its id_names entry) as a tag at the region centroid, clamped inside the frame. legend: append a strip below the image listing swatch id name rows (wrapping into columns), then re-fit the combined image inside max_width × max_height and target_pixels (plain resize of the composite); report image_size and rescale scale accordingly. The limits apply to the composite and default to the settings, not to the preview's own size, so a caller that previewed at 768 px and passes no limits gets a 384 px composite back.
6. Grid (if any) is drawn before the mask colours.
7. Error: alpha outside [0, 1], line_width < 0, unknown annotation → ValueError.

Out of scope¶

RGB-encoded panoptic masks; polygonization; per-ID custom colours.

preview_video¶

Sample a video at fps (default 1 frame/s, at most max_frames) and preview each frame at a bounded size.

Returns the frames in time order followed by one JSON metadata object (timestamps, duration, thinning, first-frame size/crop/scale). save_to is a directory for the frame files.

Goal¶

Show a video as a bounded sequence of small frames — sampled at a low frame rate, capped in count, and each rendered through the image preview pipeline — instead of loading the whole video into a model.

Interface¶

Tools: preview_video, preview_video_grid (MCP) / annotools.video.sample_frames (library)

Parameters: PreviewOptions (applied to every frame) plus

preview_video_grid adds the flat GridOptions parameters of preview_image_grid.

Returns: N image blocks (in time order) followed by one JSON text block with the base keys of the first frame plus frames: N, timestamps: [seconds…], duration: seconds, requested_fps, thinned: bool, and grid for the grid variant.

Behavior¶

1. Open the source with PyAV (annotools[media]) through a streamed fsspec handle (local and remote alike); a missing dependency raises ImportError naming the extra; undecodable content raises ValueError naming the source.
2. Decode the first video stream from start (keyframe seek, then skip) to end (or the end of the stream); select the first decoded frame whose time reaches each target start + k / fps.
3. If more than max_frames frames were selected, keep max_frames of them evenly spaced by index (always including the first) and set thinned to true.
4. Render each frame with the image preview (crop, limits, upscale rule) and, for the grid variant, the grid; encode with output_format. save_to is treated as a directory: frames are written as frame_<index>_<timestamp>.<ext> inside it.
5. Error: fps ≤ 0, start < 0, end ≤ start (start defaults to 0), max_frames < 1 → ValueError; no video stream, or no frame inside [start, end) → ValueError naming the source; unreadable source → as preview_image.

Out of scope¶

Audio; scene detection; overlays on frames (use the image tools on saved frames).

preview_video_grid¶

Like preview_video, with a semi-transparent grid (default 10x10) drawn on every frame to anchor positions.

Specification: shared with preview_video (.agents/knowledge/spec/preview-video.md).

clip_audio¶

Cut a segment from an audio (or video) file, optionally resampled, and return it as 16-bit WAV.

Returns the audio followed by one JSON metadata object (source_duration, start, end, duration, sample_rate, channels).

Goal¶

Hand a model only the audio segment it needs, at the sample rate the caller chooses, without shipping the whole file.

Interface¶

Tool: clip_audio (MCP) / annotools.audio.clip_audio (library)

Returns: one audio/wav block followed by one JSON text block: source_duration, start, end, duration, sample_rate, channels, format: "wav", saved_to when used.

Behavior¶

1. Open the source with PyAV (annotools[media]; missing → ImportError naming the extra) through a streamed fsspec handle; a source without an audio stream, or undecodable content, raises ValueError naming it.
2. Seek to start (keyframe-safe), decode the first audio stream, drop samples before start, stop at end (or the end of the stream).
3. Resample to sample_rate when given (channel layout kept), then encode 16-bit PCM WAV in memory.
4. Error: start < 0, end ≤ start (start defaults to 0), sample_rate < 1 → ValueError; start beyond the source duration → ValueError.

Out of scope¶

Compressed output formats; loudness normalization; channel mixing.

color_from_text¶

Return a stable, saturated color for any text: same text, same color; different text, unrelated color.

Goal¶

Give agents (and the segmentation tool) a stable colour for any label or ID without maintaining a palette: the same text always maps to the same colour, and slightly different text maps to an unrelated one.

Interface¶

Tool: color_from_text (MCP) / annotools.color.color_from_text (library)

Returns (structured): {"hex": "#rrggbb", "rgb": [r, g, b]} with hex lowercase.

Behavior¶

1. digest = sha256(text.encode("utf-8")).
2. Hue = digest[0:2] as a big-endian integer / 65536 (a 0–1 hue as colorsys expects; × 360 for degrees); saturation = 0.75; lightness = 0.5. Two bytes give 65 536 hues, which already exceeds what 8-bit RGB can distinguish on one HSL ring, so the "first 3 bytes" sketch in issue #24 was reduced to 2.
3. Convert HSL → RGB (0–255, rounded) and format hex.

No error conditions.

Out of scope¶

Palette optimisation for contrast between arbitrary label sets.

rotated_bbox_to_polygon¶

Convert rotated boxes (cx, cy, w, h, theta; theta clockwise) into DOTA-style 8-number corner polygons.

Corners are not clipped: a box touching the border can yield coordinates outside 0-1, which preview_image_polygons rejects — clamp or shrink such boxes before previewing them. Pass the source aspect ratio so rotation on non-square images does not shear.

Goal¶

Turn rotated detections (cx, cy, w, h, theta) into the DOTA-style 8-number corner lists that preview_image_polygons renders and that SQLite exports store, and let agents verify that a 4-point polygon is actually a rectangle.

Interface¶

Tool: rotated_bbox_to_polygon (MCP) / annotools.geometry.rotated_box_to_corners and annotools.geometry.is_rectangle (library)

Returns (structured): {"polygons": [[x1, y1, x2, y2, x3, y3, x4, y4], ...]} — one entry per box, corners in clockwise image order starting at the top-left corner of the unrotated box, normalized to the same frame as the input.

Behavior¶

1. theta is the clockwise rotation (image coordinates, y down) of the box about its centre; angle_unit="radians" converts first. This matches DOTA/mmrotate le90 after their own conversion to degrees; the spec does not normalize the angle range.
2. Rotation happens in an isotropic frame: x is scaled by aspect_ratio before rotating and divided afterwards, so a box on a non-square image rotates without shearing. With the default 1.0 the result is exact only for square sources.
3. Corners may fall outside [0, 1]; they are returned unclipped. preview_image_polygons rejects such coordinates, so callers clamp or shrink border boxes before previewing (a clip option is a possible follow-up).
4. is_rectangle(points, *, angle_tol_deg=2.0, length_tol=0.02) (tolerances keyword-only): true when the polygon has 4 points, adjacent edges are perpendicular within angle_tol_deg, and opposite edges have equal length within length_tol (relative). Angles are measured in the space the points are given in: normalizing scales x and y differently on a non-square image, which shears a rotated rectangle until it fails the test, so run it on the model's pixel answer. Axis-aligned rectangles are unaffected.
5. Error: empty boxes, w/h ≤ 0, cx/cy outside [0, 1], aspect_ratio ≤ 0 → ValueError naming boxes[i].<field> or the parameter.

Out of scope¶

Polygon → rotated box conversion; angle-range normalization.

normalize_coordinates¶

Convert a model's coordinates into the storage convention (normalized 0-1, x-first, uncropped source).

Ask each model in its native convention, then pass its answer here with the frame it used: for pixel answers (Claude, Qwen2.5-VL) base_width/base_height are the preview's output_width/output_height; for 0-1000 answers (Gemini, Qwen3-VL) or 0-999 (GPT) use 1000 or 999. Pass the preview's crop so a zoomed view maps back into the full image. Results are clamped to [0, 1].

Goal¶

Models answer localization questions in their own frame — Claude and Qwen2.5-VL in pixels of the image they were shown, Gemini and Qwen3-VL in a 0–1000 space (Gemini y-first), GPT in a 0–999 space — and perform worse when asked to normalize themselves. The storage convention is normalized 0–1, x-first, relative to the uncropped source. These two tools do the conversion in code so an agent can ask every model natively and store one format, including when the model looked at a crop.

Interface¶

Tools: normalize_coordinates, denormalize_coordinates (MCP) / annotools.geometry.normalize_coordinates, annotools.geometry.denormalize_coordinates (library).

Returns: {"coordinates": list[list[float]]} with the same shape as the input.

• normalize_coordinates: base frame (of the cropped view) → normalized source coordinates, always x, y order, each value clamped to [0, 1].
• denormalize_coordinates: normalized source coordinates (x, y order, each in [0, 1]) → base frame of the cropped view, written y, x per pair when axis_order="yx"; not clamped or rounded.

Behavior¶

1. Validate base_width/base_height > 0 and crop (same rules as PreviewOptions.crop).
2. For each entry, split into pairs (swap when axis_order="yx" on the base side).
3. normalize: x_norm = crop.x_min + x / base_width × (crop.x_max − crop.x_min), likewise y; clamp. denormalize: x_base = (x − crop.x_min) / (crop.x_max − crop.x_min) × base_width, likewise y.
4. Error coordinates[i]: expected an even number of values, got n for an odd-length entry.
5. Error coordinates: base_width and base_height must be > 0 for a non-positive base.
6. Error coordinates[i]: (x, y) is outside [0, 1] when denormalizing a value outside the range.
7. Error crop: … from the shared crop validation.

Out of scope¶

Choosing the convention for a model (the mllm-multimodal-input skill); rotated boxes (use rotated_bbox_to_polygon first, then convert the 8-number entry); rounding to integer pixels.

denormalize_coordinates¶

Convert stored normalized coordinates into a model's frame (pixels of a preview or a 0-1000 space).

The inverse of normalize_coordinates: the input is normalized 0-1 relative to the uncropped source, the output is in the base_width x base_height frame of the given crop (y-first pairs when axis_order='yx'). Values are not clamped, so a point outside the crop maps outside the frame.

Specification: shared with normalize_coordinates (.agents/knowledge/spec/coordinates.md).