---
title: Multimodal Correctness Evaluator
tags: [multimodal-evaluation, vision]
sidebar:
  label: "Multimodal Correctness"
---

## Overview

The `MultimodalCorrectnessEvaluator` assesses whether an agent response is factually correct given the image content. It catches errors in objects, counts, colors, positions, readable text, and described actions.

## Key Features

- **Output-Level Evaluation**: Scores a single agent response per case
- **Binary Scoring**: `1.0` for correct, `0.0` if any factual error is found
- **Automatic Reference Comparison**: Appends a reference suffix to the rubric when `expected_output` is provided on the case
- **Fact-Checking Focus**: Designed to catch factual errors in image descriptions and VQA answers

## When to Use

Use the `MultimodalCorrectnessEvaluator` when you need to:

- Verify factual accuracy of image captions, VQA answers, or chart/document QA responses
- Measure exact correctness on benchmark tasks with a known answer
- Catch small but important errors (off-by-one counts, wrong colors, misread text)

## Evaluation Level

This evaluator operates at the **OUTPUT_LEVEL**, scoring a single agent response per case.

## Parameters

### `rubric` (optional)
- **Type**: `str | None`
- **Default**: `CORRECTNESS_RUBRIC_V0`
- **Description**: Custom rubric. Leave unset to use the default rubric.

### `model` (optional)
- **Type**: `Model | str | None`
- **Default**: `None` (uses default Bedrock model)
- **Description**: Multimodal judge model.

### `include_inputs` (optional)
- **Type**: `bool`
- **Default**: `True`

### `system_prompt` (optional)
- **Type**: `str | None`
- **Default**: `None` (uses the built-in `MLLM_JUDGE_SYSTEM_PROMPT`)

### `reference_suffix` (optional)
- **Type**: `str | None`
- **Default**: `None` (uses the built-in default suffix)
- **Description**: Override to customize reference-based grading.

## Scoring System

| Score | Label | Meaning |
|-------|-------|---------|
| 1.0 | Correct | No factual errors found |
| 0.0 | Incorrect | One or more factual errors found |

A response passes only if the score is `1.0`.

## Basic Usage

### Reference-Free (fact-check against the image)

```python
from strands_evals import Case, Experiment
from strands_evals.evaluators import MultimodalCorrectnessEvaluator
from strands_evals.types import MultimodalInput
from strands_evals.types.evaluation_report import EvaluationReport


def task_function(case: Case) -> str:
    # Replace with your multimodal agent invocation.
    return "There are 3 people sitting on the red couch."


cases = [
    Case(
        name="scene-count",
        input=MultimodalInput(
            media="/path/to/living_room.jpg",
            instruction="How many people are on the couch and what color is it?",
        ),
    ),
]

experiment = Experiment(cases=cases, evaluators=[MultimodalCorrectnessEvaluator()])
reports = experiment.run_evaluations(task_function)
EvaluationReport.flatten(reports).run_display()
```

### Reference-Based (compare against a known answer)

```python
cases = [
    Case(
        name="chart-value",
        input=MultimodalInput(
            media="/path/to/revenue_chart.png",
            instruction="What is the Q3 revenue for Product A?",
        ),
        expected_output="$4.2M",
    ),
]

experiment = Experiment(cases=cases, evaluators=[MultimodalCorrectnessEvaluator()])
reports = experiment.run_evaluations(task_function)
```

When `expected_output` is set, the evaluator automatically appends the reference suffix so the judge compares the response to the reference answer.

## Related Evaluators

- [**MultimodalOutputEvaluator**](multimodal_output_evaluator.md): Parent class with full parameter reference
- [**MultimodalFaithfulnessEvaluator**](multimodal_faithfulness_evaluator.md): Catches hallucinations (claims not verifiable from the image)
- [**CorrectnessEvaluator**](correctness_evaluator.md): Text-only counterpart
