Visual Reasoning

Visual reasoning combines several observations from an image before producing an answer or decision.

It goes beyond simply naming an object or reading one visible fact.

Intuition

Basic VQA:

What does the sign say? No Parking.

Visual reasoning:

Is it safe and legal to park here?

Now the model may need to inspect signs, road markings, vehicle position, time restrictions, and possibly outside knowledge.

How it works

What makes it different from basic VQA?

Basic VQA Visual reasoning
Reads one visible fact Combines several observations
“What color is the light?” “Is it safe to cross?”
Often one perception step Perception + comparison + inference

Common tasks include:

Model choice

Both LLaVA-style and Qwen-VL-style models can reason:

Decompose before deciding

Direct prompt:

Is it safe to park here?

This invites a confident yes/no jump.

Better prompt:

First list every visible sign, marking, and object relevant to parking.
Second, explain which rule each item suggests.
Third, check for contradictory evidence.
Finally answer: allowed, not allowed, or uncertain.
flowchart LR IMG[Image] --> OBS[1. List visible evidence] OBS --> RULES[2. Apply relevant rules] RULES --> CHECK[3. Check contradictions] CHECK --> DEC[4. Allowed / not allowed / uncertain]

The benefit is auditability: a reviewer can see whether the model noticed the No Parking sign.

A reasoning trace is not proof

A model can write a polished five-step explanation whose first observation is wrong.

Example:

  1. “The sign says Parking Allowed.” ← wrong visual read
  2. “The vehicle is inside the marked bay.”
  3. “No obstruction is visible.”
  4. “Therefore parking is allowed.” ← conclusion inherits step 1

This is a compounding error: one bad observation contaminates every later step.

flowchart TB WRONG[Wrong first observation] --> STEP2[Apparently logical step 2] STEP2 --> STEP3[Apparently logical step 3] STEP3 --> ANSWER[Confident wrong answer]

Do not confuse a detailed chain with a correct chain.

Add a verification pass

For important decisions:

  1. Generate the evidence list.
  2. Independently verify key observations.
  3. Check calculations or rules with a deterministic tool when possible.
  4. Allow uncertain instead of forcing an answer.
observations = vlm.inspect(image, task="list parking signs and markings")
decision = reasoner.decide(observations)

verification = verifier.check(image, observations)
if verification.has_conflict or verification.low_confidence:
    final = {"decision": "uncertain", "needs_review": True}
else:
    final = {"decision": decision, "needs_review": False}

The verifier can be a second prompt, another model, OCR, a rule engine, or a human — depending on risk.

Why visual reasoning is hard

Difficulty Why it matters
Multi-step inference Several facts must remain correct at once
External knowledge Physics, maths, laws, or domain rules may be needed
Occlusion Important evidence may be partly hidden
Counterfactuals “What would happen if…” is not directly visible
Verification Fluent pattern matching can look like real reasoning

Benchmarks

Dataset Focus
MMMU College-level multimodal questions across subjects
MathVista Visual mathematical reasoning
ScienceQA Science questions with diagrams
NLVR2 Relationships across image pairs
Winoground Difficult compositional language–image reasoning

The lecture notes that visual reasoning still has a large gap between current models and human experts. Reported benchmark gains can also be affected by test-data contamination.

Training patterns

Modern systems often mix:

A strong text-reasoning LLM can transfer some of that ability once the visual features are properly grounded. Better “eyes” help perception; they do not automatically provide better reasoning.

What goes wrong

One-line summary

Visual reasoning combines multiple observations; expose the evidence, check key steps independently, and allow uncertainty because a convincing chain can still be wrong.

Key terms