Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events

Aditya Chinchure1,2*, Sahithya Ravi1,2*, Raymond Ng1, Vered Shwartz1,2, Boyang Li3, Leonid Sigal1,2
1University of British Columbia, 2Vector Institute for AI, 3Nanyang Technological University
*Indicates Equal Contribution
Accepted at CVPR 2025
arXiv Code 🤗 Data Leaderboard


BlackSwanSuite involves three tasks:

  1. Forecaster evaluates a model's ability to hypothesize future events.
  2. Detective involves abductive reasoning by explaining the hidden event, and defeasible reasoning by validating existing hypotheses.
  3. Reporter again tests defeasability and the model's ability to describe the unexpected event.

Abstract

The commonsense reasoning capabilities of vision-language models (VLMs), especially in abductive reasoning and defeasible reasoning, remain poorly understood. Most benchmarks focus on typical visual scenarios, making it difficult to discern whether model performance stems from keen perception and reasoning skills, or reliance on pure statistical recall. We argue that by focusing on atypical events in videos, clearer insights can be gained on the core capabilities of VLMs. Explaining and understanding such out-of-distribution events requires models to extend beyond basic pattern recognition and regurgitation of their prior knowledge. To this end, we introduce BlackSwanSuite, a benchmark for evaluating VLMs' ability to reason about unexpected events through abductive and defeasible tasks. Our tasks artificially limit the amount of visual information provided to models while questioning them about hidden unexpected events, or provide new visual information that could change an existing hypothesis about the event. We curate a comprehensive benchmark suite comprising over 3,800 MCQ, 4,900 generative and 6,700 yes/no tasks, spanning 1,655 videos. After extensively evaluating various state-of-the-art VLMs, including GPT-4o and Gemini 1.5 Pro, as well as open-source VLMs such as LLaVA-Video, we find significant performance gaps of up to 32% from humans on these tasks. Our findings reveal key limitations in current VLMs, emphasizing the need for enhanced model architectures and training strategies.

Examples from BlackSwan


How BlackSwan was constructed

BibTeX

@misc{chinchure2024blackswanabductivedefeasible,
        title={Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events}, 
        author={Aditya Chinchure and Sahithya Ravi and Raymond Ng and Vered Shwartz and Boyang Li and Leonid Sigal},
        year={2024},
        eprint={2412.05725},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2412.05725}, 
  }