Introducing TACO

February 25, 2025in Data, Salesforce

Advancing Multi-Modal AI with TACO: A Breakthrough in Reasoning and Tool Integration

Developing effective multi-modal AI systems for real-world applications demands mastering diverse tasks, including fine-grained recognition, visual grounding, reasoning, and multi-step problem-solving. However, current open-source multi-modal models fall short in these areas, especially when tasks require external tools like OCR or mathematical calculations. These limitations largely stem from the reliance on single-step datasets that fail to provide a coherent framework for multi-step reasoning and logical action chains. Addressing these shortcomings is crucial for unlocking multi-modal AI’s full potential in tackling complex challenges.

Challenges in Existing Multi-Modal Models

Most existing multi-modal models rely on instruction tuning with direct-answer datasets or few-shot prompting approaches. Proprietary systems like GPT-4 have demonstrated the ability to effectively navigate CoTA (Chains of Thought and Actions) reasoning, but open-source models struggle due to limited datasets and tool integration. Earlier efforts, such as LLaVa-Plus and Visual Program Distillation, faced barriers like small dataset sizes, poor-quality training data, and a narrow focus on simple question-answering tasks. These limitations hinder their ability to address complex, multi-modal challenges requiring advanced reasoning and tool application.

Introducing TACO: A Multi-Modal Action Framework

Researchers from the University of Washington and Salesforce Research have introduced TACO (Training Action Chains Optimally), an innovative framework that redefines multi-modal learning by addressing these challenges. TACO introduces several advancements that establish a new benchmark for multi-modal AI performance:

Synthetic CoTA Dataset Generation
- Over 1.8 million traces were generated using GPT-4 and Python programs, with 293K high-quality examples curated through rigorous filtering.
- These datasets ensure diversity in reasoning and action sequences, providing a robust foundation for multi-modal learning.
Tool Integration for Complex Tasks
- TACO leverages a comprehensive set of 15 tools, including OCR, object localization, and mathematical solvers, enabling the system to handle sophisticated tasks effectively.
Advanced Filtering and Data Mixing
- Optimized dataset curation emphasizes the seamless integration of reasoning and actions, fostering superior learning outcomes.

Training and Architecture

TACO’s training process utilized a carefully curated CoTA dataset of 293K instances from 31 sources, including Visual Genome, offering a diverse range of tasks such as mathematical reasoning, OCR, and visual understanding. The system employs:

LLaMA3 as the linguistic foundation.
CLIP as the visual encoder for robust multi-modal learning.
Fine-tuned hyperparameters, including lowered learning rates and extended epochs, to address complex multi-modal challenges.

Benchmark Performance

TACO demonstrated significant performance improvements across eight benchmarks, achieving an average accuracy increase of 3.6% over instruction-tuned baselines and gains as high as 15% on MMVet tasks involving OCR and mathematical reasoning. Key findings include:

The curated 293K CoTA dataset outperformed larger, less refined datasets, highlighting the importance of high-quality data.
Hyperparameter adjustments, including fine-tuning vision encoders and optimizing learning rates, further enhanced performance.

Transforming Multi-Modal AI Applications

TACO represents a transformative step in multi-modal action modeling by addressing critical deficiencies in reasoning and tool-based actions. Its innovative approach leverages high-quality synthetic datasets and advanced training methodologies to unlock the potential of multi-modal AI in real-world applications, from visual question answering to complex multi-step reasoning tasks.

By bridging the gap between reasoning and action integration, TACO paves the way for AI systems capable of tackling intricate scenarios with unprecedented accuracy and efficiency.