Compound AI Systems: Beyond Single Models

As tempting and mouth watering the image looks, ask any chef, it is as such a complex piece of things to do. It’s the perfectly cooked soft pan cakes, temperature, stacking, syrup, the fruit, the presentation.. yes it a complex breakfast with a lot of work going into it.

Now coming to our AI world, what is a Compound AI system? An AI system that does more than just being a Large Language Model (LLM), but is an LLM in conjunction with other components like integrating with databases, ability to search on the internet etc into a system or a chain that allows it to achieve its State of the Art result.

Examples of Compound AI Systems

Some examples of compound AI systems:

• Autonomous vehicles
  Combining computer vision, sensor data, speech recognition and information retrieval

• Prompt engineering
  Prompt engineering logic + LLM

• RAG (Retrieval Augmented Generation)
  LLM + Retrieval systems, Data API’s like Databricks Online Table, Text-To-SQL engines

• And many more like Healthcare diagnostics, Fraud detection systems, Siri, Alexa…

Core Concepts: Tasks, Intents, and Pipelines

Compound AI systems are designed to tackle complex tasks by combining multiple AI components. Three fundamental concepts: tasks, intents, and pipelines need to be understood.

Tasks

A task is a specific activity or operation that an AI system is designed to perform.

Examples:

• Sentiment analysis
• Language translation
• Image classification
• Text summarization

Characteristics:

• Well-defined input: Tasks typically have a clear and well-defined input, such as text or an image
• Specific output: Tasks produce a specific output, such as a sentiment score or a translated text
• Measurable performance: Task performance can be measured using metrics like accuracy, precision, or recall

Intents

An intent is the underlying purpose or goal that a user wants to achieve when interacting with an AI system.

Examples:

• Booking a flight
• Finding a restaurant
• Getting directions
• Making a purchase

Characteristics:

• User-centric: Intents are focused on the user’s needs and goals
• Context-dependent: Intents can depend on the user’s context, such as location or previous interactions
• Flexible input: Intents can be expressed in various ways, such as through text, voice, or gestures

Pipelines

A pipeline is a series of tasks or processes that are executed in a specific order to achieve a particular intent.

Examples:

A language translation pipeline might involve:

1. Text analysis
2. Language detection
3. Machine translation
4. Post-editing

A sentiment analysis pipeline might involve:

1. Text preprocessing
2. Feature extraction
3. Model training
4. Prediction

Characteristics:

• Sequential processing: Pipelines involve a sequence of tasks that are executed one after the other
• Data flow: Pipelines typically involve the flow of data from one task to the next
• Configurable: Pipelines can be configured to accommodate different tasks, intents, or AI models

Putting it all Together

Compound AI systems integrate multiple tasks, intents, and pipelines to create a comprehensive AI solution. By combining these components, developers can build AI systems that:

• Handle complex tasks: Compound AI systems can tackle complex tasks that involve multiple intents, tasks, and pipelines
• Improve accuracy: By integrating multiple AI components, compound AI systems can improve overall accuracy and performance
• Enhance user experience: Compound AI systems can provide a more seamless and intuitive user experience by understanding user intents and adapting to their needs

In summary, tasks, intents, and pipelines are the building blocks of compound AI systems. By understanding these concepts and how they interact, developers can create sophisticated AI solutions that tackle complex challenges and provide real value to users.

Key Components of a Compound AI System

A Compound AI system typically includes:

• Language Models: Understand and respond to user inputs in a natural way
• Information Retrievers: Fetch relevant data from databases or external sources
• Databases: Store and manage data for easy access
• External Tools: Use APIs and services to perform specific tasks
• Data Converters: Turn data into a format that’s easy to search and retrieve
• Result Filters: Prioritize and evaluate retrieved results for relevance
• Prompt Builders: Create effective inputs for language models
• Output Refiners: Ensure generated outputs are high-quality and coherent
• Caching Systems: Store frequently accessed responses for faster retrieval
• Task Managers: Orchestrate tasks and data flow for optimal performance
• Evaluation Tools: Assess system performance and output quality
• Monitoring and Feedback: Track performance and gather user feedback for improvement
• Model Adapters: Fine-tune pre-trained models for specific tasks or domains
• Agent Builders: Create autonomous agents that can perform tasks and make decisions

These components work together to create a powerful and adaptable Compound AI system.

Designing Compound AI Systems

In summary here’s a brief summary of a system approach to Application Development Life-cycle:

Analysis → Design → Development → Production → Monitoring

To simplify:

01. Identify Intents

• Define possible intent based on some user requirements
• Define intent dependencies

02. Identify Tools

• Tools you may need to accomplish a task. Like web search, APIs, code execution etc.

03. Build the Chain

• Build a workflow based on identified tasks
• Architecture you may need to solve the problem, like RAG, Text2SQL, etc.
• Iterate the process

An example: 3 main intents

1. Intent — retrieval/search piece: Taking those requests, responses from that search
2. Building the augmentation: Grabbing those summaries and building the context
3. Generating answers: Once we have the context, sending to the model then generating answers

Components

The Compound AI System is a modular, scalable, and flexible architecture designed to integrate multiple AI components to achieve complex tasks. This architecture enables the creation of a robust and adaptable AI system that can be applied to various domains.

Illustration of key components of a compound AI system:

• Data Ingestion Layer
• AI Component Layer
• Knowledge Graph
• Orchestration Layer
• Decision Support System
• Continuous Learning and Improvement

Challenges of Compound AI Systems

Building

• Complexity in integrating multiple components
• Managing multiple models and processing steps
• Adapting to different hardware configurations
• Ensuring robust metrics and logging systems

Optimizing

• Balancing latency, throughput, and resource utilization
• Co-optimizing system components
• Cost optimization while maintaining performance
• Harmonizing interactions among components

Deploying

• Meeting diverse hardware, software, and scalability requirements
• Managing operational complexity with MLOps and DataOps
• Ensuring scalability and elasticity
• Integrating with existing infrastructure
• Addressing security, privacy, and explainability concerns

These challenges highlight the complexity and nuance of building, optimizing, and deploying Compound AI systems.

Further Reading

• The Shift from Models to Compound AI Systems
• Compound AI Systems - Databricks Glossary

Compound AI Systems represent the future of AI - where multiple specialized components work together to achieve what no single model can accomplish alone.