For years, IT departments have wanted to bring AI on-premise to keep sensitive data like proprietary code, financial records, and employee logs safe. Until recently, hosting capable AI meant spending millions on server farms. Thanks to software optimisations, that barrier has dropped.
As an IT professional managing self-hosted infrastructure, I now deploy open-source models like Gemma 4 31B right onto our company servers. It works brilliantly for daily tasks like document summarisation. The catch? Standalone performance still lags behind frontier cloud models, and buying massive GPUs to close the gap is a tough business sell. So, for complex workloads, we often default back to paid cloud APIs.
However, local AI does not need to beat frontier cloud models outright to deliver value. By using Mixture of Agents (MoA) orchestration, an internal server running smaller models for reference, synthesis, and routing becomes surprisingly capable, flexible, and cost-effective.
Why local models work (and where they fail)
Running a local model gives me total data sovereignty, zero per-token API fees, and offline reliability. A medium-sized, “Goldilocks” 31B-parameter model easily handles routine administrative work.
While server maintenance, power use, and hardware depreciation are factors, the main limitation is deep reasoning. A local model handles routine emails or system logs fine, but it stumbles on complex tasks, like scanning a massive codebase for security vulnerabilities while matching compliance rules. Traditionally, my choices were accepting weaker results or abandoning local setups for cloud APIs.
Enter mixture of agents (MoA)
Popularised in 2024 by Together AI, MoA is not a SaaS product or a software library; it is an architectural methodology used during inference. Instead of relying on one model, it breaks workflows into a collaborative system:
- Reference models (proposers): Multiple models independently answer the same prompt.
- The aggregator: A final, capable model evaluates the answers, synthesises the data, and writes the definitive response.
- Zero fine-tuning: This collaboration happens on the fly with no extra model training.
Tools like Sakana AI’s Fugu and Hermes Agent have productised this setup to save developers from building complex routing logic from scratch. Fugu acts as a multi-agent orchestrator behind a single API, while Hermes Agent lets you configure an aggregator to consult reference models via a virtual provider.
Does it work?
Yes, but it comes with trade-offs:
- The good: A panel of models reliably beats a single model. Hermes Agent benchmarks show a two-model MoA panel outperforms either model individually, and Fugu hits state-of-the-art benchmark results for coding and reasoning.
- The bad: MoA scales up token costs across multiple model calls and introduces latency, stretching response times from seconds to minutes. Managed tools like Fugu can also feel like a black box when auditing which model contributed what.
What this looks like in my company servers setup
In my environment, Gemma 4 handles summaries well but struggles with architectural trade-offs. To fix this without moving everything to the cloud, I added GPT-5.6-luna as an external reference model. I route complex queries out to GPT first, then use local Gemma 4 as the aggregator to synthesise the response. (Note: Sending prompts to external models breaks pure local privacy boundaries, so handle data carefully.)
Using Hermes Agent, I configured this hybrid stack in a simple config.yaml file, setting my local Ollama server as aggregator_model and adding the cloud provider details under reference_models:
- Reference model: GPT API (~US0.2/M input tokens, US1.2/M output tokens)
- Aggregator Model: Local Gemma 4 31B via Ollama ($0 per-token fee)
The evidence
Nous Research tested MoA on HermesBench, demonstrating clear performance gains:
| Configuration | HermesBench Score |
|---|---|
| Claude Opus 4.8 alone | 0.7607 |
| GPT-5.5 alone | 0.7412 |
| MoA (Opus aggregator + GPT-5.5 reference) | 0.8202 |
While this test used frontier models, the principle applies locally: pairing an aggregator with cheaper reference models beats running the aggregator alone.
Where MoA fits (and where it does not)
- Where it excels: Complex tasks like architecture planning, creative brainstorming, or deep code reviews where synthesising distinct perspectives catches edge cases.
- Where it struggles: Fast, simple tasks like basic summaries, fact retrieval, or real-time chatbots. Weak reference models can also introduce hallucinations that confuse the aggregator.
Final thoughts
MoA isn’t about replacing enterprise cloud models; it is about making your local stack smarter, more resilient, and more capable. If your local setup is almost good enough, testing an MoA configuration is a practical next step. Just keep realistic expectations regarding latency, context limits, and API budgets.
References & further reading
- Wang, J. et al. (2024). Mixture-of-Agents Enhances Large Language Model Capabilities. arXiv:2406.04692
- Together AI. Together MoA — collective intelligence of open-source modelsTogether MoA — collective intelligence of open-source models.
- Sakana AI. Sakana Fugu — Multi-Agent System as a Model & Technical Report (arXiv:2606.21228).
- Xu et al. TRINITY: An Evolved LLM Coordinator. ICLR 2026.
- Nielsen, C. et al. Learning to Orchestrate Agents in Natural Language with the Conductor. ICLR 2026. arXiv:2512.04388
- Nous Research. Mixture of Agents & Profiles: Running Multiple Agents (Hermes Agent Documentation).
Want to explore how Mixture of Agents could work in your local setup? Get in touch with us at SRC Innovations. We can assess your environment, identify practical MoA use cases, and design an approach that balances capability, privacy, and cost. Get it touch!