LLM orchestration with multi-provider failover

Arka’s LlmFallbackEngine manages all LLM calls across 25 providers with automatic failover, API key rotation, and per-task or per-skill model overrides.

Supported providers

Anthropic, OpenAI, Gemini, Groq, xAI, DeepSeek, Moonshot, Z.AI, MiniMax, Venice, Mistral, Cohere, Together, Fireworks, Perplexity, Hugging Face, OpenRouter, Sakana Fugu, Bedrock, Azure, LiteLLM, Ollama, LM Studio, vLLM, and vLLM Cloud.

How failover works

The orchestrator builds a candidate list in order:

AI_PREFERRED_PROVIDER + AI_PREFERRED_MODEL (if set)
Live model lists from each provider API (when keys allow)
Built-in chain: Gemini 2.5/2.0 Flash → Groq Llama 3.3/3.1 → Ollama cloud/local

On 429, 401, or timeout, Arka tries the next model. API key rotation (API_KEY_ROTATION=1) cycles backup keys before switching providers.

AI_PREFERRED_PROVIDER=gemini
AI_PREFERRED_MODEL=gemini-2.0-flash
LLM_AUTO_FALLBACK=1
API_KEY_ROTATION=1

Per-task profiles

Task	Used for
`route`	Skill routing decisions
`summarize`	Email and document summaries
`chat`	Conversational replies
`research`	Deep web research
`agent`	Goal agent steps
`pdf`	PDF question answering
`predictions`	Stock and forecast analysis
`compose_video`	Video script generation
`compose_slides`	Presentation slide script generation

Override per task:

LLM_FALLBACK_ROUTE=gemini/gemini-2.0-flash,groq/llama-3.3-70b-versatile
LLM_FALLBACK_SUMMARIZE=groq/llama-3.3-70b-versatile

Per-skill models

arka ai-skill-model web_answer groq/llama-3.3-70b-versatile
arka ai-skill-model profiles
arka ai-models

Or edit ~/.config/arka/llm-skill-models.json.

Benchmark orchestration

If you want model selection to follow live benchmark winners, run a benchmark suite first and then enable orchestration:

arka benchmark run
arka benchmark show
ARKA_BENCHMARK_ORCHESTRATE=1 arka orchestrate --benchmark "how to get free ai credits"

The benchmark profile follows the routed task, so a request that routes to web_answer uses the chat benchmark results, while agent-style requests use agent.

Shared exhaustion cache

One session cache — exhausted models are skipped across skills until reset:

arka ai reset-exhaustion

Enable stderr notification on failover: LLM_FALLBACK_NOTIFY=1. Debug attempts: LLM_VERBOSE=1.

Local LLM servers

OLLAMA_HOST=127.0.0.1:11434
LLM_AUTO_START_SERVERS=1
LLM_AUTO_STOP_SERVERS=1

Why skills stay concise

Arka does not ask the LLM to memorize 70 skill manuals. Skills are local programs — the model only picks the right name. On LLM route, Arka passes:

A compact skill catalog (registered names only)
Curated routing rules (high-signal patterns)
Shell aliases via ROUTE_ALIASES for your machine

The route task uses low temperature and a dedicated chain so a fast/cheap model handles classification; heavy lifting runs inside the skill locally.

Inspect at runtime

arka ai-models
arka ai models --task summarize
arka ai models --skill web_answer
arka ai active-model
arka ai reset-exhaustion

Set SHOW_MODEL=1 (default) to display Model: provider/name under answers.

​Supported providers

​How failover works

​Per-task profiles

​Per-skill models

​Benchmark orchestration

​Shared exhaustion cache

​Local LLM servers

​Why skills stay concise

​Inspect at runtime

​Related

Supported providers

How failover works

Per-task profiles

Per-skill models

Benchmark orchestration

Shared exhaustion cache

Local LLM servers

Why skills stay concise

Inspect at runtime

Related