• Cactus runs Google DeepMind's FunctionGemma at up to 3000 toks/sec prefill speed on M4 Macs.
• While decode speed reaches 200 tokens/sec, all without GPU, to remain energy-efficient.
• FunctionGemma is great at tool calling, but small models are not the smartest for some tasks.
• There is a need to dynamically combine edge and cloud (Gemini Flash) to get the best of both worlds.
• Cactus develops various strategies for choosing when to fall back to Gemini or FunctionGemma.

• FunctionGemma is just a tool-call model, but tool calling is the core of agentic systems.
• You MUST design new strategies that decide when to stick with on-device or fall to cloud.
• You will be objectively ranked on tool-call correctness, speed and edge/cloud ratio (priortize local).
• You can focus on prompting, tool description patterns, confidence score algorithms, anything!
• Please ensure at least 1 team member has a Mac, Cactus runs on Macs, mobile devices and wearables.

• Step 1: Fork this repo, clone to your Mac, open terminal.
• Step 2: git clone https://github.com/cactus-compute/cactus
• Step 3: cd cactus && source ./setup && cd .. (re-run in new terminal)
• Step 4: cactus build --python
• Step 5: cactus download google/functiongemma-270m-it --reconvert
• Step 6: Get cactus key from the cactus website
• Sept 7: Run cactus auth and enter your token when prompted.
• Step 8: pip install google-genai
• Step 9: Obtain Gemini API key from Google AI Studio
• Step 10: export GEMINI_API_KEY="your-key"
• Step 11: Click on location to get Gemini credits - SF, Boston, DC, London, Singapore, Online
• Step 12: Join the Reddit channel, ask any technical questions there.
• Step 13: read and run python benchmark.py to understand how objective scoring works.
• Note: Final objective score will be done on held-out evals, top 10 are then judged subjectively.

• Your main task is to modify the internal logic of the generate_hybrid method in main.py.
• Do not modify the input or output signature (function arguments and return variables) of the generate_hybrid method. Keep the hybrid interface compatible with benchmark.py.
• Submit to the leaderboard python submit.py --team "YourTeamName" --location "YourCity", only 1x every 1hr.
• The dataset is a hidden Cactus eval, quite difficult for FunctionGemma by design.
• Use python benchmark.py to iterate, but your best score is preserved.
• For transparency, hackers can see live rankings on the leaderboard.
• Leaderboard will start accepting submissions once event starts.
• The top hackers in each location will make it to judging.

• Rubric 1: The quality of your hybrid routing algorithm, depth and cleverness.
• Rubric 2: End-to-end products that execute function calls to solve real-world problems.
• Rubric 3: Building low-latency voice-to-action products, leveraging cactus_transcribe.

import json from cactus import cactus_init, cactus_complete, cactus_destroy model = cactus_init("weights/lfm2-vl-450m") messages = [{"role": "user", "content": "What is 2+2?"}] response = json.loads(cactus_complete(model, messages)) print(response["response"]) cactus_destroy(model)

model = cactus_init("weights/lfm2-vl-450m") model = cactus_init("weights/lfm2-rag", corpus_dir="./documents")

# Basic completion messages = [{"role": "user", "content": "Hello!"}] response = cactus_complete(model, messages, max_tokens=100) print(json.loads(response)["response"])

# Completion with tools tools = [{ "name": "get_weather", "description": "Get weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"] } }] response = cactus_complete(model, messages, tools=tools) cactus_complete(model, messages, callback=on_token)

Response format (all fields always present):

{ "success": true, "error": null, "cloud_handoff": false, "response": "Hello! How can I help?", "function_calls": [], "confidence": 0.85, "time_to_first_token_ms": 45.2, "total_time_ms": 163.7, "prefill_tps": 619.5, "decode_tps": 168.4, "ram_usage_mb": 245.67, "prefill_tokens": 28, "decode_tokens": 50, "total_tokens": 78 }

Cloud handoff response (when model detects low confidence):

{ "success": false, "error": null, "cloud_handoff": true, "response": null, "function_calls": [], "confidence": 0.18, "time_to_first_token_ms": 45.2, "total_time_ms": 45.2, "prefill_tps": 619.5, "decode_tps": 0.0, "ram_usage_mb": 245.67, "prefill_tokens": 28, "decode_tokens": 0, "total_tokens": 28 }

• When cloud_handoff is True, the model's confidence dropped below confidence_threshold (default: 0.7) and recommends deferring to a cloud-based model for better results.

• You will NOT rely on this, hackers must design custom strategies to fall-back to cloud, that maximizes on-devices and correctness, while minimizing end-to-end latency!

whisper = cactus_init("weights/whisper-small") prompt = "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>" response = cactus_transcribe(whisper, "audio.wav", prompt=prompt) print(json.loads(response)["response"]) cactus_destroy(whisper)

embedding = cactus_embed(model, "Hello world") print(f"Dimension: {len(embedding)}")

Reset model state (clear KV cache). Call between unrelated conversations.

cactus_reset(model)

Stop an ongoing generation (useful with streaming callbacks).

cactus_stop(model)

Free model memory. Always call when done.

cactus_destroy(model)

Get the last error message, or None if no error.

error = cactus_get_last_error() if error: print(f"Error: {error}")

Query RAG corpus for relevant text chunks. Requires model initialized with corpus_dir.

model = cactus_init("weights/lfm2-rag", corpus_dir="./documents") chunks = cactus_rag_query(model, "What is machine learning?", top_k=3) for chunk in chunks: print(f"Score: {chunk['score']:.2f} - {chunk['text'][:100]}...")

• Join the Reddit channel, ask any technical questions there.
• To gain some technical insights on AI, checkout Maths, CS & AI Compendium.