Cross-platform & energy-efficient kernels, runtime and AI inference engine for mobile devices.
┌─────────────────┐ │ Cactus FFI │ ←── OpenAI compatible C API for integration (tools, RAG, cloud handoff) └─────────────────┘ │ ┌─────────────────┐ │ Cactus Engine │ ←── High-level transformer engine (NPU support, INT4/INT8/FP16/MIXED) └─────────────────┘ │ ┌─────────────────┐ │ Cactus Models │ ←── Implements SOTA models using Cactus Graphs └─────────────────┘ │ ┌─────────────────┐ │ Cactus Graph │ ←── Unified zero-copy computation graph (think NumPy for mobile) └─────────────────┘ │ ┌─────────────────┐ │ Cactus Kernels │ ←── Low-level ARM-specific SIMD operations (think CUDA for mobile) └─────────────────┘ #include cactus.h CactusGraph graph; auto a = graph.input({2, 3}, Precision::FP16); auto b = graph.input({3, 4}, Precision::INT8); auto x1 = graph.matmul(a, b, false); auto x2 = graph.transpose(x1); auto result = graph.matmul(b, x2, true); float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f}; float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}; graph.set_input(a, a_data, Precision::FP16); graph.set_input(b, b_data, Precision::INT8); graph.execute(); void* output_data = graph.get_output(result); graph.hard_reset(); #include cactus.h cactus_set_pro_key(""); // email founders@cactuscompute.com for optional key cactus_model_t model = cactus_init( "path/to/weight/folder", // section to generate weigths below "txt/or/md/file/or/dir/with/many", // nullptr if none, cactus does automatic fast RAG ); const char* messages = R"([ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "My name is Henry Ndubuaku"} ])"; const char* options = R"({ "max_tokens": 50, "stop_sequences": ["<|im_end|>"] })"; char response[4096]; int result = cactus_complete( model, // model handle from cactus_init messages, // JSON array of chat messages response, // buffer to store response JSON sizeof(response), // size of response buffer options, // optional: generation options (nullptr for defaults) nullptr, // optional: tools JSON for function calling nullptr, // optional: streaming callback fn(token, id, user_data) nullptr // optional: user data passed to callback );Example response from Gemma3-270m
{ "success": true, // when successfully generated locally "error": null, // returns specific errors if success = false "cloud_handoff": false, // true when model is unconfident, simply route to cloud "response": "Hi there!", // null when error is not null or cloud_handoff = true "function_calls": [], // parsed to [{"name":"set_alarm","arguments":{"hour":"10","minute":"0"}}] "confidence": 0.8193, // how confident the model is with its response "time_to_first_token_ms": 45.23, // latency (time to first token) "total_time_ms": 163.67, // total execution time "prefill_tps": 1621.89, // prefill tokens per second "decode_tps": 168.42, // decode tokens per second "ram_usage_mb": 245.67, // current process RAM usage in MB "prefill_tokens": 28, "decode_tokens": 50, "total_tokens": 78 }- Models: LFM2-VL-450m & Whisper-Small
- Precision: Cactus smartly blends INT4, INT8 and F16 for all weights.
- Decode = toks/sec, P/D = prefill/decode, VLM = 256×256 image, STT = 30s audio
- Cactus Pro: Uses NPU for realtime and large context (Apple for now), scores are marked with *
| Device | Short Decode | 4k-P/D | VLM-TTFT | VLM-Dec | STT-TTFT | STT-Dec |
|---|---|---|---|---|---|---|
| Mac M4 Pro | 170 | 989/150 | 0.2s/0.1s* | 168 | 0.9s/0.2s* | 92 |
| Mac M3 Pro | 140 | 890/123 | 0.3s/0.1s* | 149 | 1.5s/0.4s* | 81 |
| iPad/Mac M4 | 134 | 603/106 | 0.3s/0.1s* | 129 | 1.8s0.3s* | 70 |
| iPad/Mac M3 | 117 | 525/93 | 0.4s/0.1s* | 111 | 2.8s/0.7s* | 61 |
| iPhone 17 Pro | 126 | 428/84 | 0.5s/0.1s* | 120 | 3.0s/0.6s* | 80 |
| iPhone 16 Pro | 106 | 380/81 | 0.6s/0.2s* | 101 | 4.3s/0.7s* | 75 |
| iPhone 15 Pro | 90 | 330/75 | 0.7s/0.3s* | 92 | 4.5s/0.8s* | 70 |
| Galaxy S25 Ultra | 80 | 355/52 | 0.7s | 70 | 3.6s/- | 32 |
| Nothing 3 | 56 | 320/46 | 0.8s | 54 | 4.5s | 55 |
| Pixel 6a | 25 | 108/24 | 2.3s | 25 | 9.6 | 15 |
| Raspberry Pi 5 | 20 | 292/18 | 1.7s | 23 | 15s | 16 |
- Cactus smartly and compactly blends INT4, INT8 and F16 for all weights.
- You can still quantize everything with one precision, but mixed is optimal
| Model | Zipped Size | Completion | Tools | Vision | Embed | Speech | Pro |
|---|---|---|---|---|---|---|---|
| google/gemma-3-270m-it | 252MB | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| google/functiongemma-270m-it | 252MB | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| openai/whisper-small | 283MB | ✗ | ✗ | ✗ | ✓ | ✓ | Apple |
| LiquidAI/LFM2-350M | 244MB | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| LiquidAI/LFM2-VL-450M | 448MB | ✓ | ✗ | ✓ | ✓ | ✗ | Apple |
| nomic-ai/nomic-embed-text-v2-moe | 451MB | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| Qwen/Qwen3-0.6B | 514MB | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| Qwen/Qwen3-Embedding-0.6B | 514MB | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| LiquidAI/LFM2-700M | 498MB | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| google/gemma-3-1b-it | 642MB | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| LiquidAI/LFM2.5-1.2B-Instruct | 474MB | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| LiquidAI/LFM2-1.2B-RAG | 474MB | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| LiquidAI/LFM2-1.2B-Tool | 474MB | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
| openai/whisper-medium | 658MB | ✗ | ✗ | ✗ | ✓ | ✓ | Apple |
| LiquidAI/LFM2.5-VL-1.6B | 954MB | ✓ | ✗ | ✓ | ✓ | ✗ | Apple |
| Qwen/Qwen3-1.7B | 749MB | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
git clone https://github.com/cactus-compute/cactus && cd cactus && source ./setup-
[model]is a HuggingFace name from the table above (default:google/gemma-3-270m-it) - Common flags:
--precision INT4|INT8|FP16(default: INT4),--token <hf_token> - Always run
source ./setupin any new terminal.
| Command | Description |
|---|---|
cactus run [model] | Opens playground (auto downloads model) |
cactus download [model] | Downloads model to ./weights |
cactus convert [model] [dir] | Converts model, supports LoRA merging (--lora <path>) |
cactus build | Builds for ARM (--apple or --android) |
cactus test | Runs tests (--ios / --android, --model [name/path]) |
cactus clean | Removes build artifacts |
cactus --help | Shows all commands and flags |