com.harryaskham.talos · Kotlin · C++

Talos

An OpenAI-compatible model server that runs on an Android phone. It downloads models from Hugging Face, loads each one when a request first names it, keeps it while it is in use and unloads it after an idle timeout. With nothing loaded, it costs one foreground service with a listening socket: no inference process, timer, wake lock or GPU context.

It serves chat and completions, embeddings and reranking, speech recognition and synthesis, spoken replies, music and images, on port 8484 by default.

Talos's Logs tab: CPU and memory graphs, the gateway and one model slot running SmolLM2 135M on the CPU, and log lines showing the gateway listening on 127.0.0.1:8484 and a cold chat completion request answered in 15.3 seconds.
The Logs tab after one cold request.

Processes and engines

A gateway and one process per loaded model Talos's processes. A foreground service holds the listening socket, port 8484 by default, and serves the OpenAI-compatible API. When a request names a model that is not loaded, Talos starts a separate slot process for it with the right engine: llama.cpp for GGUF language models, LiteRT-LM for .litertlm bundles, audio.cpp for speech and music, or stable-diffusion.cpp for images. Unloading a model ends its process, so all of its native and GPU memory goes back to Android. With nothing loaded, Talos is just the service and its socket. any OpenAI clientphone or network gatewayforeground service:8484 · OpenAI API :slot1 · llama.cppGGUF chat · embeddings :slot2 · LiteRT-LM.litertlm on the GPU :slot3 · audio.cppspeech · music stable-diffusion.cppimages not loaded
Unloading ends the process. Up to four models can be resident at once, each in its own slot.
EngineModelsRuns on
llama.cpp (a pinned fork)GGUF language models, including 1- and 2-bit ternary packsCPU, Vulkan, OpenCL with Adreno kernels, Hexagon NPU
LiteRT-LM 0.17.1.litertlm bundlesGPU, CPU, NPU
audio.cpp 0.9.1speech, spoken replies, musicCPU only
stable-diffusion.cppsmall image modelsCPU, Adreno OpenCL
A model's life The lifecycle of one model. It starts unloaded, with no process. The first request for it loads it into a new slot process; a streaming client receives keep-alive events while it loads, and the request is answered once the model is ready. The model stays resident while requests keep coming, and after an idle timeout Talos ends its process. A model can also be loaded or unloaded by hand from the app, a widget or the admin API. unloadedno process loadingSSE keep-alive residentanswers requests evictedprocess ended first request idle timeout

Measured on an Adreno 830

Language models, single-thread host probe
ModelBackendPrefillDecode
Gemma 4 E2B Q4_0OpenCL143.5 tok/s12.3 tok/s
Gemma 4 E2B Q4_0Vulkan5.9 tok/s6.0 tok/s
Bonsai 4B PQ2_0Vulkan5.0 tok/s12.1 tok/s
Bonsai 4B PQ2_0OpenCL0.4 tok/s0.3 tok/s
Bonsai 2 27B PTQ1_0Vulkan0.8 tok/s2.8 tok/s

These were measured on a Snapdragon 8 Elite phone over SSH through a proot linker, so they are lower bounds rather than app benchmarks. They decide Talos's automatic placement: OpenCL for standard quantisations, Vulkan for ternary packs, with Vulkan's fused kernels turned off because Adreno's driver crashes in them.

They also say what a phone is good for. An agent's system prompt and tool schemas are often over 10,000 tokens, and at these prefill rates a ternary model on Vulkan takes many minutes to read them, and the 27B model hours. For agents, a LiteRT-LM bundle on the GPU or a small standard GGUF on OpenCL is the practical choice, with the client's context window set to the model's real one.

For speech, Qwen3-ASR 0.6B transcribed 11 seconds of speech in 8.1 seconds with its encoder on the GPU. Qwen3-TTS 1.7B produced 1.8 to 3.0 frames a second (12.5 frames make one second of audio) with its audio stages on the CPU, against 0.04 with everything on OpenCL, so speech models keep those stages on the CPU by default.

Beyond chat

  • Speech: transcription, synthesis with saved voices for cloning (importable from a Git repository, Hugging Face or a folder), and native spoken replies.
  • Music: short stereo clips from a prompt, which can continue earlier audio.
  • Images: generation and masked edits with small models.
  • Typed decisions: /v1/systemone answers yes/no, choice and score questions about a piece of text with probabilities. These are next-token scores, not calibrated confidences, and LiteRT-LM is not supported because it does not expose token probabilities.
  • Logs: two minutes of CPU, GPU and memory per Talos process, and request metadata without prompts, outputs or audio. Sampling only runs while the tab is on screen, and a value Android will not reveal shows as "—" rather than an estimate.
Talos's Music tab with the prompt 'Steady funk groove', duration and step sliders, and a continuous-generation option.
Music in the playground.
# after downloading the Gemma 4 E2B preset in the Models tab
curl http://127.0.0.1:8484/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model": "gemma-4-e2b-gpu", "stream": true,
       "messages": [{"role": "user", "content": "hi"}]}'

The listener can bind to 127.0.0.1, all interfaces or one interface. API keys are optional and separate from the address; the admin routes need an admin key from other devices unless explicitly opened.

Limits