llmspark is running llmspark — Ollama-shaped CLI, llama.cpp engine Usage: llmspark serve llmspark list llmspark ps llmspark pull NAME [--fresh] # Qwen/Qwen3.8-27B ; --fresh starts from 0 llmspark clear-pull [NAME] # delete saved/partial download llmspark run NAME [PROMPT] # load bar, then stream think+answer char by char llmspark stop [NAME] llmspark rm NAME llmspark show NAME llmspark version # llama.cpp + llmspark llmspark update # git pull + rebuild llama-server llmspark reset llmspark restart llmspark purge llmspark docs # integration guide (progress bars, models, chat) Same HTTP API as Ollama on :11435 GET / homepage + this help + integration guide GET /docs integration guide only (for Cursor / tools) GET /api/tags list installed models GET /api/ps what is in RAM + context_length POST /api/load load model (NDJSON progress) POST /api/chat chat (auto-loads if needed; "tools", "think": false) POST /v1/chat/completions OpenAI passthrough (tools, tool_choice, thinking, sampling) POST /api/pull download from Hugging Face (NDJSON progress) POST /api/reset kill llama-server POST /api/purge kill + drop_caches llmspark integration guide — for Cursor and any tool ===================================================== Read this before wiring llmspark into an app, CLI, Open WebUI, Continue, or a custom UI. What it is Ollama-shaped HTTP API + CLI. The engine is llama.cpp llama-server, NOT Ollama. Base URL: http://127.0.0.1:11435 Do NOT use :11433 or :11434 — those are real Ollama / ollama-ctl. Leave them alone. Do NOT kill Ollama when resetting llmspark. Live docs GET http://127.0.0.1:11435/ GET http://127.0.0.1:11435/docs GET http://127.0.0.1:11435/api/docs CLI: llmspark docs -------------------------------------------------------------------- RULES -------------------------------------------------------------------- 1. Streaming routes default to stream=true unless body has "stream": false /api/chat /api/generate /api/pull /api/load /api/update 2. Stream response: HTTP chunked, Content-Type: application/x-ndjson One JSON object per line. Parse UTF-8 by LINE, never byte-by-byte (Hebrew / emoji will break if you decode one byte at a time). 3. Non-stream JSON (tags, ps, version, reset): Content-Length + Connection: close. 4. reset / stop / purge must work even while a model is loading. Do not queue them. 5. Model names come from GET /api/tags (the "name" field). Use that exact string. 6. First chat auto-loads the model. For a visible load bar, call /api/load first. 7. Only ONE model in RAM. Loading a different model while one is loaded/loading returns HTTP 409 {"error":"...","code":"load_refused"}. Stop/reset first. 8. Load is also refused (409) if MemAvailable < model file size + ~4GB reserve. 9. Chat streams thinking and answer character-by-character. Append each chunk immediately. Do not wait for done=true to show text. 10. Tools go in "tools" (OpenAI shape), never in the prompt text. The chat template renders them and llama.cpp returns structured tool_calls. /v1/chat/completions forwards the whole OpenAI body (tools, tool_choice, chat_template_kwargs, sampling, ...). See TOOLS / THINKING / SAMPLING below. -------------------------------------------------------------------- PROGRESS BARS -------------------------------------------------------------------- Always POST with "stream": true. Read NDJSON until status=success or error. LOAD — POST http://127.0.0.1:11435/api/load Body: {"model":"NAME","stream":true} Events: {"status":"loading","line":"human text from llama-server"} {"status":"loading","total":BYTES,"completed":BYTES} ← draw the bar from this {"status":"success","line":"llama_server: model loaded (ctx=N)"} {"status":"success","already_loaded":true,"line":"..."} {"status":"error","error":"...","code":"load_refused"} another model already in RAM, or not enough MemAvailable HTTP 409 when not streaming How to draw (same as `llmspark run`): if event has "completed": pct = 100 * completed / total one line, rewritten in place with \r; fixed-width bar (~30 cells) show: loading 42.3% ############------------------ 12GB/28GB (llmspark's own CLI uses smooth unicode blocks: █▋░) ignore "line" for the bar; you may print it above or skip it. On success: newline, then "llama_server: model loaded". Timeout: up to 900s for a ~87GB GGUF. PULL — POST http://127.0.0.1:11435/api/pull Body: {"name":"unsloth/GLM-5.3-Flash-GGUF:UD-IQ1_S","stream":true,"fresh":false} GGUF only. Split models (NAME-00001-of-00006.gguf) are downloaded as one set. llama.cpp loads the first shard; siblings stay in the same folder. Not pulled: NVIDIA / TensorRT / safetensors / non-GGUF. Those return an error. Events: {"status":"pulling manifest"} {"status":"pulling N file(s) (SIZE)"} ← once, then ONE bar for the whole set {"status":"downloading","total":ALL_BYTES,"completed":M,"speed":...,"eta":..., "file":2,"files":3} ← file/files only when the GGUF is split into shards {"status":"success","digest":"..."} {"status":"error","error":"..."} Draw ONE \r bar from completed/total. Do not print a new line per shard or Xet chunk. "fresh":true deletes a partial download first. UPDATE — POST http://127.0.0.1:11435/api/update Body: {"stream":true} Events: {"status":"...","line":"cmake output"} then {"status":"success"} -------------------------------------------------------------------- CHAT -------------------------------------------------------------------- POST http://127.0.0.1:11435/api/chat Body: { "model": "NAME", "messages": [{"role":"user","content":"hello"}], "stream": true } Optional: "options": {"temperature":0.7,"top_p":0.8,"top_k":20,"num_predict":2048} "think": false (Ollama-style; see THINKING below) "tools": [...] (Ollama-style; see TOOLS below) Each line (not done) is ONE character — flush and append immediately: thinking: {"message":{"role":"assistant","content":"","thinking":"א"},"done":false} answer: {"message":{"role":"assistant","content":"ב","thinking":""},"done":false} Tool calls (only when the request had "tools"), one line before done: {"message":{"role":"assistant","content":"","tool_calls":[ {"id":"...","function":{"index":0,"name":"get_weather","arguments":{"city":"Tel Aviv"}}}]},"done":false} Last line: {"message":{"role":"assistant","content":""},"done":true,"done_reason":"stop",...} If load is blocked: {"error":"...","code":"load_refused"} UI (like GPT): if message.thinking: append to a dim "Thinking..." buffer as chars arrive if message.content: close thinking (once), then append answer chars never buffer the whole think/answer before painting /api/generate: same, field is "response" (one char per line), no thinking. OpenAI-compatible: POST http://127.0.0.1:11435/v1/chat/completions (SSE, forwarded with low latency) Recommended client flow: 1) POST /api/load {model, stream:true} → show load bar until success 2) POST /api/chat {model, messages, stream:true} → paint think + answer live -------------------------------------------------------------------- TOOLS (function calling) — use the template, not the prompt -------------------------------------------------------------------- POST http://127.0.0.1:11435/v1/chat/completions is a FULL OpenAI passthrough to llama-server --jinja. Send "tools" and "tool_choice" exactly as with OpenAI. The model's own chat template renders the tool catalog; llama.cpp constrains the output with a grammar and parses it. Do NOT paste a JSON block into the system prompt and do NOT regex the text — you get structured calls back: stream: choices[0].delta.tool_calls[{index,id,function:{name,arguments}}] (arguments arrive as string fragments; join them per index) final chunk: finish_reason "tool_calls" no-stream: choices[0].message.tool_calls "tool_choice": "auto" | "none" | "required" "required" is enforced by the grammar — the model cannot end the turn with plain text. Use it on action turns (edit / build) that must produce a call. "parallel_tool_calls": true|false is honoured when the template supports it. FINISH TOOL — 100% coverage: "required" on EVERY turn, still able to end. Problem: with "required" the model can never say "done, I changed X and Y" — that is text without a tool — so agents only force the first few turns. Fix (as in Cline / Roo): a tool called "finish" with one argument "summary". llmspark adds it for you and converts the call back into a normal answer, so the UI sees exactly what it sees today: request: "tool_choice": "required" (+ your tools) ← that is all llmspark: appends finish({summary}) to "tools" (LLMSPARK_FINISH_TOOL=auto) model: must call one of your tools, or finish to end response: finish's summary arrives as delta.content, word by word while the model is still writing it; finish_reason "stop"; no tool_call is visible. Other tool calls in the same turn pass through unchanged. "Installed. Now building the site..." + EOS without a tool is impossible — the grammar does not allow it — at any point of the run, not just the start. Put the summary in history as a plain assistant message (as you do today). Control: "finish_tool": false → never (plain "required": model cannot end) "finish_tool": true → add it even when tool_choice is "auto" "finish_tool": "report_result"→ your own tool of that name; llmspark converts its "summary" string argument to text LLMSPARK_FINISH_TOOL=auto|off server default (auto = on every "required") LLMSPARK_FINISH_TOOL_NAME=... rename the injected tool If your tools already contain a tool named "finish", llmspark leaves it alone unless you pass "finish_tool": true. Continue the loop with OpenAI-shaped history: the assistant message that carries "tool_calls", then one {"role":"tool","tool_call_id":"...","content":"..."} per call. All of these fields reach the template untouched. /api/chat accepts the same "tools" (Ollama shape). Tool calls come back as message.tool_calls with "arguments" already parsed to an object. Does this model's template support tools? GET /api/ps → "capabilities": ["completion","tools","thinking"], "chat_template_caps": {...} Copy-paste check (action turn: forced tool call, no thinking): curl -N http://127.0.0.1:11435/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model":"NAME","stream":true,"tool_choice":"required", "chat_template_kwargs":{"enable_thinking":false}, "messages":[{"role":"user","content":"Weather in Tel Aviv? Use the tool."}], "tools":[{"type":"function","function":{"name":"get_weather", "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}' → data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"name":"get_weather","arguments":"{"}}]}}]} ... arguments fragments ... finish_reason "tool_calls" -------------------------------------------------------------------- THINKING (reasoning) per request -------------------------------------------------------------------- Thinking is on by default for models whose template supports it. Turn it off for a single request — same effect on both routes: /api/chat: "think": false /v1/chat/completions: "chat_template_kwargs": {"enable_thinking": false} or "reasoning_effort": "none" or "think": false (llmspark extension) Effort levels (gpt-oss style templates): "reasoning_effort": "low"|"medium"|"high" Token budget: "reasoning_budget_tokens": 512 (llama.cpp; -1 = unlimited) Agent tip: a small model plans inside , then "has already thought" and emits one short sentence + EOS. Keep thinking for planning turns; send think:false (or reasoning_effort:"none") on action turns (edit / build). Server-wide default for requests that carry "tools": LLMSPARK_TOOL_THINK=auto (template decides) | off | on current: off "off" covers every tool turn with no client code. A request that says think:true / enable_thinking:true still wins (use it on planning turns). -------------------------------------------------------------------- SAMPLING defaults (only fill what the client did not send) -------------------------------------------------------------------- Qwen's recommendation; llama.cpp's own default (temp 0.8, top_p 0.95) raises the chance of a random EOS right after one complete sentence. thinking off: temperature 0.7 top_p 0.8 top_k 20 min_p 0.0 thinking on: temperature 0.6 top_p 0.95 top_k 20 min_p 0.0 with "tools": presence_penalty 1.0 (loops in quantized models) Anything you send explicitly wins (OpenAI names on /v1, "options" on /api/chat). Override the defaults in the service environment: LLMSPARK_TEMP LLMSPARK_TOP_P LLMSPARK_TOP_K LLMSPARK_MIN_P LLMSPARK_THINK_TEMP LLMSPARK_THINK_TOP_P LLMSPARK_THINK_TOP_K LLMSPARK_THINK_MIN_P LLMSPARK_TOOL_PRESENCE_PENALTY (0 disables) -------------------------------------------------------------------- MODEL MANAGEMENT -------------------------------------------------------------------- List disk: GET http://127.0.0.1:11435/api/tags → {"models":[{"name","size","digest","modified_at"}]} What's in RAM: GET http://127.0.0.1:11435/api/ps → {"models":[{"name","size","context_length","processor", "capabilities":["completion","tools","thinking"], "chat_template_caps":{...}}]} empty list = nothing loaded (or still starting) Show one: POST http://127.0.0.1:11435/api/show {"name":"NAME"} Copy alias: POST http://127.0.0.1:11435/api/copy {"source":"A","destination":"B"} Delete file: DELETE http://127.0.0.1:11435/api/delete {"name":"NAME"} Stop current: POST http://127.0.0.1:11435/api/generate {"model":"NAME","keep_alive":0,"prompt":""} or POST http://127.0.0.1:11435/api/reset Kill engine: POST http://127.0.0.1:11435/api/reset (always succeeds; unloads even mid-load) Free RAM: POST http://127.0.0.1:11435/api/purge (reset + drop_caches if sudo -n works) Status: GET http://127.0.0.1:11435/api/ctl Versions: GET http://127.0.0.1:11435/api/version Clear pull: POST http://127.0.0.1:11435/api/pull-clear {"name":"NAME"} CLI equivalents llmspark list | ps | pull NAME | run NAME | stop | rm NAME | show NAME llmspark reset | restart | purge | version | update | docs -------------------------------------------------------------------- HOW LOADING WORKS (so you do not fight it) -------------------------------------------------------------------- CTX is auto: one llama-server start with --fit on, min 8192 tokens, ~4GB RAM left for the desktop, KV cache q8_0, 1 parallel slot. After load, read context_length from /api/ps or the success line (ctx=N). Override: LLMSPARK_CTX=65536 or LLMSPARK_CACHE_TYPE=f16 Guard: refuses a second model while one is loaded, and refuses if MemAvailable < GGUF size + reserve (~4GB). HTTP 409 / code=load_refused. Switch models: POST /api/reset (or llmspark stop) then /api/load. -------------------------------------------------------------------- MINIMAL CLIENT (Python) -------------------------------------------------------------------- import json, urllib.request def ndjson(req): with urllib.request.urlopen(req) as resp: for raw in resp: if raw.strip(): yield json.loads(raw) def load(name): body = json.dumps({"model": name, "stream": True}).encode() req = urllib.request.Request("http://127.0.0.1:11435/api/load", data=body, method="POST") for ev in ndjson(req): if ev.get("error"): raise RuntimeError(ev["error"]) if ev.get("completed") is not None: print(f"\rloading {ev['completed']}/{ev.get('total') or 0}", end="") print() def chat(name, text): body = json.dumps({ "model": name, "messages": [{"role": "user", "content": text}], "stream": True, }).encode() req = urllib.request.Request("http://127.0.0.1:11435/api/chat", data=body, method="POST") for ev in ndjson(req): msg = ev.get("message") or {} if msg.get("thinking"): print(msg["thinking"], end="") if msg.get("content"): print(msg["content"], end="") print() -------------------------------------------------------------------- PORTS / DO NOT TOUCH -------------------------------------------------------------------- 11435 llmspark (this tool) 11433 real Ollama 11434 ollama-ctl 18088 inner llama-server (127.0.0.1 only) -------------------------------------------------------------------- FUNNEL (public https://HOST.tailXXXX.ts.net) -------------------------------------------------------------------- Tailscale Funnel proxies the public hostname to http://127.0.0.1:11435. A network drop (lost IPv4, DERP unreachable) takes Funnel down until Tailscale rebinds. That cannot be served from the internet *during* the outage. After the LAN/WAN is back, a user timer (every 30s) runs funnel-watch.sh: - restart llmspark if :11435 is dead - re-apply `tailscale funnel --bg 11435` if the serve config vanished - `tailscale debug rebind` (+ `tailscale up` if the node looks logged out) when the backend is not Running, or https://HOST:443 via the Tailscale IP does not answer Status: systemctl --user status llmspark-funnel-watch.timer Log: ~/.local/share/llmspark/funnel-watch.log