Case Study
Open-Jet
Live demoA terminal coding agent that runs on your own GPU. It connects to Ollama, LM Studio or llama.cpp, or picks a model that fits your card and runs llama-server for you. Runs Qwen 27B at 70 tokens/s on an RTX 3090, up from 33, after tuning multi-token prediction, KV cache and GPU offload. Install with pip install open-jet.
Outcomes
Impact
- Reads files, edits code, and runs commands fully offline. The Claude Code workflow without the cloud.
- Collapses a weekend of backend, runtime, and agent setup into a single install. Shipped on PyPI.
Stack
Tools
Decisions
Key decisions
- Most agentic coding systems still depend on cloud connectivity for orchestration or inference, which breaks in offline and restricted environments.
- Running an OS-level LLM agent directly on self-owned edge hardware enables local device control without sending code, logs, or shell state to external services.
- Security posture improves when execution, model weights, and tool outputs stay local, with explicit approval gates for mutating actions.
Approach
Technical approach
- A setup wizard profiles target hardware, tunes GPU layer offload, and recommends model sizing.
- Inference runs through local llama-server using quantized GGUF models from local files or Ollama pulls.
- The Textual TUI orchestrates chat, slash commands, file mentions, and tool execution requests.
- Session events and resource telemetry are written as structured logs for observability and replay.
Links