Private local AI system

ROS

A persistent, local-first personal AI system built so that identity, memory, permissions and project context survive even when the underlying model or inference machine changes.

Applied AIPythonPostgreSQLLocal LLMsLinuxmTLSAndroid

The problem

A useful assistant should outlive any one model.

I wanted one private assistant that could retain project context, study work, homelab knowledge, preferences and operating history without turning a single model checkpoint into the source of truth.

That led to a system architecture where models are replaceable execution capabilities. Durable identity, memory, permissions, tasks, audit and policy remain under the Core and PostgreSQL control plane.

Architecture

Control, inference and private data stay separate.

CLIENTSWeb · Android · CLIinterfaces, not authority
↓
CONTROL PLANEROS Core + PostgreSQLidentity · memory · policy · audit · sessions
read-only / bounded capabilities
PRIVATE DATAFiles / live evidence / toolsminimum-disclosure access
TLS 1.3 mTLS
INFERENCE PLANEDedicated GPU workerlocal model policy · private Ollama

Public diagram is intentionally sanitized: internal addressing, credentials and low-level network topology are omitted.

Implemented evidence

More than a local chat wrapper.

Control plane

Persistent

Python + PostgreSQL own durable conversations, projects, preferences, policy, audit and system state.

Inference

Local GPU

A dedicated inference worker serves replaceable local models while Ollama remains private to the worker.

Evaluation

20-model study

A fixed local model tournament compared quality, latency, memory pressure and structured-output behavior.

Model policy

Role-based

Five exact-digest model roles separate interactive, fallback, heavy reasoning, coding and slow-path use.

Transport

TLS 1.3 mTLS

The control plane reaches the inference worker through a narrowly scoped authenticated transport rather than a broad model API.

Client

Native Android

The native foreground Android client source is merged and being rebuilt chat-first; it uses canonical Core sessions, AndroidKeyStore device proof and explicit tap/PTT voice. Signed physical-device acceptance remains a separate gate.

Evaluation

Model choice was treated as an engineering decision.

20fixed candidate models in the local tournament

The benchmark compared more than raw generation speed. It tracked semantic quality, structured-output behavior, preload cost, GPU residency, host-memory pressure, swap and sustained responsiveness.

The result was deliberately not “pick the highest score.” ROS uses a role-based policy because the fastest interactive model, stronger heavy-reasoning models and the slowest maximum-quality candidate occupy different points on the local quality/latency/resource frontier.

interactivefast fallbackheavy reasoningcodingslow / offline

Real benchmark evidence

The model tournament left reproducible artifacts, not just conclusions.

This is a repository-tracked comparison from the historical Work1 evaluation lane. The production inference executor later moved to a dedicated GPU VM, but the measured tournament remains the evidence base that shaped the role policy.

ROS P2A generation throughput comparison across the fixed 20-model evaluation set
Project-owned benchmark artifact: the same fixed 20-model lock run in fresh-per-model and reverse-order conditions. Throughput was only one dimension; quality, structured output and memory pressure were evaluated separately.

Key decisions

The architecture is defined as much by what it refuses to do.

01

Memory is not model weights.

Changing facts, project state, preferences and permissions live in structured durable state. Models remain replaceable computation.

02

One assistant, multiple execution capabilities.

Inference nodes, storage and tools do not become separate personalities. ROS keeps one persistent identity while capabilities remain separately permissioned.

03

No universal root credential.

Assistant, code, ops, database, deployment and file capabilities are deliberately separated. Owner recovery authority is not handed to ROS.

04

Measure before upgrading.

Model and hardware decisions are made from measured quality, latency and resource behavior on actual hardware instead of theoretical specifications.

05

No silent cloud fallback.

Private memory and prompts remain local by default. Degraded local capability is reported honestly instead of silently routing private work elsewhere.

06

Voice is explicit foreground input.

The Android client deliberately avoids always-on listening, background capture and hidden audio persistence. Text remains available when voice degrades.

Current client direction

A phone is an interface, not a second brain.

The Android client source is merged and is being rebuilt as a chat-first native foreground app. It connects to the same canonical Core conversations and device authority rather than maintaining a mobile shadow store.

Device proof uses a non-exportable key in AndroidKeyStore. Voice is explicit tap/PTT input, requests microphone permission only when invoked, and does not introduce a wake phrase, background microphone, hidden cloud fallback or raw audio archive.

What I owned

Cross-layer decisions, not just implementation tickets.

  • Product intent and system boundaries
  • Persistent-state vs inference architecture
  • Model evaluation and role-policy decisions
  • Security, capability and network boundaries
  • Release, rollback and operational acceptance
  • Client/product direction and day-to-day dogfooding

AI-assisted workflow

I use ChatGPT and Codex extensively for implementation, research, debugging, audits and documentation. My role is to define the product and architecture, set constraints and acceptance criteria, make trade-offs and verify the resulting system through tests, CI and live operational evidence.

Current limitations

Deliberately bounded, not finished.

ROS is a private local-runtime system, not a public SaaS or hyperscale service. Current inference is bounded by a single local GPU, one active generation at a time and an evidence-backed 4096-token context policy; larger-context or concurrent operation requires new evidence.

The Android client remains in active product-finish work, with signed physical-device acceptance separate from source acceptance. Fine-tuning, broad autonomous operations and silent cloud fallback remain intentionally out of scope.

Stack / delivery

Local-runtime software with exact-release discipline.

Core

Python + PostgreSQL

Durable conversations, projects, preferences, policy, provenance and audit remain outside model weights.

Inference

Ollama + NVIDIA GPU

Exact-digest local model roles run behind a dedicated worker instead of exposing a general-purpose model API.

Security

mTLS + proof-bound devices

TLS 1.3 mTLS protects the inference path; device trust and capability authority remain separately bounded.

Delivery

Git + CI + exact SHA

Runtime changes are tied to tested source identity, host evidence, explicit acceptance and a retained rollback path.

Honest boundary

What this project does—and does not—claim.

ROS is evidence of applied AI systems engineering, local inference, model evaluation, persistent-state architecture, security boundaries and product integration.

It is not evidence that I train foundation models, operate hyperscale GPU clusters, or solved general autonomous-agent safety. Fine-tuning is deliberately deferred until stable interaction evidence justifies it.

Next case study

Rikaha — diagnostic reasoning as an evaluated product system.

Back to selected work