TOJI TOJI
ARCHITECTURE · brain

brain architecture

brain is an on-prem secure data layer replacing SpiceDB and RDBs, and the neunexus ecosystem's RAG (retrieval-augmented generation) + ontology service — a Rust gRPC service that chunks and embeds documents into a single embedded store (SurrealDB) combining vector, graph, and full-text search. Internal neunexus-ecosystem services like whoami and cogito use it for grounded (RAG) search and for syncing entity relationships (assigned_to/child_of/blocks/depends_on and the like — an ontology-style graph). It fuses dense (usearch) and sparse (BM25) hybrid search via RRF, and also handles code-repo indexing and cross-document traceability.

Project overview →

Why

00

What this project is, and why it exists

authored from source

brain is an on-prem secure data layer replacing SpiceDB and RDBs. A client ingests documents over gRPC into SurrealDB (embedded, a vector+graph+FTS multi-model store), and embedding calls an in-cluster Ollama (bge-m3).

brain overview: client calls brain via gRPC, brain embeds via Ollama and reads/writes SurrealDB

brain / overview.yaml

01

Who reaches brain, and how

deploy/k8s/deployment.yaml → BRAIN_AUTH_SERVICE_SUBS

The real callers were confirmed from the deployment manifest's BRAIN_AUTH_SERVICE_SUBS (trusted-service list) — five in total: brain-iim, cogito-svc (jaso grounded search), whoami-svc (entity sync), brain-viewer (metaviewer proxy), and oikonomos-session (asset-registry ingest).

brain context: 5 real callers (brain-iim, cogito-svc, whoami-svc, brain-viewer, oikonomos-session) call brain via gRPC; brain calls Ollama for embedding and an IdP for JWKS

brain / context.yaml

02

How the Rust modules are decomposed internally

src/lib.rs + main.rs

Reflects src/lib.rs's flat mod list and main.rs's actual service registration order. An auth interceptor (JWT/JWKS) gates four services (ingest/search/access/graph); only extraction connects directly, without the interceptor.

brain component structure: main gRPC server routes through an auth interceptor to ingest/search/access/graph services, extraction bypasses auth, all converging on tenant/authz/vector/SurrealDB

brain / structure.yaml

03

Which cluster, which path it ships through

deploy/k8s/*.yaml

The key difference from goquest: there is no Ingress in deploy/k8s/. brain is cluster-internal only (gRPC, reached via in-cluster DNS) with no external load-balancer path.

brain network topology: in-cluster callers reach a ClusterIP Service routing to the brain Deployment, which persists to a PVC and mounts ConfigMaps

brain / network.yaml

04

What structure the data is stored in

proto/brain.proto + tenant.rs (전체 확인)

brain has no relational schema (SurrealDB embedded, multi-model doc+graph+KV) — a full read of tenant.rs (2183 lines) plus the test suite found the real table set (15 tables) and 7 edge types with confirmed from/to pairs. Five more (assigned_to/child_of/blocks/depends_on/duplicates) are schema-defined but their target table isn't fixed in code — the caller supplies it at runtime — so they aren't drawn as arrows.

brain logical data model: 15 SurrealDB tables including document, chunk, term, user, query_log, repo-scoped code index tables, and dimension hub tables, connected by 7 confirmed edge types (can_view, part_of, mentions, mentions_term, references, created_by, synonym_of)

brain / erd.yaml

05

How a commit becomes a deployment

toji-ci/ci-repos/brain.yaml

Only facts confirmed in the toji-ci config are shown — manifest writeback (toji-cd bots/brain) is defined in CI, and tests run inside the Docker build stage before the build itself.

brain CI/CD pipeline: git push triggers toji-ci Tekton, which builds+tests in a Docker stage, pushes to the registry, and writes manifests back to toji-cd ArgoCD, which syncs to K8s ns:brain

brain / cicd.yaml

06

Where secrets come from, and how they reach a pod

deploy/k8s + Tekton

Unlike goquest, brain has no app-level secrets (DB passwords, embedding API keys) — SurrealDB is embedded and the Ollama call is unauthenticated. Vault only syncs two things: registry pull credentials and a Tekton geneso read token.

brain secrets flow: a ServiceAccount authenticates to VSO which syncs Vault registry credentials into an image pull secret, and a separate Tekton secret carries a geneso read token

brain / secrets.yaml

07

Which layers a single request passes through

main.rs (tonic Server builder)

Reflects main.rs's actual tonic Server wiring — four services pass through a gRPC-Web + JWT interceptor, while ExtractionService bypasses the interceptor entirely (unauthenticated, stated explicitly in the startup log).

brain API request pipeline: gRPC request passes through a JWT interceptor to a handler, tenant resolution, and SurrealDB; ExtractionService bypasses the interceptor entirely

brain / api-layers.yaml

08

How the gRPC services are grouped

proto/brain.proto

The real six service blocks from proto/brain.proto — IngestService, SearchService, AccessService, GraphService, ExtractionService — with every rpc name confirmed directly in code.

brain gRPC service groups: brain.proto branches into IngestService, SearchService, AccessService, GraphService, and ExtractionService, each listing its real rpc names

brain / api-endpoints.yaml

09

Where a document-ingest request goes

src/ingest.rs → IngestDocument

Summarizes the real code path of the IngestDocument RPC in src/ingest.rs. archview has no shape concept, so steps are approximated as color-coded boxes, and the original 11 steps are folded into 7 for readability.

brain ingest dataflow: an IngestDocument request flows through chunking, embedding, and extraction before being written to SurrealDB

brain / dataflow-ingest.yaml

rendered via archview · d2lang/d2 · ELK layout