Application reference. This is an illustrative architecture and scope checklist, not a verified customer project or measured deployment. Technologies, integration scope and delivery items are agreed for the actual brief.
Scenario and Constraints
A business wanting AI answering over its own product documentation, policies or support history. The requirements that actually decide the architecture are: answers must cite their source, the system must say it does not know rather than invent an answer, cost per conversation must be bounded, and quality must be measurable before and after any prompt or model change.
Architecture
Layers and responsibilities
| Layer | Technology | What it is responsible for |
|---|---|---|
| Ingestion | Python workers | Document parsing, chunking, embedding and metadata extraction. Runs as a pipeline so a re-index is a routine operation, not a rebuild. |
| Vector store | PostgreSQL + pgvector | Chunk embeddings alongside relational metadata. One database instead of a separate vector service keeps permissions and backups simple at this scale. |
| Retrieval | Go service | Hybrid search: vector similarity plus keyword matching, then reranking. Keyword recall is what catches exact part numbers and codes that embeddings routinely miss. |
| Answering | Go service calling the model API | Prompt assembly with retrieved context, citation enforcement, streaming response, token budget per request. |
| Guardrails | Application layer | Scope check, refusal when retrieval confidence is low, PII filtering, escalation to a human queue. |
| Admin | Vue admin panel | Document management, re-index triggers, conversation review, evaluation set and cost dashboard. |
Data Model
The tables below carry the rules that everything else depends on. Getting these wrong is what turns a working demo into a system that cannot be extended.
Core entities
| Entity | Key fields, relations and rules |
|---|---|
| document / version | Source document with version history and effective dates, so an answer can be traced to the document revision that produced it. |
| chunk | Text segment with embedding vector, heading path, token count and a pointer back to its document and position — the position is what makes a citation link to the right place rather than the whole file. |
| access_scope | Which roles or tenants may retrieve a chunk. Filtering happens inside the retrieval query, never after generation, otherwise restricted content can leak into an answer before the check runs. |
| conversation / message | Turn history with retrieved chunk ids, model used, token counts and latency per message, which is what makes cost and quality attributable per conversation. |
| feedback | Thumbs, corrections and escalation flags per answer, feeding the evaluation set instead of disappearing into a log file. |
| eval_case | Question, expected source and acceptance note. Run against every prompt or model change so regressions are caught before release rather than by customers. |
Key Interfaces
Endpoints that carry the business rules
| Endpoint | Purpose and concurrency notes |
|---|---|
| POST /api/v1/ingest | Queues a document for parsing, chunking and embedding; idempotent per (document, version) so a re-submission replaces rather than duplicates. |
| POST /api/v1/chat | Streams an answer. Applies the access scope to retrieval, caps context tokens, enforces citations, and returns a refusal when retrieval confidence is below the configured threshold. |
| POST /api/v1/chat/{id}/escalate | Hands the conversation with full context to a human queue — the release valve that keeps a bounded system honest instead of forcing it to answer everything. |
| POST /api/v1/eval/run | Runs the evaluation set against the current configuration and reports pass rate and citation accuracy per case. |
| GET /api/v1/usage | Token and cost accounting per tenant and per period, the basis for rate limits and budget alerts. |
Technical Decisions
- Hybrid retrieval, not vector-only. Vector-only retrieval may miss exact identifiers such as part numbers, error codes or article references. Compare retrieval approaches on the actual evaluation questions. Keyword matching covers that gap and reranking merges the two.
- Citations enforced, not requested. Asking a model politely to cite sources is unreliable. Answers are validated against retrieved chunk ids and rejected when they cite nothing, which is the difference between a system you can trust and a plausible-sounding one.
- Refusal threshold on retrieval confidence. A confident wrong answer costs more than no answer. Below the threshold the system says it does not know and offers escalation.
- pgvector rather than a dedicated vector database. At document volumes typical for a single business, one PostgreSQL instance handles both vectors and metadata, keeping operations, permissions and backup to one system. A dedicated store earns its complexity only at much larger scale.
- Evaluation set from the first week. Without it, every prompt change is a guess and regressions are discovered by users. It is the cheapest quality control in an AI project and the most commonly skipped.
- Model access behind our backend. API keys never reach the client, and rate limiting, budget caps and model switching stay server-side configuration rather than an app release.
Hypothetical Design Targets
The figures below are hypothetical capacity-planning examples, not measured client results, a validated sizing recommendation or an SLA. Set project-specific acceptance targets after defining the workload, devices, data, hardware and third-party dependencies; verify them through a pilot and load tests before committing to a production target.
Capacity and performance targets
| Target | Value and assumption |
|---|---|
| First token latency | Define a hypothetical first-token budget, then benchmark it with the chosen provider, retrieval pipeline and evaluation set. Provider latency is outside the application's control; streaming can show progress without promising a response time. |
| Retrieval | Under 200 ms for hybrid search plus reranking on a corpus in the low hundreds of thousands of chunks, with an appropriate pgvector index. |
| Cost per conversation | Bounded by a context token cap and a per-tenant budget with alerting; the actual figure depends on model choice and conversation length and is measured during the pilot. |
| Answer quality | Tracked as pass rate and citation accuracy on the evaluation set. The baseline is established during the pilot rather than promised in advance. |
What Delivery Includes
- Ingestion pipeline, retrieval and answering service source, and admin panel source in your repository
- Database schema including chunk, scope and evaluation tables, plus index configuration
- Prompt templates and the evaluation set, both version-controlled with the code
- Cost and usage dashboard, rate limit and budget alert configuration
- Deployment notes, key management procedure, and a documented model-switching path
Planning This for Your Business
The modules above are a starting point, not a fixed package. We normally cut the first release to one core workflow plus one admin workflow, then extend once real usage shows where the load and the edge cases actually are. Send a scoped brief and we will map it against this architecture and tell you which parts you do not need yet.