**Data Persistence Strategies — navigating relational, NoSQL, and vector-based storage architectures.** Data is the asset clients trust us with, and the choices we make about where it lives, how it's shaped, and how it evolves outlive every application that touches it. This page describes how we approach the database landscape — relational, document, key-value, graph, time-series, vector, and the data warehouse — and how to match the store to the workload.
The database is no longer one thing. Twenty years ago, "the database" meant a relational database, with a schema, with SQL. Today, a single application typically uses four or five distinct stores — relational for transactional truth, key-value for sessions, search index for full-text, document store for flexible schemas, message broker for event streams, vector store for embeddings, time-series for telemetry, graph for relationship traversal, data warehouse for analytics. Pretending one store fits all workloads forces every workload into the same shape, which is wrong for most of them. Choosing the right store per workload — polyglot persistence — is the architectural commitment that makes modern data platforms work.
The cost is real: more stores mean more operational surface, more skills, more integration. The discipline is choosing deliberately, bounding the polyglot, and treating the integration between stores as architecturally significant.
Most architectural decisions are revisable. A framework can be replaced over a year. A microservice can be rewritten. An infrastructure provider can be changed. But the database — once it holds production data, with schemas evolved over years, with operational tooling built around it, with a team that knows its quirks — is the architectural decision that sticks. Migrating from PostgreSQL to MongoDB, from Cassandra to a relational store, from one cloud's managed service to another, is multi-quarter work that disrupts the team and risks the data. The implication is that database choices deserve a longer evaluation horizon than other architectural decisions, and the criteria should weight long-term operability and team familiarity more heavily than short-term feature fit.
Martin Kleppmann — Designing Data-Intensive Applications — the canonical reference for the long-horizon thinking that database choices require, including the migration patterns and operational properties that distinguish stores in practice.
The database schema is the longest-lived artefact in any application — the code changes monthly, the team changes annually, the schema changes daily and persists for years. A schema migration that requires downtime, a rollback that loses data, a column rename that breaks every consumer — these are operational events that interrupt every release. The mature alternative is an evolution discipline: additive changes only, dual-running readers and writers during transitions, contraction phases that remove deprecated columns long after no consumer reads them. Done well, the schema evolves continuously and invisibly; done poorly, every schema change is a coordination event the whole organisation feels.
Martin Kleppmann — Designing Data-Intensive Applications, Chapter 4 — the canonical treatment of evolvability of data, including backward and forward compatibility patterns that scale to systems with many independently-deployed consumers.
Vector databases (Pinecone, Weaviate, Milvus, pgvector) are not magic — they are indexes over high-dimensional embedding spaces, with the same lifecycle concerns as any other index. The embeddings need to come from somewhere (which model, with which dimensions, normalised how), they need to stay current (when the source data changes, the embedding must too), they need to be tuned for retrieval (the right ANN algorithm, the right recall/latency trade-off). Treating "we put it in Pinecone" as the architecture, without thinking about the embedding lifecycle, produces semantic search that quietly degrades as the source data drifts, and the team only notices when a customer reports irrelevant results.
Pinecone — Vector Database Guide — a practical industry treatment of vector indexing, ANN trade-offs, and the lifecycle considerations that distinguish working from failing semantic search.
Time-series workloads (metrics, telemetry, IoT, financial ticks) have specific structural properties — high write throughput, append-only, queries that aggregate by time window, retention that decreases for older data, compression that exploits temporal locality. General-purpose databases handle these workloads, but at significant cost: storage that grows unbounded, queries that slow down as data accumulates, expensive aggregations that re-scan months of data. Purpose-built time-series databases (InfluxDB, TimescaleDB, ClickHouse) optimise for the structural pattern: chunked storage, automatic downsampling, retention policies, columnar compression. Choosing the time-series structure deliberately — even when the data volume seems small — is what prevents the painful migration two years later when it isn't.
TimescaleDB — Time-series data architecture — an industry treatment of why general-purpose stores fail at time-series workloads and what the structural pattern actually requires.
Some queries — multi-hop relationship traversals, shortest path, community detection, recursive ancestry — get exponentially harder in relational form because each hop is a join, and three hops produce a query that scans most of the database. Graph databases (Neo4j, Amazon Neptune, ArangoDB) optimise for these traversals natively; what takes minutes in SQL takes milliseconds in a graph store. The architectural insight is to recognise the graph workload before forcing it into relational form — fraud detection, recommendation engines, organisational hierarchies, supply-chain dependencies, social networks all have graph DNA that's expensive to fight.
Neo4j — Graph Database Concepts — the canonical introduction to graph data modelling, query patterns, and the workloads where graph stores fundamentally outperform relational alternatives.
Operational databases (the OLTP path) hold today's truth — the current balance, the in-flight transaction, the latest customer record. The data warehouse (the OLAP path) holds yesterday's truth, organised for analysis — historical patterns, aggregations, longitudinal cohorts. They are different stores for different questions, with different freshness, different cardinality, and different workloads. Modern lake-house architectures (Databricks, Snowflake, Microsoft Fabric) blur the boundary, but the principle remains: the operational path optimises for transaction throughput, the analytical path optimises for retrospective query, and the pipeline that moves data between them is its own architectural concern with its own SLOs.
Bill Inmon — Building the Data Warehouse — the foundational text that defined the OLTP/OLAP distinction; modern lake-house architectures (Snowflake, Databricks, Microsoft Fabric) are the contemporary evolution.
The diagram below shows a canonical polyglot data architecture: an operational tier with relational and document stores; a key-value cache for hot data; a search index for full-text; a vector store for semantic retrieval; a message broker carrying changes between stores; a data warehouse for analytics fed by a deliberate ETL/ELT pipeline.
Forcing every workload into the relational database the team started with. Sessions, full-text search, time-series telemetry, and analytical queries all live in Postgres because "we already have Postgres." Each workload underperforms; the database becomes a bottleneck no amount of vertical scaling can fix.
Polyglot by deliberate choice. Each workload uses the store whose engineering matches its access pattern; the integration cost is paid in exchange for proper performance characteristics.
Schema changes coupled to deployments, requiring downtime, requiring all consumers to upgrade simultaneously. Releases become coordination exercises; the team avoids schema work because the cost is so high.
Continuous additive evolution. Online schema change tools, dual-run periods, contraction phases. Schema changes happen weekly or daily, invisibly, without ceremony.
Embeddings stored in a vector database without lifecycle thinking — when the embedding model is upgraded, when source records change, when the index needs to be rebuilt. Search quietly degrades as the index drifts from the source of truth.
Treat embeddings as a versioned, lifecycle-managed dependency. Re-embed on source changes, rebuild on model upgrades, monitor for drift, audit relevance regularly.
High-throughput time-series data in a normalised relational table with a timestamp column. Storage grows unbounded, queries slow as data accumulates, retention is manual and unreliable.
Use time-series stores (TimescaleDB, ClickHouse, InfluxDB) for time-series workloads. Retention, downsampling, and chunked storage come built in; storage and query performance stay predictable.
Reporting, BI dashboards, and analytical queries hitting the operational database directly. Long-running queries lock tables, degrade transaction throughput, and produce stale dashboards anyway because the data hasn't been shaped for analysis.
A deliberate ETL/ELT pipeline to a separate analytical store. The OLTP path stays fast for transactions; the OLAP path is shaped for the questions analysts actually ask.
Polyglot is deliberate, not accidental; the choice survives review by an engineer who didn't make it; the rationale is current to the workload as it exists today, not as it was when the system launched.
ALTER TABLE downtime is rare and avoidable; dual-run windows let consumers migrate at their own pace; contraction phases remove deprecated columns long after no consumer reads them.
Vectors are an index over a source of truth; the index stays current; embedding-model upgrades trigger documented rebuild procedures; no mixed-version embeddings in production.
Retention, downsampling, and chunked storage come from the platform, not from custom cleanup jobs; query performance stays bounded as data accumulates.
Multi-hop queries, recursive traversals, fraud-pattern detection, supply-chain analysis — these are recognised as graph problems and routed to graph stores or extensions.
Reporting does not hit production; the pipeline has SLOs and is monitored; analytical freshness is documented and known to consumers.
Backup integrity is verified through routine restore drills to a separate environment; the team has actually done this, not just configured it.
Async replication has lag; the team knows the lag bounds; replicas are not used for transactions that require read-your-writes consistency.
Connection-limit exhaustion is a common outage cause; explicit pool sizing per service prevents one runaway service from starving the rest.
When data flows from Postgres to Elasticsearch, from Postgres to a warehouse, from event streams to read models — the consistency window, the failure modes, the reconciliation strategy are explicit.
Other substantive pages in the library that link here: