Knowing the history of databases is not scholarly trivia: it is the most efficient way of understanding why today's tools are the way they are. Every design decision that now looks obvious —that tables have no order, that a declarative language exists, that the database guarantees integrity— was at the time an answer to a painful, concrete problem. And every model that looks obsolete today was, for years, the best solution available.

This lesson covers sixty years of evolution following the thread of the problems, not of the dates. By the end you will understand why the relational model was a break with the past, why SQL outlived every fashion, what real pressures gave rise to NoSQL and why the outcome of all this is not a single winner but an ecosystem in which every option keeps its niche.

1. Before databases: flat files and batch processing
2. The first models: hierarchical and network
3. 1970: Codd's paper and the relational break
4. System R, Ingres and the birth of SQL
5. The commercial era and the standardisation of SQL
6. The 2000s: the web, the ORM and the scaling problem
7. The NoSQL movement and the pressures behind it
8. NewSQL and distributed databases
9. The cloud: managed and serverless databases
10. The recent chapter: vector databases
11. Complete timeline
12. The lessons history leaves us
13. Common mistakes and tips
14. Exercises
15. Conclusion

The fifties and sixties. Computers exist, but storage is magnetic tape: a strictly sequential medium. To read record number 8,000 you have to go past the previous 7,999 first.

That conditions everything. Work is organized as batch processing: during the day, transactions pile up on paper or punched cards; at night, a program runs through the master tape from beginning to end, applies the changes and writes a new master tape. There is no such thing as "check the balance now": there is this morning's printout.

The data lives in flat files, and each application has its own, in its own format defined inside the program's code. The problems this creates are exactly the ones we diagnosed in BiblioRed's spreadsheet in lesson 01-01, multiplied:

• Massive redundancy: payroll, accounting and HR each keep their own copy of the employee's data.
• Guaranteed inconsistency: one copy gets updated and the others do not.
• Total dependency between program and data: if a field is added to the file, you have to recompile every program that reads it, even those that never use that field. The physical format is written inside the code.
• No ad hoc queries: any new question requires a programmer to write a new program. Weeks of waiting to find out how many members the North branch has.

The arrival of the direct-access disk in the mid-sixties changes the equation: for the first time it is feasible to go straight to record 8,000. That opens the door to online, interactive processing, and with it to the very idea of a database management system: a common piece of software, separate from the applications, that centralizes the data.

IBM develops IMS (Information Management System) for the Apollo programme, to manage the bill of materials of the Saturn V rocket: millions of parts organized into subassemblies. A naturally hierarchical problem, and the model reflects it: the data is organized in a tree, each record has one parent and may have several children.

It was an enormous success —IMS is still in production today in banks and insurers, sixty years later— but its limitation is structural:

• A child can only have one parent. A many-to-many relationship cannot be represented without duplicating data. In BiblioRed, a book with two authors forces you to repeat the book under each author, or the author under each book.
• The programmer has to navigate the tree explicitly, from the root and in the prescribed order.
• Changing the structure of the tree breaks the programs.

The CODASYL consortium —the same one that standardised COBOL— defines the network model, of which IDMS is the best-known product. Charles Bachman, its main driving force, receives the Turing Award in 1973.

The network model generalises the hierarchical one: a record can belong to several sets, through explicit pointers. Many-to-many relationships can now be represented.

But the underlying problem remains, and it is the one that gives meaning to everything that follows:

The programmer has to know and traverse the physical storage structure. To answer "which books has Marta read" you have to write a program that opens the member set, locates Marta, follows the pointer to the first loan, moves on to the next, and so on. How to get to the data is the responsibility of whoever is programming.

This means there is no data independence: if the administrator reorganises the storage to gain performance, every program stops working. Information systems become impossible to evolve. That is the exact point at which Codd steps in.

In June 1970, Edgar Frank Codd, a British mathematician working at IBM's San Jose laboratory, publishes "A Relational Model of Data for Large Shared Data Banks" in Communications of the ACM. Thirteen pages that reorganise the entire discipline.

The proposal, in essence:

1. Data is represented as relations (tables): sets of tuples (rows) with attributes (columns). Nothing more. No pointers, no hierarchies, no implicit ordering.
2. Relationships between data are expressed through values, not through pointers: if a loan refers to member 14, it does so by storing the value 14, not a disk address.
3. Access is declarative: the user describes what they want, not how to get it. The system decides the route. Codd proposes two equivalent formalisms for this, relational algebra and relational calculus.
4. Mathematical foundation: by resting on set theory and predicate logic, the properties of the model are provable, not a matter of opinion.

Why was it a break and not an incremental improvement?

The practical consequence is data independence: you can change how something is stored without touching how it is queried. That is what turns a database into a lasting investment. We will formalise this concept in lesson 01-04 and study the relational model in detail in 02-01.

Codd received the Turing Award in 1981. It is striking that IBM itself took years to bet on his idea: it had IMS selling very well and little interest in cannibalising it. The usual criticism at the time was that the relational model would be too slow, precisely because it left the choice of access path to the system rather than the programmer. Proving that criticism unfounded was the work of the following decade.

Two research projects, running in parallel, turn the theory into working software.

The prototype with which IBM proves that a relational system can be fast. Its contributions survive to this day in any database you use:

• The cost-based optimizer: the system estimates the cost of several possible execution plans and picks the best one. It is the piece that makes declarative access viable, and it is the reason you can write a query today without thinking about how it will be executed.
• The transaction manager with an operation log (write-ahead log) and two-phase locking, the basis of what would later be called ACID (lesson 06-01).
• The SEQUEL language (Structured English Query Language), designed by Donald Chamberlin and Raymond Boyce, meant to be readable by non-mathematicians. A trademark conflict led to it being renamed SQL.

Compare the intent with what came before:

Against the equivalent in a CODASYL system, which would be a program dozens of lines long opening sets and following pointers. That leap in expressiveness, together with the optimizer that made it competitive on speed, is what won the argument.

Michael Stonebraker and Eugene Wong build Ingres with public funding and distribute it, source code included, to universities. It uses its own language, QUEL, technically very elegant, which would eventually lose out to SQL for commercial rather than technical reasons.

Its legacy is enormous by another route: out of Ingres and the people who passed through it come Sybase, Microsoft SQL Server, Informix and, above all, Postgres (1986), Stonebraker's successor project, which years later would adopt SQL and be renamed PostgreSQL. The manager we will use in this course descends directly from that university project. Stonebraker received the Turing Award in 2014.

In 1979, a small company called Relational Software launches Oracle V2, the first commercial RDBMS with SQL, beating IBM itself to it (IBM releases SQL/DS in 1981 and DB2 in 1983). The company would end up being called Oracle Corporation.

The eighties consolidate the model: Informix (1980), Sybase (1984), and in the PC world dBase and later Access. The nineties bring the free alternatives that democratise access: MySQL (1995), PostgreSQL under its current name (1996) and, in 2000, SQLite, which packs a complete relational engine into a file under a megabyte.

Meanwhile, SQL is standardised. This is the evolution worth understanding:

Two important observations:

• No product implements the full standard, and they all add extensions of their own. That is why we speak of "dialects": PostgreSQL, MySQL, Oracle and SQLite share the vast majority of everyday SQL, but they differ in types, functions and advanced syntax. In this course we will use standard SQL with PostgreSQL as the reference, pointing out the differences from SQLite where they exist.
• Look at SQL:2016 and SQL:2023: the relational standard absorbed JSON and graphs, that is, it took in what NoSQL databases had popularised. It is a pattern that repeats throughout this history.

The internet changes the load profile. A corporate database in the nineties served a few hundred employees during office hours; a website serves millions of anonymous users twenty-four hours a day, from anywhere in the world.

Two tensions appear at the same time.

The dominant languages (Java, C#, later Python, Ruby, PHP) are object-oriented: graphs of objects with inheritance and references. Databases are relational: flat tables. Translating between the two worlds by hand is tedious and repetitive. That friction is called the object-relational impedance mismatch.

The answer is the ORM (Hibernate in 2001, Django ORM, Active Record, Entity Framework, SQLAlchemy): layers that automatically translate objects into rows and back. They speed up development enormously, but they have a side effect that reaches us today: many people develop on top of relational databases without understanding what SQL is being executed, and only find out when performance collapses. It is one of the reasons a course like this one still makes sense.

Until then, the answer to "the database is slow" was to scale vertically: buy a bigger machine. It works until it stops working: there is a physical limit, the price grows non-linearly and there is still a single point of failure.

The alternative is to scale horizontally: spread the load across many cheap machines. But a classic RDBMS is designed around one node with a coherent view of all the data. Splitting a relational database across nodes (sharding) forces you to give up joins across fragments and global transactions, and to do it by hand, in the application.

Companies like Google, Amazon and Facebook hit that wall before anybody else, simply because they reach that scale first. And they publish what they do: BigTable (Google, 2006) and Dynamo (Amazon, 2007) are the two papers that light the fuse.

The term "NoSQL" becomes popular in 2009, at a meetup in San Francisco about non-relational distributed databases. It is soon reinterpreted as "Not Only SQL", which better describes what actually happened.

There were four real pressures, not a fashion:

1. Volume. Petabytes of data that do not fit on one server, however big it is.
2. Geographic distribution. Global services that need to replicate data between continents and keep working even if a whole data centre goes down.
3. Changing schemas. Web products that deploy several times a day and whose data constantly changes shape. A schema migration on a table with a billion rows can block the service for hours.
4. Loosely structured data. Documents, activity logs, user-generated content, which do not fit naturally into rows and columns.

The trade-off these systems accept is framed around the CAP theorem (Eric Brewer, 2000): in a distributed system that can suffer network partitions, you have to choose between consistency and availability. Many NoSQL systems choose availability and eventual consistency: after a write, the replicas converge on the same value, but not instantly. For a shopping basket that is acceptable; for an accounting entry it is not. The CAP theorem and its implications are studied in detail in lesson 03-04.

The product chronology is dense: BigTable (2006), Dynamo (2007), Cassandra (Facebook, 2008), Redis and MongoDB (2009), Neo4j popularising graphs, DynamoDB as a service (2012).

And then came the correction. Between 2012 and 2016 many organizations discover they had adopted NoSQL for problems they did not have: without the volume to justify it, they had given up integrity, transactions and SQL in exchange for nothing. The NoSQL ecosystem itself reacts: MongoDB adds multi-document transactions in 2018 and schema validation, that is, it recovers part of what it had discarded. In parallel, PostgreSQL incorporates jsonb (2014) and becomes a perfectly capable document store.

The net result was not the replacement of one model by another, but a widening of the catalog of options and the convergence of the families.

Around 2011 the natural question arises: is it really unavoidable to choose between horizontal scaling and transactional guarantees? The answer from a new generation of systems —christened NewSQL— is no, provided the engine is redesigned from scratch to run distributed.

• Google Spanner (2012) is the milestone: a globally distributed database, with ACID transactions and external consistency, resting on atomic clocks and GPS (the TrueTime API) to order transactions across continents.
• CockroachDB (2015) and YugabyteDB bring those ideas into open source, with compatibility with the PostgreSQL protocol.
• VoltDB, TiDB, Vitess (the system with which YouTube scaled MySQL) and Citus (a PostgreSQL extension) attack the same problem from different angles.

The common idea: SQL and ACID were not the obstacle to scaling; the monolithic architecture of traditional engines was. With distributed consensus (Raft, Paxos) and automatic partitioning, you can have both, at the price of higher latency on writes that cross regions.

Alongside the evolution of the models, who operates the database changes radically.

• Amazon RDS (2009) inaugurates the managed database: the provider takes care of installation, patching, backups, replicas and failover. The team just uses it.
• Azure SQL Database, Google Cloud SQL and managed offerings for almost every product (MongoDB Atlas, Redis Cloud) follow.
• Amazon Aurora (2014) redesigns the engine by separating compute from storage, while keeping compatibility with MySQL and PostgreSQL.
• The serverless stage (Aurora Serverless, Neon, PlanetScale, Supabase, Turso) takes the idea further: there is no server to size, capacity adjusts itself and you pay per use, even dropping to zero when there is no activity.

The effect for anyone learning today is twofold. On the one hand, the barrier to entry drops enormously: you can have a production-grade PostgreSQL in two minutes, administering nothing. On the other, the knowledge that matters shifts: administering the server loses weight, and designing the schema well and writing good queries take it all. Which is exactly what this course teaches.

The latest addition to the landscape arrives with language and vision models. These models turn a text, an image or an audio clip into a vector of hundreds or thousands of dimensions —an embedding— whose key property is that two pieces of content with similar meaning produce vectors that are close together in the space.

That creates a new storage need: storing millions of vectors and answering "give me the hundred most similar to this one" quickly, with approximate nearest-neighbour indexes (HNSW, IVF) that trade exactness for speed.

Between 2019 and 2023 Milvus, Pinecone, Weaviate and Qdrant appear, and the technique goes mainstream with the systems that retrieve relevant documents to hand to a language model as context.

And the usual pattern repeats itself: the established systems absorb the novelty. pgvector turns PostgreSQL into a competent vector database, and search engines and document stores add vector search of their own. History suggests that specialized vector databases will keep their niche at the very highest scale, while most projects will use the extension of the database they already have.

This is the content you really need to retain.

1. Every model was born to solve a specific problem. The hierarchical one, for tree structures like a rocket's bill of materials. The relational one, to end the dependency between programs and storage. NoSQL, for volume and distribution on a planetary scale. Vector databases, for semantic similarity search. Asking "what problem did this solve?" is the best way to know whether it is your problem.

2. No model completely replaced the previous one. IMS is still processing banking transactions sixty years on. COBOL and CODASYL are still alive. Relational did not eliminate hierarchical, and NoSQL did not eliminate relational: relational is still, by a wide margin, the most used family in the world. Technology accumulates far more than it replaces.

3. Declarative beats procedural. Codd's great insight was separating the what from the how. Every time the industry has gone back to asking programmers to manage the access path by hand —the early NoSQL systems, largely— it has ended up reintroducing declarative layers on top.

4. Guarantees you discard have to be reimplemented. If the database does not guarantee integrity, somebody will do it in the application code, worse and in more places. That MongoDB eventually added transactions and schema validation is the historical proof of that principle.

5. The families converge. PostgreSQL stores JSON and vectors; MongoDB has transactions; the SQL standard incorporates graphs. The border between "SQL" and "NoSQL" is far blurrier today than it was in 2010, and the useful question is no longer "SQL or NoSQL?" but "what guarantees do I need and what load am I going to carry?".

6. Real scale is almost never the imagined scale. A good share of the NoSQL adoptions of the 2010s solved Google's problems in organizations that were not Google. For BiblioRed —12,000 members— the right answer is the boring, proven one: PostgreSQL.

• Reading history as a staircase of progress. There is no succession of ever-better models; there is an accumulation of tools with different trade-offs. The "most modern" is not the most suitable by default.
• Believing NoSQL was born against SQL. It was born against scaling a single server, not against the language or the relational model. The proof is that many NoSQL systems today offer SQL-like interfaces.
• Assuming legacy systems are bad systems. An IMS that has gone forty years without losing a transaction has a reliability record few modern technologies can show. Migrating has costs and risks that need justifying.
• Confusing the SQL standard with what your manager does. No product implements it fully and they all add extensions. Before assuming a "standard" query is fine, check it in your dialect.
• Tip: when you evaluate a data technology, look for the original paper that introduced it (the Dynamo, BigTable, Spanner and Codd ones are available and readable). They usually state very clearly which problem they attack and which trade-offs they accept; the marketing material that follows, much less so.

Match each historical problem with the innovation that solved it and explain the link in one sentence:

Problems

1. Changing the physical format of a file forced you to recompile every program.
2. A book with two authors could not be represented without duplicating data.
3. A global company's data does not fit on a single server.
4. Translating by hand between Java objects and table rows was slow and repetitive.
5. There was no way to search for "documents that talk about something similar to this".
6. Scaling horizontally forced you to give up ACID transactions.

Innovations: ORM · relational model and its data independence · NewSQL and distributed consensus · vector databases · network model (CODASYL) · distributed NoSQL

Put these milestones in chronological order and state the decade of each one:

• Publication of Codd's paper
• The appearance of MongoDB and Redis
• SQL-92
• IMS for the Apollo programme
• Google Spanner
• The release of SQLite
• Oracle V2, the first commercial RDBMS with SQL
• Amazon RDS

A consultant proposes that BiblioRed migrate its whole system to a distributed NewSQL database "because that's what Google and modern banks use". Write a 6-10 line assessment that uses historical arguments from this lesson: which problem each technology solved, whether BiblioRed has that problem, and what you would recommend.

Solution 1

Solution 2

(MongoDB, Redis and RDS are from the same year; the order among them does not matter.)

Solution 3

Model answer:

NewSQL databases such as Spanner or CockroachDB were designed for a very specific problem: keeping ACID transactions when the data is spread across dozens or hundreds of nodes, often on several continents. Google needed it because its volume and its geographic distribution made a single server impossible; a global bank, for similar reasons. BiblioRed does not have that problem: 12,000 members, four branches in the same city and a few hundred thousand loans a year fit comfortably in a single PostgreSQL, with room for years of growth. Adopting a distributed system would mean taking on the operational complexity of consensus, replication and observability, plus higher write latency, without getting anything in return that we do not already have. The historical lesson is exactly that: a good share of the migrations of the 2010s solved Google's problems in organizations that were not Google. I recommend PostgreSQL, self-hosted or as a managed service, with backups and a read replica, and revisiting the question only if real multi-region or volume requirements ever appear, which today they do not.

We have covered sixty years following the thread of the problems, not of the dates:

• Flat files and batch processing produced redundancy, inconsistency and total dependency between programs and storage.
• The hierarchical (IMS) and network (CODASYL) models centralized the data, but forced the programmer to navigate the physical structure.
• Codd's 1970 paper broke with that: data as relations, links through values, declarative access and a mathematical foundation. Its practical fruit is data independence.
• System R and Ingres made it viable, contributing the cost-based optimizer, transactions and the SQL language, standardised from SQL-86 through to SQL:2023.
• The web and the ORM brought new scales and new frictions; the limit of vertical scaling led to NoSQL, pushed by volume, distribution, changing schemas and loosely structured data.
• NewSQL proved that ACID and horizontal scaling could coexist; the cloud changed who operates databases; vector databases added similarity search.
• And the underlying conclusion: every model solved a real problem, none completely replaced the previous one, and the families are converging.

We now know what databases are, what types exist and where each one comes from. What remains is to open the box: what exactly is inside the software that manages them. In lesson 01-04, Database Management Systems and Architecture, we will follow the complete journey of a query from the moment it is written until it returns rows, the three-level architecture that makes possible the data independence Codd proposed, the difference between client-server and embedded —PostgreSQL and SQLite, one at each extreme— and, very concretely, we will install the environment and create the biblioredb database where we will run all the SQL in this course.