How to store data

A single save is rarely a single action underneath. The simplest approach, truncate then write, means there is a moment where the old data is gone and the new data is not there yet. A crash in that gap loses everything. The fix is to write the change atomically, or to use a store that keeps a replayable log.

One writer is easy. Two writers racing to change the same thing is where flat files quietly corrupt. Real databases serialize this for you with a lock or a transaction; files and git do not.

Fetching one item by its name (key-value) is something almost anything can do. “Everyone who signed up last week and has not logged in since” is a query, and the moment you need those, a file forces you to load everything and loop by hand. That is the line where a database stops being optional.

Your agent will happily “add backups” by writing a second copy next to the first, on the same disk, corrupted by the same crash. A backup only counts once you have restored from it and watched the system come back. Until then it is a hope.

The honest answer is usually “one machine, and I am over-worrying.” But the moment data lives on more than one machine you inherit a genuine law: you cannot have perfect consistency, constant availability, and survival of a network split all at once. You pick which bends.

If nobody chooses, the system chooses for you, usually at the worst possible moment.

What it is. You read the whole thing into memory, change it, write it back. No server, no setup, you can open it in a text editor. For a config file or a one-off script it is the right call.

Concurrency and durability the moment the app is real. Two writers clobber each other; a crash mid-write can leave a half-file; there is no query beyond “load it all and loop.”

What it is. Still files, but with a complete, auditable history and the ability to branch and merge. Wonderful for content, configuration, and anything a human edits deliberately.

It is not built for programs writing constantly or at once. Two processes committing at the same time is a conflict, not a transaction.

What it is. A full SQL database with transactions and real queries, that lives in one file and needs no server. Durability and safe concurrent reads with almost none of the operational cost of a server.

Heavy concurrent writes from many processes at once, and running across multiple machines. For most apps that never happens, which is why “just use SQLite” is good advice more often than people admit.

What it is. A separate program your app talks to over a connection. Built for many clients writing at once (high concurrency), strong guarantees, and serious querying. The workhorse once an app has real concurrent users.

Simplicity and operational cost. Now there is a server to run, secure, back up, and connect to. Taking that on before you need it is the most common over-engineering your agent will cheerfully do.

What it is. A company runs the database, backups, and scaling for you, behind an API or a connection string. You trade money and control for not being the one paged at 3am.

Control and portability. Your data and often your data model now live in someone else’s shape; moving off later is real work. Every call also crosses the network, so “fast” has a floor you do not set.

A file is bytes; everything else is your code’s job. A database bundles the hard parts: a change either fully happens or not at all (a transaction), two writers do not corrupt each other (concurrency), and you can query across the data without loading it all. You are not paying for storage, you are paying for those guarantees.

Atomicity (all-or-nothing), Consistency (rules never break), Isolation (concurrent writes do not see each other half-done), Durability (once it says saved, it survives a crash). When people say “a real database,” this is mostly what they mean.

The law behind “Scale” above. The network will split sometimes; when it does, you either refuse writes to stay consistent, or accept them and reconcile later to stay available. Pick on purpose, because the default is chosen for you under failure.

Before changing the data, the database records “I am about to do X” in an append-only log. If it crashes, on restart it replays the log and finishes or discards cleanly. It is why a database can promise durability and a naive file-overwrite cannot.

One person, one device, a few hundred items at most. Walk the five questions and listen to how fast they collapse.

Durability matters a little: you would hate to lose today’s list to a crash, but a careful atomic save covers that. Concurrency is a non-issue, because there is exactly one writer, you, and you are not tapping the same checkbox from two phones at once. Query is trivial: you load “my lists” and show them, you are never asking “all todos across all users where due date is Tuesday.” Recovery is mostly handled for you, since the phone’s own backup already copies the file to the cloud. Scale is one device, probably forever.

Add it up and the honest answer is a single file on the device. The only thing that nudges you up a rung is wanting instant search across thousands of items, or the same list on your laptop too, and then it is on-device SQLite, still no server.

Now almost every answer flips. Concurrency is real: a background job writes fresh numbers every few minutes while several teammates load the page at once, so you have concurrent readers and a writer touching the same data. Query is the entire point: “revenue by week,” “top ten customers this month,” all of which is querying across records, the one thing files are worst at. Durability and recovery stop being shrugs, because this is shared business data and losing it, or failing to restore it, is an actual incident. Scale is likely just one machine, but it has to be dependably up during the workday.

Add it up and you want a real database server. A flat file would corrupt under the concurrent writes and choke on the queries by the second week.

This one sits in between, and it is the most instructive. One primary user and modest data, which sounds like a file. But you will absolutely want to query it, “everyone I have not talked to in ninety days” is the whole reason the thing exists, and that pushes you past a flat file to SQLite. Then the second pressure shows up: you want it on your phone and your laptop, and the instant two devices need to see the same data, a local file on one of them stops working.

Notice the deciding question here is not durability or scale. It is “do I need this in two places,” and only you know the answer. That single yes-or-no moves you a whole rung.