Tech Behind ThingsHow the ordinary machinery actually works

Software

A file is a promise the filesystem makes, not an object on the disk

The name, the contents and the location are three separate things held together by bookkeeping, and most storage mysteries live in that gap.

Vibrant and engaging code displayed on a computer screen, showcasing programming concepts.
Photograph by Seraphfim Gallery via Pexels
Editorial note. Independent reporting and analysis. Nothing here is sponsored or paid for. How we work.

This is written to be used rather than admired. Each section below is a decision about filesystems, and each one has a default.

Before you start

  • A directory is a list of names pointing at records held elsewhere.
  • A journal records intent so an interrupted write can be resolved.
  • Copy-on-write never overwrites live data, which makes snapshots cheap.

Names, records and blocks are separate things

A filesystem keeps a record for each file holding its size, timestamps, permissions and a list of where its contents live. The name is not in that record; it sits in a directory, which is itself a file containing names paired with record numbers.

This separation is why renaming a large file is instant, because only an entry in a list has changed. It also allows one set of contents to have several names in different places, with the record counting how many refer to it. The contents themselves occupy blocks scattered across the device, and the record's list is what makes them a file.

Free space is a list too

The filesystem maintains a structure describing which blocks are in use, and allocation means finding suitable free ones. When a file grows, new blocks are taken from wherever is convenient, which need not be near the existing ones. On spinning media that scattering costs seek time, which is why defragmentation was once a routine maintenance task.

At the protocol level, on flash storage there is no seek penalty, so fragmentation matters far less and rewriting everything merely wears the device. What does still matter is free space, because an almost full device gives the allocator no good choices to make.

A journal turns half-finished writes into decisions

Updating a file usually requires several separate changes, and losing power between them leaves the structure inconsistent. A journalling filesystem writes a description of what it intends to do before doing it, in a dedicated area on the device.

Mechanically, after an unexpected restart it reads that journal and either completes or discards each interrupted operation cleanly. Most journals record only the structural changes rather than the file contents, which is fast and leaves data itself unprotected. That is why a crash can leave a file that exists, has the right size and contains nothing useful whatsoever.

Copy-on-write changes the rules

Some filesystems never overwrite live data, writing modified blocks to new locations and updating the pointers afterwards. Because the old blocks remain untouched until nothing refers to them, the previous state is still complete and consistent.

Retaining that state costs almost nothing, which makes snapshots a natural feature rather than an expensive add-on. The same structure allows checksums on every block, so silent corruption is detected rather than quietly returned to programs.

The cost is that files become fragmented as they are modified, and free space accounting becomes considerably more complicated.

Why removing safely actually matters

Writes are buffered in memory and reported as complete long before they reach the physical medium, because that makes everything faster. Unplugging a device with buffered writes outstanding loses them, and the loss may include structural updates rather than only content. The safe removal step flushes buffers and marks the filesystem cleanly unmounted, which takes a moment on a busy device.

Removable media are often configured to write through immediately for exactly this reason, at a noticeable cost in speed. A filesystem left dirty is repaired on the next mount, and repair means making it consistent rather than making it correct.

Firmware updates change this behaviour more often than hardware does.

Different filesystems make different promises

Some prioritise crash safety, some raw throughput, some enormous volumes, and none of them optimise for everything at once. Case sensitivity, permitted characters and maximum name lengths differ, which is why files sometimes refuse to copy between systems. Timestamp resolution and the set of timestamps kept vary too, so metadata is quietly lost when moving between filesystems.

Under load, extended attributes storing tags or security labels frequently do not survive a copy to a filesystem that lacks them. The visible file is the same, which is what makes these losses so easy to miss until something depends on them.

The takeaway

The name, the record and the blocks are three things, and every storage puzzle sits between them.

The constraint is almost always physical, and marketing rarely mentions which one.

Questions readers ask

Why does a copied folder show a different size?

Block allocation, compression and metadata differ between filesystems. The contents are identical while the space consumed is not.

Is defragmenting a solid state drive useful?

No. There is no seek penalty to remove, and rewriting every block consumes write endurance for no measurable benefit.

Softwarestoragesoftwareoperating systemsdata
Junko Ishida
Contributing writer, Tech Behind Things

Junko covers batteries, charging and energy density, and is unimpressed by most battery claims.

Also by Junko Ishida