Tech Behind ThingsHow the ordinary machinery actually works

Devices

Your phone camera is mostly software, and that is not cheating

The sensor is smaller than a fingernail. Almost everything that makes the picture look good happens in the fraction of a second after the shutter.

Detailed close-up of a modern smartphone camera against a dark background.
Photograph by Jatin Jangid via Pexels
Editorial note. Independent reporting and analysis. Nothing here is sponsored or paid for. How we work.

Everything here earned its place by changing an outcome. Nothing about computational photography is included to round the number up.

What matters most

  • A small sensor collects little light, and every technique is a response to that.
  • Most phone photographs are several exposures merged rather than one capture.
  • Depth effects are estimated, which is why they fail at hair and glass.

The sensor sets the hard floor

A camera can only work with the photons that land on it, and a phone sensor has perhaps a hundredth of the collecting area of a full-frame one. Fewer photons per pixel means a weaker signal relative to the random variation in arrival, which is what noise actually is.

No amount of processing creates information that never reached the sensor, so every phone camera technique is a way of gathering more light or making better use of what arrived. That is why larger sensors still win in dim rooms despite decades of processing improvement.

One press, several exposures

Phones capture a continuous stream of frames before you press the button and keep a rolling buffer, so the shutter selects frames rather than starting the capture. A handful of those frames are aligned and averaged, which reduces random noise roughly in proportion to the square root of the number combined.

Alignment has to compensate for hand movement and for subjects that moved between frames, which is where merging artefacts come from. This is also why the shutter feels instant: the picture partly existed before you asked for it.

High dynamic range is exposure arithmetic

A sensor cannot record a bright sky and a shaded face in one exposure because the range between them exceeds what the pixel wells can hold. Capturing several exposures at different lengths and combining the usable parts of each recovers detail at both ends.

In the datasheet, the characteristic flat, shadowless look comes from tone mapping that compresses the recovered range into what a screen can display. Vendors tune that compression very differently, which is most of what people perceive as a brand having a look.

Noise reduction is informed guessing

Removing noise means deciding which variation is signal and which is randomness, and that decision is a model rather than a measurement. Aggressive reduction produces the waxy skin and smeared foliage that appear when phone photographs are viewed at full size. Sharpening applied afterwards restores apparent detail by exaggerating edges, which is not the same as recovering it.

In the datasheet, zooming in on a low-light phone photograph reveals both processes arguing with each other.

Depth effects are estimated, not measured

Background blur on a phone is computed by estimating how far away each part of the scene is, then blurring in proportion. The estimate comes from parallax between lenses, from dedicated depth sensors on some models, or from a model trained to recognise subjects.

It fails predictably at fine structures such as hair, at transparent objects, and at anything the model has not seen much of. An optical blur from a large aperture has no such failure mode because it is a property of the light path rather than a segmentation decision.

What to look at when comparing

Sensor size and lens aperture set the ceiling; processing decides how close a camera gets to it in ordinary conditions. Shooting in a raw format bypasses most of the pipeline and gives you the noisy, flat, honest capture to process yourself.

Optical stabilisation genuinely helps because it allows longer exposures, which is more light rather than better guessing. Digital zoom beyond the range of the physical lenses is upscaling, and the marketing number for it is not a measurement of anything.

Everything above, in order of what to do first

  1. The sensor sets the hard floor. A camera can only work with the photons that land on it, and a phone sensor has perhaps a hundredth of the collecting area of a full-frame one.
  2. One press, several exposures. Phones capture a continuous stream of frames before you press the button and keep a rolling buffer, so the shutter selects frames rather than starting the capture.
  3. High dynamic range is exposure arithmetic. A sensor cannot record a bright sky and a shaded face in one exposure because the range between them exceeds what the pixel wells can hold.
  4. Noise reduction is informed guessing. Removing noise means deciding which variation is signal and which is randomness, and that decision is a model rather than a measurement.
  5. Depth effects are estimated, not measured. Background blur on a phone is computed by estimating how far away each part of the scene is, then blurring in proportion.
  6. What to look at when comparing. Sensor size and lens aperture set the ceiling; processing decides how close a camera gets to it in ordinary conditions.

The takeaway

The lens gathers light; the software decides what the picture means. Both limits are real.

Understanding the failure mode tells you more than the feature list does.

Questions readers ask

Why do my photographs look worse when I view them on a computer?

Phone screens are small and bright, which hides noise reduction artefacts and softness. The same file at full size on a large display shows the processing decisions clearly.

Is shooting raw worth it on a phone?

Only if you intend to process the file yourself. Raw discards the multi-frame merging and tone mapping that make phone pictures look good, so an unprocessed raw usually looks worse than the ordinary output.

Devicescamerasphotographysensorsimage processing
Farida Osei
Networks writer, Tech Behind Things

Farida writes about wireless standards and spent six years in network engineering before switching to explaining it.

Also by Farida Osei