dedup
also: dedup, data deduplication, content-aware storage
Deduplication is a data storage technique that eliminates redundant copies of data by storing only one instance and referencing it multiple times, reducing storage space and improving efficiency.
Deduplication (dedup) works by identifying identical blocks or chunks of data across a filesystem or backup system and replacing duplicates with pointers to a single stored copy. This is especially valuable in backup systems, virtual machine storage, and large datasets where the same data often appears multiple times.
There are two main types: block-level deduplication divides files into fixed or variable-sized blocks and compares them, while file-level deduplication compares entire files. For example, if you backup the same Linux system daily, dedup recognizes that 95% of files haven't changed and only stores the new data, reducing backup size from terabytes to gigabytes.
Deduplication can occur inline (during write operations, slower but saves immediate space) or post-process (after data is written, faster but uses more temporary space). Tools like btrfs, ZFS, and backup software such as restic and commercial solutions support deduplication natively.