Version 1.0.0 — Windows-native duplicate file manager for local drives and network shares.
Duplicate Manager finds and safely cleans up duplicate files across local drives, external drives, and network shares (including UNC paths like \servershare). It’s built with safety, transparency, and performance in mind: nothing is permanently deleted by default, and every action is written to an audit log so you always have a record of what happened.
Duplicate Manager offers three interchangeable front ends that all share the same scan engine, profiles, and audit log: a desktop GUI for everyday use, a command-line tool for scripting and automation, and an optional local REST API for integrating scans into your own tools.
Getting Started
- Launch Duplicate-Manager.exe.
- In the left sidebar, under Folders / Drives / UNC Paths, click Add Folder… (or Add Path… for a drive letter or
\serversharepath) and pick a location to scan. - Optionally expand Filters to limit the scan by file extension, size range, or hidden/system files, and expand Settings to choose a hash algorithm or adjust the quarantine folder.
- Click Start Scan. The results panel shows live progress — files enumerated, files hashed, elapsed time, and an ETA — while the scan runs. You can pause, resume, or cancel it at any time.
- When the scan finishes, the results panel lists duplicate clusters — groups of files with identical content. Within each cluster, one file is treated as the copy to keep; the rest are the duplicates.
- Use Select All / Select None, or check individual files, to choose which duplicates to act on. Click a file to preview it in the right-hand preview panel.
- Pick an action from the bottom bar: Delete (Recycle Bin), Quarantine, Permanent Delete, or Replace with Symlink, then confirm in the dialog that appears. Permanent Delete is blocked entirely while Safe Mode is on.
- If you quarantined or symlink-replaced files and change your mind, click Undo Last Batch right away to reverse the most recent batch.
- Use Export JSON / CSV / HTML in the bottom bar at any time to save a report of the scan results.
The Three Interfaces
| Interface | Command | Typical user |
|---|---|---|
| Desktop GUI | dupmgr-gui | Everyday users reviewing results visually |
| Command line | dupmgr | Power users, scripts, scheduled tasks |
| REST API | dupmgr-api (port 8756) | Automation, integration with other tools |
All three drive the same scanning engine and write to the same audit log, quarantine folder, and profile store — a scan profile saved from the GUI can be loaded from the CLI, and files quarantined from the CLI can be restored from the GUI.
How Duplicate Detection Works
- Hashing algorithms. xxHash64 (default — very fast, not cryptographic) or SHA-256 (slower, cryptographic-grade — useful for compliance or audit requirements).
- Size threshold for large files. Files below a configurable size threshold (32 MB by default) are hashed in full directly. Files at or above the threshold are first block-hashed (just the first and last 4 MB); only matches on size and block hash go on to a full-content hash.
- Byte-by-byte final check. Whenever two or more files share size and full hash, a last byte-by-byte comparison runs before they’re treated as true duplicates.
- Perceptual hashing for images (optional). Computes a perceptual hash (pHash) for image files to surface near-duplicates — e.g. a resized or recompressed copy — that aren’t byte-identical. Off by default.
- Incremental scan cache. Each file’s hash is cached by path, size, and modified time, so repeat scans of largely unchanged folders are much faster. Can be disabled for a guaranteed from-scratch scan.
- Keep vs. duplicate. Within a cluster, the first file (after sorting) is treated as the one to keep. Sort mode — largest, most duplicates, oldest, or newest — decides which file that is.
Safety Features
- Safe Mode is on by default and blocks the two truly irreversible operations — Permanent Delete and permanently purging the quarantine folder. It must be explicitly turned off (GUI Settings, or
--forceon the CLI) before either can run. - Delete sends a file to the Windows Recycle Bin.
- Quarantine moves a file into a managed holding folder, recording where it came from so it can be restored later from either the GUI or CLI.
- Permanent Delete removes a file with no recovery path from within the app, and only runs when Safe Mode is off.
- Replace with Symlink removes a duplicate and puts a symbolic link in its place, pointing at the kept file. Symlinks on Windows require Developer Mode or running elevated — if creation fails, the app automatically copies the original file back instead, and logs the detail.
- The audit log records every action — success, failure, or skip — with a timestamp.
- No network transmission. All hashing happens locally.
Scan Profiles
A scan profile is a named, saved set of roots, filters, and hashing settings, stored as one JSON file per profile — so you can re-run the same scan (e.g. “Downloads folder, images only, SHA-256”) without re-entering every option. In the GUI, the Profiles group in the sidebar loads a saved profile or saves the current configuration under a new name. From the CLI, use dupmgr profile save / list / show / delete, or dupmgr scan --profile <name> to load one as a base (other flags on the command line override it).
Reports and Exports
- JSON — the full machine-readable report: a summary plus every cluster and file, and any actions taken.
- CSV — one row per file, so it opens flat in Excel.
- HTML — a styled, self-contained report page with summary tiles and a per-cluster breakdown.
In the GUI, use the Export JSON / CSV / HTML buttons after a scan. From the CLI, pass --json, --csv, and/or --html (with a file path) to dupmgr scan.
Command-Line Reference (dupmgr)
dupmgr drives the same scan engine as the GUI, with no server required. Every subcommand supports -h/--help for its full option list.
dupmgr scan
Scans one or more roots and reports duplicate clusters. Key flags: --root PATH (repeatable), --profile NAME, --hash-algo {xxh64,sha256}, --min-size / --max-size, --include-ext / --exclude-ext, --exclude-path, --include-hidden, --include-system, --block-threshold-mb, --block-size-mb, --perceptual-hash, --no-cache, --workers N, --sort {largest,most_duplicates,oldest,newest}, --limit N, --json/--csv/--html PATH, --save-as NAME, --quiet.
dupmgr scan --root D:Photos --root \nasbackup --include-ext jpg,png --hash-algo sha256 --sort most_duplicates --json report.jsondupmgr act
Applies one action to one or more specific files: dupmgr act {delete|permanent-delete|quarantine|symlink-replace} FILE [FILE...] [options]. Flags: --force (disable Safe Mode, required for permanent-delete), --quarantine-dir PATH, --kept-path PATH (required for symlink-replace), --cluster-hash HASH.
dupmgr act quarantine "D:PhotosIMG_0002_copy.jpg"
dupmgr act permanent-delete "D:PhotosIMG_0002_copy.jpg" --forcedupmgr quarantine
quarantine list [--quarantine-dir PATH] [--json PATH]— list quarantined files.quarantine restore ID [ID...] [--quarantine-dir PATH]— restore quarantined file(s) to their original location.quarantine purge [--quarantine-dir PATH] [--older-than-days N] [--force]— permanently delete quarantined files. Without--forcethis is a dry run that only lists what would be deleted. Irreversible.
dupmgr audit & dupmgr profile
dupmgr audit [--limit N] [--json PATH] shows or exports the action audit log, most recent first (default limit 200). dupmgr profile save/list/show/delete manages saved scan profiles.
REST API Reference (dupmgr-api)
Start the API with dupmgr-api, or directly via uvicorn duplicate_manager.api.app:app --port 8756. It exposes the same engine as the GUI and CLI over HTTP, for local automation and integration.
| Method & Path | Description |
|---|---|
GET /health | Health check |
POST /scan/start | Start a scan from a ScanProfile body. Returns a scan_id. |
GET /scan/status/{scan_id} | Current progress (status, files enumerated/hashed, ETA, errors). |
POST /scan/pause/{scan_id} | Request the scan pause. |
POST /scan/resume/{scan_id} | Resume a paused scan. |
POST /scan/cancel/{scan_id} | Cancel a scan. |
GET /duplicates/list/{scan_id} | Duplicate clusters for a completed/cancelled scan (sort param). |
POST /action/delete | Execute a batch action (delete, permanent_delete, quarantine, symlink_replace). |
POST /action/undo | Reverse a previous quarantine or symlink-replace action. |
GET /audit/log | Fetch recent audit log entries (max 10,000, default limit 200). |
Where Your Data Lives
Everything Duplicate Manager keeps on disk lives under %AppData%DuplicateManager, regardless of which interface you use.
| Folder | Contents |
|---|---|
logs | Application logs and audit_log.jsonl (the full action audit trail). |
quarantine | Default quarantine folder and its manifest. |
cache | hash_cache.sqlite3, the incremental scan cache. |
profiles | Saved scan profiles (one JSON file per profile). |
reports | Default location suggested for exported reports. |
Troubleshooting
Replace with Symlink fails on a file
This is almost always because Windows Developer Mode isn’t enabled and the app isn’t running elevated. Duplicate Manager automatically copies the original file back so nothing is lost, and logs the detail. To get a real symlink, enable Developer Mode (Settings → Update & Security → For developers) or run the app as Administrator, then retry.
Permanent Delete (or quarantine purge) is blocked
This is Safe Mode doing its job. Turn Safe Mode off in GUI Settings, or add --force to the CLI’s act permanent-delete or quarantine purge command. Both are irreversible once run.
I quarantined the wrong file(s)
In the GUI, click Undo Last Batch immediately after the action. From the CLI (or for anything older than the last batch), run dupmgr quarantine list to find the id, then dupmgr quarantine restore <id>. This only works if the file hasn’t since been purged.
A scan of a large drive or network share is taking a long time
Expected for very large datasets. The scan shows live progress with an ETA, and can be paused and resumed rather than needing to run start-to-finish in one sitting.
Some files show up as scan errors
Files the scanner couldn’t read (permission denied, file in use, etc.) are recorded in the scan’s error list — and in the JSON report — rather than stopping the whole scan.
Frequently Asked Questions
Where is my data stored?
All application data — logs, quarantine, the incremental hash cache, and saved scan profiles — lives under %AppData%DuplicateManager on your own machine.
Does Duplicate Manager send my files anywhere?
No. All hashing and comparison happens locally; file contents are never transmitted over a network by the app itself.
Which hash algorithm should I use?
xxHash64 (default) is fast and more than sufficient for everyday duplicate detection. Choose SHA-256 if you specifically need a cryptographic-grade guarantee, e.g. for compliance — it’s slower.
What’s the difference between Delete and Quarantine?
Delete sends a file to the Windows Recycle Bin. Quarantine moves it to a Duplicate Manager-managed folder the app itself can restore from later, from either the GUI or CLI.
Can I automate scans instead of using the GUI?
Yes — use the dupmgr command line for scripts and scheduled tasks, or run dupmgr-api and drive it over HTTP for tighter integration with other tools.