Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
Linux server disk full: how to find and free space safely A step-by-step routine for a full disk on a Linux server: confirm what is full, find the space, clear the usual suspects safely and stop it happening again.
The alert says the API is returning 500s. You SSH in, try to open a file, and the editor complains it cannot write a swap file. Uploads fail, the database has stopped accepting writes, and cron jobs are silently doing nothing. The disk is full.
The short answer: run df -h and df -i to see which filesystem is full and whether it is space or inodes. Use du -xh to find the directories that grew. Clear logs, caches and unused container images first. If df still shows full after deleting files, a running process is holding a deleted file open; find it with lsof +L1.
Confirm what is actually full A server usually has several filesystems, and freeing space on the wrong one wastes time. Check all of them, by size and by inode count.
df -hT -x tmpfs -x devtmpfs -x overlay
df -i -x tmpfs -x devtmpfs -x overlayThe first command shows size, used and available space with the filesystem type. The second shows inodes. A filesystem can report free gigabytes and still refuse new files because every inode is used. That happens with millions of tiny files: PHP session files, mail queues, cache directories, or an application writing one file per request.
One detail confuses people on ext4. By default it reserves 5% of blocks for root, so df can show 100% use while used plus available is less than the size. Root-owned services can still write into that reserve, which is why some things keep working. XFS, the RHEL-family default, has no such reserve.
Find where the space went Work down from the full mount point one level at a time. The -x flag keeps du on one filesystem, so it does not wander into other mounts or network shares.
# Largest top-level directories on the root filesystem
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -15
# Then drill into whatever is biggest, for example /var
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -15
# Individual files over 500 MB
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null
# If inodes are the problem, count files instead of bytes
sudo du -x --inodes --max-depth=1 /var 2>/dev/null | sort -n | tail -15If ncdu is installed, sudo ncdu -x / does the same job interactively. You may not be able to install it on a full disk, so add it to your base image in advance.
The usual suspects, and how to clear them safely On the servers we look after, a full disk almost always comes from one of a short list of places. Each has a safe way to reclaim space that does not involve rm -rf.
The systemd journal in /var/log/journal. Check it with journalctl --disk-usage and trim it with sudo journalctl --vacuum-size=500M. Application and web server logs in /var/log or the application directory, often because logrotate is not configured for a custom log path. Docker images, build cache and container logs under /var/lib/docker. Inspect with docker system df before removing anything. Package caches and old kernels. On Debian and Ubuntu use sudo apt-get clean and sudo apt autoremove --purge. On RHEL-family systems use sudo dnf clean all. Core dumps in /var/lib/systemd/coredump or /var/crash after a service has been crashing in a loop. Local backups and database dumps that were meant to be copied elsewhere and then deleted, but never were. For Docker, these commands are the safe ones. They remove stopped containers, images not used by any container, and build cache. They do not touch volumes, which is where your data lives.
docker system df
docker container prune
docker image prune -a
docker builder prune
# Find the biggest container log files
sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log' | sort -h | tail -5Be careful with docker volume prune and docker system prune --volumes. A volume counts as unused once no container references it. Stopped containers still count, but the moment a database container is removed, by docker compose down or by the container prune above, its volume becomes a target. Since Docker Engine 23, volume prune only removes anonymous volumes unless you add --all, and many compose files still use anonymous ones. List candidates with docker volume ls -f dangling=true and read the list before pruning anything.
For a log file that a process is still writing to, truncate it rather than deleting it. sudo truncate -s 0 /var/log/app/app.log frees the space immediately and the process carries on writing to the same file. If you need the contents, compress a copy to another filesystem first.
When df still says full after you deleted files This is the most common reason a disk incident drags on. Someone deletes a 20 GB log, but df does not move. On Linux, deleting a file only removes its name. The blocks are freed when the last process holding it open closes it, so a service still writing to that deleted log keeps every byte allocated.
# List open files whose link count is zero (deleted but still held)
sudo lsof -nP +L1
# Free the space without restarting, using the PID and FD from lsof
# (FD column shows e.g. 4w; use the number only)
sudo truncate -s 0 /proc/1234/fd/4
# Or restart or reload the service so it reopens its log files
sudo systemctl restart nginxRestarting is the cleaner fix where you can afford it. Truncating through /proc helps when you cannot restart right now, such as a database or a long-running job.
If du and df still disagree and lsof shows nothing, check for files hidden under a mount point. Data written to /mnt/data before a volume was mounted there is invisible while the volume is mounted, but still uses space on the root filesystem. Bind-mount the root filesystem elsewhere to look underneath:
sudo mkdir -p /mnt/rootfs
sudo mount --bind / /mnt/rootfs
sudo du -xh --max-depth=1 /mnt/rootfs/mnt/data
sudo umount /mnt/rootfsWhat not to delete, even in a hurry Under pressure, the biggest directory looks like the obvious target. For databases it is usually the worst one to touch by hand.
PostgreSQL pg_wal. Deleting WAL files by hand can leave the cluster unable to start. If pg_wal is growing, the cause is usually an inactive replication slot or a failing archive_command. Fix the cause and PostgreSQL will remove old segments itself. MySQL or MariaDB binary logs. Remove them with PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY, not with rm, after checking no replica still needs them. Set binlog_expire_logs_seconds (MySQL 8+, MariaDB 10.6+) so they expire automatically. MongoDB data files. Space freed by deleting documents is reused by MongoDB but not returned to the operating system. Reclaiming it is a planned maintenance task, not an emergency rm. Anything inside /var/lib/docker by hand, other than truncating container log files. Use the docker commands so the daemon's metadata stays consistent. Files under /boot, unless you are removing old kernels through the package manager. To check for a stuck PostgreSQL replication slot, this query shows how much WAL each slot is holding back:
SELECT slot_name, active,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots;An inactive slot holding many gigabytes usually belongs to a replica or a change-data-capture tool that no longer exists. Confirm that with whoever owns it, then drop it with SELECT pg_drop_replication_slot('slot_name'). Since PostgreSQL 13, max_slot_wal_keep_size can cap how much WAL a slot may retain.
Stop it happening again Clearing space fixes today. These settings fix next month, and we apply them by default on servers we set up.
# /etc/docker/daemon.json (applies to containers created after a daemon restart)
{
"log-driver": "local",
"log-opts": { "max-size": "20m", "max-file": "5" }
}
# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=1G
# then
sudo systemctl restart systemd-journaldAdd a logrotate rule for every custom log path, with copytruncate or a postrotate reload so the process stops writing to the old file. Alert on disk usage at around 80% and again at 90%, and alert on inode usage as well. A full disk at 3am is far worse than a warning the afternoon before. Keep databases, Docker data and uploads on their own volumes where you can, so one runaway log cannot stop the database. Ship backups off the server and delete local copies on a schedule. Know how to grow the disk. On most cloud VMs you resize the volume in the console, then run sudo growpart /dev/nvme0n1 1 followed by sudo resize2fs /dev/nvme0n1p1 for ext4 or sudo xfs_growfs / for XFS. Check device names with lsblk first. Growing a volume is a legitimate fix, but make it a decision rather than a reflex. If the growth is logs nobody reads, a bigger disk only delays the next incident.
A short checklist for the next full disk Run df -hT and df -i to find the full filesystem and whether it is space or inodes. Use du -x level by level, or ncdu, to find what grew. Clear the journal, package caches, unused Docker images and old logs first. Truncate active log files rather than deleting them. If df does not drop, check lsof +L1 and restart or truncate the holding process. Never rm database WAL, binlogs or data files. Fix the cause instead. Afterwards, set log limits and disk alerts, and write down what filled the disk. If the disk keeps filling and you cannot see why, or the full filesystem holds your database, stop deleting and get a second pair of eyes. A few minutes of investigation is cheaper than restoring a database you removed by accident.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789