4

I have just done a fresh install of Raspian Buster Lite on a Raspberry Pi 3B, and installed Prometheus along with Node Exporter to just scrape a few simple diagnostics - particularly the available disk space.

The problem is that the available disk space appears to be continuously decreasing. I can check the bytes availble by using df -B1, which returns the following (13.08 GB from the "16 GB" SD card used):

pi@raspberrypi:/home $ df -B1 Filesystem 1B-blocks Used Available Use% Mounted on /dev/root 15383740416 1647845376 13078351872 12% / devtmpfs 480808960 0 480808960 0% /dev tmpfs 485527552 0 485527552 0% /dev/shm tmpfs 485527552 6459392 479068160 2% /run tmpfs 5242880 4096 5238784 1% /run/lock tmpfs 485527552 0 485527552 0% /sys/fs/cgroup /dev/mmcblk0p1 264289280 54748672 209540608 21% /boot tmpfs 97103872 0 97103872 0% /run/user/1000

If I examine this periodically, it just reduces. Checking it a few minutes later shows the following:

Filesystem 1B-blocks Used Available Use% Mounted on /dev/root 15383740416 1648463872 13077733376 12% / devtmpfs 480808960 0 480808960 0% /dev tmpfs 485527552 0 485527552 0% /dev/shm tmpfs 485527552 6459392 479068160 2% /run tmpfs 5242880 4096 5238784 1% /run/lock tmpfs 485527552 0 485527552 0% /sys/fs/cgroup /dev/mmcblk0p1 264289280 54748672 209540608 21% /boot tmpfs 97103872 0 97103872 0% /run/user/1000

I can examine this metric in the Prometheus database using Grafana:

enter image description here

It may well just be some logs or something, but I want to understand it a little better. Based on this graph, it seems the space is reducing at a rate of around 2.9 MB per hour, which seems like a lot to me.

Ideally I suppose I am looking for a way to view only files that have changed in size, let's say in the last minute or something. What would be a good way to track down the source of this? Thanks for your patience - I am brand new with linux.

Edit

In case anyone is interested, I discovered that almost all of the 2.9 MB/hour can be attributed to the write ahead logs (WAL) in Prometheus - you can read about how it stores data here. I was able to observe the WAL directory building up by using the command

watch -d "sudo du -s -B1 /home/pi/prometheus/data/*"

which highlights nicely the changing files:

enter image description here

As Richie Frame hinted at, every so often (I believe it is supposed to be every 2 hours in Prometheus by default, but I need to check that) the data is compacted from the WAL and is put into more permenant storage, which is more efficient in terms of required disk space. You can see on Grafana the compaction happening (this was taken overnight whilst the Pi was otherwise just sitting there idle), and some of the disk space being released:

enter image description here

I am not sure why the disk space seems to jump by a large amount only every 4 hours, instead of the expected 2 hours, but that is a task for another day. Thanks for everyone for your help!

Improve this question
4
  • 3
    To check what logs are written, you could do something like watch -d "ls -lt /var/log/**/* | head". Changes will be highlighted. Commented May 22, 2020 at 12:39
  • 2
    3MB per hour actually seems very reasonable, at some point prometheus will archive and compact the data, and it will be much smaller Commented May 22, 2020 at 21:28
  • @RichieFrame I can now see the compaction happening - I have added an edit to the post in case you are curious. Thanks! Commented May 23, 2020 at 16:08
  • 1
    compaction time varies based on several factors, if the files are small it will wait longer (i think this is what it said when i read the docs a while ago) Commented May 24, 2020 at 2:53

3 Answers 3

Reset to default
4

My suggestion would be this command:

watch -d "ls -lt /var/log/**/* | head"

watch runs the following command by default every 2 seconds. The -d flag highlights differences after each execution.

ls -lt lists files according to their last modified date (newest ones first), **/* is a glob to find all files recursively.

Lastly, head is used to output only the first 10 lines.

Improve this answer
0
4

You can find the recently updated files on your system using something like 

find / -type f -newermt '-5 minutes'`

You can combine that with a size display: 

find / -type f -newermt '-5 minutes' -exec stat -c '%10s %n' {} + `
Improve this answer
2
  • 1
    better /proc to be excluded Commented May 22, 2020 at 13:10
  • +1 thanks for your answer. I accepted Panki's answer, only simply because it was the first one I tried and it did the trick! Thank you for your contrbution. Commented May 22, 2020 at 14:37
1

as root do crontab -e and do something like

30 * * * * * /root/checkmydiskspace.sh

that will run the bash script /root/checkmydiskspace.sh, every half hour, which you have to create. Look up crontab and decide if you need to check more often.

That script would be this, and would output necessarily info to help you pinpoint where the disk space is getting used up over time.

#!/bin/bash now=$(date '+%Y-%m-%d-%H:%M') filename=wheremydiskspacegone_$now.log du -sh /* > /root/$filename

Also, chmod 744 /root/checkmydiskspace.sh

look at the created log files, find out where stuff is happening, then alter du -sh /* to be du -sh /usr/* for example; work your way into the sub folders where the du is showing most space used.

Of course you could just do du -sh /* up front and see which folder has the most stuff and that is likely the culprit. But with all the log files, every X minutes or whatever, you have everything needed to deduce what is specifically happening and where. Then it would be a matter of determining which program or process is causing that.

Improve this answer
1
  • 5
    This will create more log files and use up space - so a bit counter-productive here imo Commented May 22, 2020 at 14:04

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.