Monday, March 11, 2013

adjust retention time for carbon and resize whisper files

some graphite storage-schema.conf examples out there only retain data for 1 day.


[default_1min_for_1day]
pattern = .*
retentions = 60s:1d


this comes as a surprise later when you go back to look at your data and its not there...its been rotated out by carbon due to the geometry of the whisper file. the whisper file has a finite size...ya dig?

Update the retention rule in /opt/graphite/conf/storage-schemas.conf


[default_1min_for_1year]
pattern = .*
retentions = 60s:525600


#adjust existing whisper files

find /opt/graphite/storage/whisper -type f -name "*.wsp" | xargs -I{} whisper-resize.py {} 60:5256000

#restart carbon

/usr/bin/python /opt/graphite/bin/carbon-cache.py --config=/opt/graphite/conf/carbon.conf start


root@nagios4.sv3:~$ ls -la /opt/graphite/storage/whisper/prod/web1/apache/
total 43272
drwxr-xr-x 3 root root    4096 Mar 11 20:15 .
drwxr-xr-x 9 root root    4096 Feb 25 05:19 ..
-rw-r--r-- 1 root root 6307228 Mar 11 20:34 busy_workers.wsp
-rwxr-xr-x 1 root root   17308 Mar 11 20:15 busy_workers.wsp.bak


the old file is there, thats how big a file is that keeps data for a single day at 1min interval.
the other file represents a year at 1min interval. makes it easy to do capacity planning for monitoring.

find /opt/graphite/storage/whisper -type f -name "*.wsp.bak" | xargs -I{} rm -f {}

7 comments:

Jake Lundberg said...

You can use the --nobackup option for whisper-resize.py. It will automatically remove the .bak file.

Anonymous said...

How to pass comma separated retention to this script? e.g 1s:1d,1m:7d,10m:1y

Jay Kumar said...

use white space instead of comma,

e.g 1s:1d 1m:7d 10m:1y

Anonymous said...

Even after using --nobackup option, bak files are not created. But the size of wsp file has increased after changing retention policy. Can anybody help whats the issue here.

Anonymous said...

After you resize, is there an easy method to check what level a database is retentioned for? We have over 30+ graphite servers, and as we are changing the retention, it would be great to check that when we change from 183d, to 90d. That it is indeed done. If you run the resize command, it takes just about as long to go through the files again and would be great to have a command that would list the string it was set to. I've tried the whisper-dump command and it prints what I think is this information, although would have to do math each time, since it has seconds per point, and points, so I guess the seconds per point is the left side of the :, and if you divide that into the right side, that would be the number of points. Checking on the calculator... yeah it came out 90 days.

So I'll leave this comment here, in case someone is wondering

lyonn said...

Hi

Is there a way to know if the current schema-storage definition is being applied correctly?

For example:

storage-schemas.conf

[default]
pattern = .*
retentions = 1s:24h, 10s:7d, 1m:30d, 10m:2y


Greetings!

Fred said...

@Jason
you can use: "whisper-info .wsp" to get the currently configured statistics for a whisper database