Notifications
Clear all

Repository disk space not clearing  

Page 1 / 2

luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

I've found recently that our Pure installation won't clear the space used by previous backups. The used space indicated within Pure stays low, but the disk space used on the system indicates that it's using a lot more space, and slowly filling it up. 

I'm wondering if it will keep filling it up, or if it sees the systems available space and will use all it can regardless of how many snapshots it's configured to save? 


Quote
Marijan Kozic
(@marijan)
Member Admin
Joined: 2 years ago
Posts: 78
 

As you probably know, the first backup for a VM is always a full backup as all data needs to be saved. The subsequent backups are incremental backups which only save the data that has changed since the last backup. In order to restore an incremental backup snapshot, Pure needs to use the data saved in that particular snapshot as well as the data that was saved as part of that first full backup.

Removing a backup snapshot from Pure (e.g. because it has gone past its retention time) removes the data blocks uniquely referenced by that snapshot. However, data blocks that are shared with subsequent snapshots (data that has not changed in the following snapshots and was therefore skipped in the incremental backups) will be kept so that the remaining snapshots can be fully restored.

You can check the storage size of the latest incremental backup as well as the total size for the whole VM in Pure GUI.


ReplyQuote
luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

Thanks for your response and info. In this case the total disk usage recorded in Pure is quite different to the disk usage recorded on our NAS. Here are a couple of screenshots to illustrate:

removed link

 

We currently have a single snapshot as we're also backing up to tape using P5 on another server, if that's useful to know. 


ReplyQuote
Marijan Kozic
(@marijan)
Member Admin
Joined: 2 years ago
Posts: 78
 

Sorry, it looks like the links got automatically removed by a security plugin. Please try posting them again.


ReplyQuote
luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

Ah sorry. Have attached 3 screenshots in one image.

This post was modified 3 months ago by luke_tearle

ReplyQuote
Marijan Kozic
(@marijan)
Member Admin
Joined: 2 years ago
Posts: 78
 

Interesting. What does the Pure report under App Configuration >> Repository?

Also, is it possible that you have some external data (not related to Pure) in the same folder used for Pure backup repository?

I mean, it is also possible that old snapshots were not deleted properly but this would be the first such issue.


ReplyQuote
luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

Attached a screenshot of the repository info. 

Nothing else is that directory. 

root@Umu:/iSCSi_LUN/PureBackups# du -h --max-depth=1
159G ./VM-a1f298ed-678f-4405-7418-2d509d08a2d0
221G ./VM-9115f3d5-6356-4525-5360-db83bf5b57b1
321G ./VM-6e49c207-1ad8-4bac-499c-acbd1459bae9
397G ./VM-4914176f-f310-44a4-5e74-21284dd8232f
264G ./VM-cc5a070a-7f2e-49cd-4f0a-05f6caa37f49
323G ./VM-ca2d3c11-acac-42b9-522c-dd3efe152dca
16K ./VM-6e2dcf13-3b83-484e-4260-366ff30dd140
39G ./VM-bcdba9bf-e363-4872-7324-f1b7f34b1d20
52G ./VM-41d878ae-ea83-4935-470c-5fc13ab1d1e3
13G ./VM-4e3d6a6f-c260-408d-4aa5-d79921e19e55
1.8T .


ReplyQuote
Marijan Kozic
(@marijan)
Member Admin
Joined: 2 years ago
Posts: 78
 

@luke_tearle

Well, I guess you know in which folders to start looking.

Each of those folders corresponds to a single VM (you can check the VMware VM name by looking at one of the xml (*.xdata.*) files inside. There should be one xdata file for each snapshot. If you have too many, it is possible that the older snapshots were not properly removed.

Each virtual disk will also have its own subfolder with many additional subfolders containing the actual data. Running something like this will give you the total size for each snapshot:

find VM-41d878ae-ea83-4935-470c-5fc13ab1d1e3 -type f -name 1234567 -exec du -ch {} +
 

where you can replace the VM folder with the one you are interested in (or you can limit the command to a particular VD folder) and the "1234567" you can replace with the snapshot ID (numbers after "xdata" in file names) you are looking for.

 
However, if everything is as it should be, Pure should prune the expired snapshots before new backup so you could also check your logs to see if there are any issues there.

ReplyQuote
luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

What's curious is it looks as though there is only one snapshot per VM:

root@Umu:/iSCSi_LUN/PureBackups# find . -name *.xdata.*
./VM-a1f298ed-678f-4405-7418-2d509d08a2d0/VM-a1f298ed-678f-4405-7418-2d509d08a2d0.xdata.1607079982
./VM-9115f3d5-6356-4525-5360-db83bf5b57b1/VM-9115f3d5-6356-4525-5360-db83bf5b57b1.xdata.1607081503
./VM-6e49c207-1ad8-4bac-499c-acbd1459bae9/VM-6e49c207-1ad8-4bac-499c-acbd1459bae9.xdata.1607081715
./VM-4914176f-f310-44a4-5e74-21284dd8232f/VM-4914176f-f310-44a4-5e74-21284dd8232f.xdata.1607080090
./VM-cc5a070a-7f2e-49cd-4f0a-05f6caa37f49/VM-cc5a070a-7f2e-49cd-4f0a-05f6caa37f49.xdata.1607081055
./VM-ca2d3c11-acac-42b9-522c-dd3efe152dca/VM-ca2d3c11-acac-42b9-522c-dd3efe152dca.xdata.1607080904
./VM-bcdba9bf-e363-4872-7324-f1b7f34b1d20/VM-bcdba9bf-e363-4872-7324-f1b7f34b1d20.xdata.1606475709
./VM-bcdba9bf-e363-4872-7324-f1b7f34b1d20/VM-bcdba9bf-e363-4872-7324-f1b7f34b1d20.xdata.1607079823
./VM-41d878ae-ea83-4935-470c-5fc13ab1d1e3/VM-41d878ae-ea83-4935-470c-5fc13ab1d1e3.xdata.1606474802
./VM-41d878ae-ea83-4935-470c-5fc13ab1d1e3/VM-41d878ae-ea83-4935-470c-5fc13ab1d1e3.xdata.1607079603
./VM-4e3d6a6f-c260-408d-4aa5-d79921e19e55/VM-4e3d6a6f-c260-408d-4aa5-d79921e19e55.xdata.1607370853
root@Umu:/iSCSi_LUN/PureBackups# 

Which logs would be relevant? 


ReplyQuote
Marijan Kozic
(@marijan)
Member Admin
Joined: 2 years ago
Posts: 78
 

Pure logs can be found in "<Pure folder>/log" (e.g. /usr/local/pure/vmsn/log). The "vmsn.log" is the current one, with the older ones receiving rolling numbers from the latest (.000) to the oldest one (e.g. .132).

 


ReplyQuote
luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

From what I can tell in the logs there doesn't appear to be any errors related to clearing data of any kind.

Should there be any mention of clearing old data at all? 


ReplyQuote
Marijan Kozic
(@marijan)
Member Admin
Joined: 2 years ago
Posts: 78
 

There would be messages such as "Error pruning the old snapshot data" along with a more detailed description of the error cause.

It's interesting that only some of the VM folders have grown to such a big size. Is it just because other VMs are much smaller? Or could this be a remnant of some older error? Are those big folders still growing out of proportion?

Do any of the existing data chunks get deleted after a new backup?


ReplyQuote
luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

I don't see any errors pruning stapshot data. There are mentions of snapshots being removed though:

[28/Nov/2020:02:43:48][3411.7f27d7fff700][-tclthread-] Notice: vm.local: Verify: verify started for snapshot ::CSM::Snapshot::VM-ca2d3c11-acac-42b9-522c-dd3efe152dca@1606477666 (11/28/2020 00:47:46)[28/Nov/2020:02:43:50][3411.7f27d7fff700][-tclthread-] Notice: vm.local: Verify: creating new VM snapshot
[28/Nov/2020:03:23:28][3411.7f27cd793700][-tclthread-] Notice: Removed expired snapshots: 1605871468

Some of the VMs are are older, and the newer ones are smaller but definitely bigger than indicated in the Pure gui. 

One of the VMs is brand new with just one backup run so far. I ran a backup again twice. The first time the backup size shot up to 23G in the gui and on disk. The second time it appeared to get pruned in the GUI to 10G as expected, but the disk space is still at 23G.

So it doesn't look as though any data is getting removed at any point. I was curious about permissions but if it can write data it ought to be able to remove it, right? 


ReplyQuote
Marijan Kozic
(@marijan)
Member Admin
Joined: 2 years ago
Posts: 78
 

Hm... Pure correctly deleted snapshot metadata but somehow failed to fully delete actual data chunks for (some) expired snapshots. Good news is that Pure 3.0 performs better in this regard and should be completely immune to this particular issue. Bad news is that it is still in beta testing so I cannot fully recommend it for production. I can include you in beta testing, though, if you like.

Alternatively, one could manually delete the unnecessary data files (or write a script to do it) as the logic is not particularly complicated - in essence, each subfolder should only have data chunks corresponding to the active snapshots plus maximum of one additional (optional) older chunk.

Since you already have all your backups independently saved to tape, it might be interesting to try new Pure as I am mostly worried about bugs during transition from version 2 to 3. Once deployed, new Pure is very similar to the old one and if cloud support is disabled, it is hard to tell them apart just by looking. Also, I would be happy to arrange a Teamviewer session to help you install new Pure and resolve any issues should they occur.


ReplyQuote
luke_tearle
(@luke_tearle)
Active Member
Joined: 10 months ago
Posts: 10
Topic starter  

It's a curious one! I think the biggest impact is in inflating the backups to tape. 

Posted by: @marijan
Alternatively, one could manually delete the unnecessary data files (or write a script to do it) as the logic is not particularly complicated - in essence, each subfolder should only have data chunks corresponding to the active snapshots plus maximum of one additional (optional) older chunk.

How are these delineated between useful and not useful chunks, if they're labeled under the same snapshot? 

 

I'd be happy to spin up a seperate instance for Beta testing? This instance is currently running under a Netgear ReadyNAS as an app. I could spin up an Ubuntu instance perhaps, or whatever else works.


ReplyQuote
Page 1 / 2
Share: