Deduplication across multiple servers / network drives

[RichCowell]

Hi All,

 

I'm going to be trying to reduce the data on the network, and I know we've got a lot of duplicated files on servers, having run the tool on the file servers, but I don't think there's any way of running that across multiple servers, I've got three file servers - shared drives, staff home drives, student home drives... so I'd like to see what's duplicated across them all, or at least the shared & staff areas.

 

Can anyone suggest a way of doing this, or reliable 3rd party software that will do what I need?

 

Thanks,

 

Rick

[mattstevenson2005]

Hi Rick

 

what tool have you been running?

 

Matt

[Steve21]

Personally I'd recommend against doing that. You're then making a single point of failure across all your servers if the one with the dedup chunks is offline. The fact you have the 3 servers says for some reason you wanted it split apart whether this is processing power, management or redundancy etc.

 

But if all you want is to find duplicated files rather than dedup them use Treesize Pro (demo I guess unless you want to buy it), then just map all the shares temporarily on one server and let it find the stuff. JAM Software - TreeSize Professional - Online Manual

 

Steve

[RichCowell]

Just the report / tool in the File Server Resource Manager - can't remember what it's called now - on one server to see how much of an issue it might be, and there's definitely a lot to be saved by clearing up...

[RichCowell]

Personally I'd recommend against doing that. You're then making a single point of failure across all your servers if the one with the dedup chunks is offline. The fact you have the 3 servers says for some reason you wanted it split apart whether this is processing power, management or redundancy etc.

 

But if all you want is to find duplicated files rather than dedup them use Treesize Pro (demo I guess unless you want to buy it), then just map all the shares temporarily on one server and let it find the stuff. JAM Software - TreeSize Professional - Online Manual

 

Steve

Sounds like what I'm looking for, thanks!

[rrrrr]

If you want to reduce the amount of space the data takes up, have you also looked at the dedup feature in server 2012? It wont tell you where duplicate files are, but changes the way they are stored on disk. I have used this and it reduced the used space on file shares by around 38%

[RichCowell]

If you want to reduce the amount of space the data takes up, have you also looked at the dedup feature in server 2012? It wont tell you where duplicate files are, but changes the way they are stored on disk. I have used this and it reduced the used space on file shares by around 38%

I'll check that out too, thanks

[stephenwelch]

Hi All,

 

I'm going to be trying to reduce the data on the network, and I know we've got a lot of duplicated files on servers, having run the tool on the file servers, but I don't think there's any way of running that across multiple servers, I've got three file servers - shared drives, staff home drives, student home drives... so I'd like to see what's duplicated across them all, or at least the shared & staff areas.

 

Can anyone suggest a way of doing this, or reliable 3rd party software that will do what I need?

 

Thanks,

 

Rick

 

Hi Rick :-)

 

I just discovered latest ver of CC Cleaner has this feature under tools. Might be worth a shot. Never used it though !! Good luck.

[pantscat]

If you want to reduce the amount of space the data takes up, have you also looked at the dedup feature in server 2012? It wont tell you where duplicate files are, but changes the way they are stored on disk. I have used this and it reduced the used space on file shares by around 38%

 

Definitely this - dedupe in 2012R2 is particularly good and the performance overhead is tiny!

[RichCowell]

Hi Rick :-)

 

I just discovered latest ver of CC Cleaner has this feature under tools. Might be worth a shot. Never used it though !! Good luck.

 

Definitely this - dedupe in 2012R2 is particularly good and the performance overhead is tiny!

Thanks - I'll look into both, as well as the previous suggestion [emoji1]

[RichCowell]

Personally I'd recommend against doing that. You're then making a single point of failure across all your servers if the one with the dedup chunks is offline. The fact you have the 3 servers says for some reason you wanted it split apart whether this is processing power, management or redundancy etc.

 

But if all you want is to find duplicated files rather than dedup them use Treesize Pro (demo I guess unless you want to buy it), then just map all the shares temporarily on one server and let it find the stuff. JAM Software - TreeSize Professional - Online Manual

 

Steve

 

I set that running and left it overnight - 800GB+ of duplicate files across the three file servers!! Now to deal with them

Edited August 4, 2016 by RichCowell