+ Post New Thread
Results 1 to 9 of 9
Windows Thread, Duplicate file replacer in Technical; Hi all, Further to my recent post about my battles with duplicate files, does anyone know of a piece of ...
  1. #1

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,617
    Thank Post
    514
    Thanked 2,442 Times in 1,890 Posts
    Blog Entries
    24
    Rep Power
    831

    Duplicate file replacer

    Hi all,

    Further to my recent post about my battles with duplicate files, does anyone know of a piece of software which allows you to choose a location for a group of duplicates and then it deletes all the duplicates and replaces then with shortcuts to the location you specified?

    The reason I ask is that doing it manually is fine for the few larger suspects, but as there are currently 236,000 files deemed duplicates, it may take a few thousand years for me to do it by hand...

    Cheers
    Tony

  2. #2

    Domino's Avatar
    Join Date
    Oct 2006
    Location
    Bromley
    Posts
    4,126
    Thank Post
    215
    Thanked 1,255 Times in 786 Posts
    Blog Entries
    4
    Rep Power
    505
    I don't know of a third party app that'll do it, but storage server 2008 has a single instance storage option (like the exchange one) built in.

    I'm going to start looking at it this month, so will report back if its any good!

  3. #3

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,617
    Thank Post
    514
    Thanked 2,442 Times in 1,890 Posts
    Blog Entries
    24
    Rep Power
    831
    From what I've read, it isn't included in 2008, only Storage Server, Unified Storage Server and Home Server...

  4. #4

    Domino's Avatar
    Join Date
    Oct 2006
    Location
    Bromley
    Posts
    4,126
    Thank Post
    215
    Thanked 1,255 Times in 786 Posts
    Blog Entries
    4
    Rep Power
    505
    Quote Originally Posted by Domino View Post
    I don't know of a third party app that'll do it, but storage server 2008 has a single instance storage option (like the exchange one) built in.

    I'm going to start looking at it this month, so will report back if its any good!
    ......?


    Okay, so maybe I could have put that clearer...

  5. #5

    Join Date
    Feb 2006
    Location
    Derbyshire
    Posts
    1,381
    Thank Post
    181
    Thanked 211 Times in 171 Posts
    Rep Power
    65
    Hmm, that's a tricky one - pretty much all apps that clean up duplicates assume you want them gone because you've forgetfully made copies all over the shop, not that you actually want the files still linked.

    One of the best apps I've used is Duplicate Cleaner (imaginative name) - Duplicate Cleaner - Find Duplicate files - Freeware it's fast, has a decent interface and a great selection assistant for choosing which dupes to wipe and which to keep, great when people insist on creating parallel folder structures.

    Realistically, I'd assume a lot of your dupes are caused by people doing the same and copying the entire tree across - so you end up with what would be 1 shortcut followed by say 200 dupe files under it. Might be worth giving it a whizz over with this and using the sort options to verify, then script the copying of a shortcut to your folder (\\server\teachers\pubdocs is the classic one here) after nuking 100% CRC matched dupes?

    Duplicate Cleaner can export the list of matched files to CSV - with a bit of a quick tidy up and search/replace in Excel you ought to be able to create a batch file to sort everything out quite easily?

    Edit : Just realised I read 'free' into your question when it wasn't necessarily there - NoClone is the classic shareware duplicate file remover, it definitely has an option for creating shortcuts and NTFS hard links on removal.
    Last edited by OutToLunch; 15th December 2008 at 10:54 AM.

  6. #6

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,617
    Thank Post
    514
    Thanked 2,442 Times in 1,890 Posts
    Blog Entries
    24
    Rep Power
    831
    Quote Originally Posted by Domino View Post
    ......?


    Okay, so maybe I could have put that clearer...
    Lol, that'll teach me to read it too fast. It isn't really an option though, as you can't use it for other, non-storage, tasks - which we need also.

  7. #7


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    Apparently Moodle 2.0 has a filesystem that does de-duplication.
    Just need to link it into an smb/cifs frontend like alfresco

  8. #8
    Hollie1985's Avatar
    Join Date
    Mar 2009
    Location
    Sutton Coldfield
    Posts
    115
    Thank Post
    81
    Thanked 24 Times in 14 Posts
    Blog Entries
    1
    Rep Power
    26
    i know of a clever software which is free that helps you identify space being wasted accross the network, duplicates, files no-one is using anymore and files that should not be on there -eg holiday snaps, look at MailMeter File Archiver - File Archiving - Saving Storage Space

  9. #9

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,619
    Thank Post
    1,231
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by localzuk View Post
    does anyone know of a piece of software which allows you to choose a location for a group of duplicates and then it deletes all the duplicates and replaces then with shortcuts to the location you specified?
    It's actually dead easy to write a script to do this, Just calculate the MD5 checksum of each file, place that file in a central repository and create a hard link to the file from the originla location. If you find a duplicate file just create a hard link, skip the copy stage.

    --
    David Hicks

SHARE:
+ Post New Thread

Similar Threads

  1. Duplicate File Finder
    By alan-d in forum Downloads
    Replies: 1
    Last Post: 4th January 2012, 09:07 PM
  2. Help me duplicate CDs!
    By Daniel12 in forum Other Stuff
    Replies: 19
    Last Post: 3rd December 2008, 05:56 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •