browolf Posted February 13, 2009 Posted February 13, 2009 years ago I had the idea that all these computers in our school, sitting there using 1/4 or less now of their drivespace, could be turned into a distributed storage array. maybe to run somekind of vast shadowcopy array or storing video feeds from cameras. I envisaged each pc would have a driver than made the diskspace available to the array but invisible to users to the pc. I dont know how you'd make a backup array of many parts function, there would certainly be room for many raid component. It was just a crazy idea, light-years beyond my ability to create. any thoughts?
tmcd35 Posted February 13, 2009 Posted February 13, 2009 I had the same idea a while back. Windows does include the facility for creating such a distributed file system. The idea was shot down when I explained it to a colleague who then asked... What if the file you want is on a PC in Room 10 that hasn't been turned on yet?
browolf Posted February 13, 2009 Author Posted February 13, 2009 yeah it neads to be like that but feature raid capability and better security
Domino Posted February 13, 2009 Posted February 13, 2009 I think the problems and complexity arising from this far outweighs the benefits. Its not like disk is especially expensive is it..? Plus it'll have an impact on your network infrastructure to have many machines running DFS-like around the school
dhicks Posted February 13, 2009 Posted February 13, 2009 (edited) years ago I had the idea that all these computers in our school, sitting there using 1/4 or less now of their drivespace, could be turned into a distributed storage array. My thoughts exactly - I start sorting ours out on Monday, hopefully. We have a bunch of Windows XP machines (SCL all-in-ones) that we're going to fit second harddrives in. These machines also have a second network card each to avoid network traffic problems - these machines basically have their own dedicated SAN. The idea is to get all of those machines to each share their second harddrive as a network-available block device of some sort, then have a fast file server aggregate those shares into a RAID array which can then be shared back out as a Samba file share. The details of how, exactly, we share the harddrives out are a little hazy - I could have sworn I found a Windows XP driver at some point for a block-sharing protocol of some kind, but that's about all I remember about it. -- David Hicks Edited February 13, 2009 by dhicks
Chillibear Posted July 12, 2009 Posted July 12, 2009 I could have sworn I found a Windows XP driver at some point for a block-sharing protocol of some kind, but that's about all I remember about it. It may have been the windows DFS drivers you found (Distributed File System: Frequently Asked Questions) Otherwise I'd take a look at systems like Gluster (Gluster - Home) for a distributed file system or perhaps Cloudstore (WELCOME TO CLOUDSTORE) - been a while since I deployed one. Most are *NIX based systems I'm afraid and my experiences were all with deploying into *NIX computation cluster environments.
dhicks Posted July 13, 2009 Posted July 13, 2009 Most are *NIX based systems I'm afraid and my experiences were all with deploying into *NIX computation cluster environments. Nope - I definitely found a driver for Windows XP that could share a spare partition from a Windows machine to a Linux server as a block device. I can't remember the name of the driver, or even the protocol used, as I'm on holiday at the moment and not on my work PC to look stuff up. -- David Hicks
matt40k Posted July 13, 2009 Posted July 13, 2009 Would hate to try and recover any lost data!! Sounds like a crazy idea. If you want loads of disc space, remove the hdds, get USB pens and make the workstations thin client and make a load of file servers.
budgester Posted July 13, 2009 Posted July 13, 2009 Would hate to try and recover any lost data!! Sounds like a crazy idea. If you want loads of disc space, remove the hdds, get USB pens and make the workstations thin client and make a load of file servers. Bah, now your just being sensible, where is the fun in that I ask you.
matt40k Posted July 13, 2009 Posted July 13, 2009 Bah, now your just being sensible, where is the fun in that I ask you. If you want to do something fun, use RAM as storage.
tmcd35 Posted July 13, 2009 Posted July 13, 2009 If you want to do something fun, use RAM as storage. That's dynamic RAM not static RAM Watches as the cleaner unplugs the file server to plug the hoover in
SYNACK Posted July 13, 2009 Posted July 13, 2009 For real fun use the network itself as storage, racetrack memory over utp all fun and games till someone touches a patch lead.
dhicks Posted July 13, 2009 Posted July 13, 2009 Would hate to try and recover any lost data!! Sounds like a crazy idea. If you want loads of disc space, remove the hdds, get USB pens and make the workstations thin client and make a load of file servers. That's the idea of exporting the extra harddrives as block devices, so I can create a RAID array with them. I actually wanted to make a RAID-1 array but mirroring between three drives instead of two, just to be safe, but I don't think mdadm will let me do that. This storage network is intended for storing large video files, nothing else - more important files go on the normal file servers. -- David Hicks
browolf Posted July 13, 2009 Author Posted July 13, 2009 This storage network is intended for storing large video files, nothing else - more important files go on the normal file servers. - David Hicks I like that idea. I'm intrigued about this driver you found. how long before you get back!?
dhicks Posted July 13, 2009 Posted July 13, 2009 I'm intrigued about this driver you found. Ah - found the ServerFault post I did: Cluster file systems - Server Fault -- David Hicks
browolf Posted July 13, 2009 Author Posted July 13, 2009 Ah - found the ServerFault post I did: Cluster file systems - Server Fault -- David Hicks like WOW! "nbdsrvr What is it? NBD-Server is a program for sharing an image of a filesystem or a partition on your system with an Linux system over a network. This nbd-server can run on a Microsoft Windows system" nbdsrvr
dhicks Posted July 13, 2009 Posted July 13, 2009 like WOW! It might work, it might not - that driver is (exactly!) 6 years old tomorrow. I'd also be careful about the command line you use - one typo and you tell it to use the wrong partition, which could be embarrassing. I'm waiting until we get around to reimaging our workstations until I give it a try. We'll see if it works or if there's too much networking traffic created in just mirroring files. -- David Hicks
browolf Posted July 15, 2009 Author Posted July 15, 2009 I'm just trying it out but I'm confused over "filename must be a disk-identifier. Something like: \\.\PHYSICALDRIVE0 for the first phsyical(!) disk, \\.\PHYSICALDRIVE1 for the second physical disk, etc. When using a physical disk, you can can enter a partition-number. Default is partition 0. For example. nbdsrvr \\.\PHYSICALDRIVE0 9000 0" what/where are device IDs in winxp?
Guest monkeyx Posted July 15, 2009 Posted July 15, 2009 recently came accross a cluster TDB samba project and looking to play with it during summer See CTDB Anyone using this already?
Chillibear Posted July 15, 2009 Posted July 15, 2009 (edited) what/where are device IDs in winxp? The where: I expect they are listed in 'Device Manager', select properties for a given device and then look on the details tab - the drop down list gives things like Device ID, etc. EDIT: Just downloaded and looking through the readme for ndbsrv and I'm not really sure if the 'disk-identifier' mentioned is the same as the Device IDs. Best be careful! Edited July 15, 2009 by Chillibear 1
browolf Posted July 16, 2009 Author Posted July 16, 2009 (edited) clearly we're not the only ones having these crazy ideas: Virtual Large-Scale Disk System for PC-Room SpringerLink - Book Chapter cept you have to pay to read the whole thing found this on technet Microsoft Windows may report event messages in the event log for various hard disk device issues using the following syntax: The device, \Device\Harddisk#\Partition # if I do c:\>nbdsrvr \bob\bob 9000 1 nbdsrvr v0.2 (c) 2003 by [email protected] connection made with 192.168.0.8 error opening file \bob\bob: 3 failed to close handle: 6 or c:\>nbdsrvr \\device\harddisk0 9000 1 nbdsrvr v0.2 (c) 2003 by [email protected] connection made with 192.168.0.8 error opening file \\device\harddisk0: 53 failed to close handle: 6 Edited July 16, 2009 by browolf
Arthur Posted August 4, 2009 Posted August 4, 2009 There another distributed file-system called Tahoe. It's looks really good and pretty easy to setup. More info here... http://arstechnica.com/open-source/news/2009/08/p2p-like-tahoe-filesystem-offers-secure-storage-in-the-cloud.ars 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now