Theldron Posted September 13, 2016 Posted September 13, 2016 Hi guys, I need help. For a while I have been having an issue with our file server that we would have random server crashes. The server would just freeze up and I couldn't do anything. other than force shutdown and restart. This could happen a couple of days in a row other times it wouldn't happen a couple of weeks. Over the summer I changed to a another file server and moved all the files over to it, for more than a week everything has been fine, yesterday we had our first server crash on the new file server and today it has happened again. We have moved from a HP server - Server 2008 to a Dell server - Server 2012, I transferred the files over and it has been fine for over a week. There is nothing on the event viewer, virus and malware has been checked. I have reset all profiles. All i have left to check is if there is a problem with a switch or a PC ..... i just dont know what else to check!!! Any advice or help would be appreciated. Cheers
ITGURU Posted September 13, 2016 Posted September 13, 2016 Use all the available server tools you have to monitor for issues. BAD memory maybe? I once had a HP ML350 G5 server which did exactly that, randomly reboot without warning and it was the staff server. Turned out to be a firmware update missing on the power controller, once updated issue was resolved and now it runs without any issues. 1
pantscat Posted September 13, 2016 Posted September 13, 2016 Is the server completely unresponsive? 1
Theldron Posted September 13, 2016 Author Posted September 13, 2016 Yeah the server doesn't reboot or anything it just stalls completely and I cant do anything with it. I can't mouse click, I can't access file shares nothing. All i can do it press and hold power switch.
HPlum78 Posted September 13, 2016 Posted September 13, 2016 Has the underlying storage stayed the same when you moved servers? 1
pantscat Posted September 13, 2016 Posted September 13, 2016 This might help to get you along the way to troubleshooting it - https://blogs.technet.microsoft.com/askpfeplat/2015/04/05/how-to-force-a-diagnostic-memory-dump-when-a-computer-hangs/ 1
Theldron Posted September 13, 2016 Author Posted September 13, 2016 The new or new server to us as it was from our sponsor, but everything about it is different from the old server nothing was carried over hardware wise. I just copied over the data. We run ESET anti-virus.
Theldron Posted September 13, 2016 Author Posted September 13, 2016 The crashes didn't happen over the summer, so I was wondering if it could be a bad port on a switch or network card in a PC that could cause an issue?
truebluesteve Posted September 13, 2016 Posted September 13, 2016 Does the server crash at particular times of the day or are there any other patterns to it that you can see. Have you tried turning AV off for a while? 1
Theldron Posted September 13, 2016 Author Posted September 13, 2016 No its quite random yesterday it was 3:30pm today it was 9:10am. Previously it was at different times of the day as well. I just have disabled ESET (dont like doing it) so will see how it goes?
3s-gtech Posted September 13, 2016 Posted September 13, 2016 I had similar with a new server - it was a duff RAID card. Very difficult to diagnose such a broad issue - I would try a check on every disk in the array(s), have a look at temperature (thermal issues could cause this, but modern CPUs tend to keep going nowadays), plus a sweep through of any driver or firmware updates. It's odd how this has followed the old server behaving like this - does the new one have the same IP address? 1
HPlum78 Posted September 13, 2016 Posted September 13, 2016 Sorry I am still trying to understand how the file transfer has taken place? you say it a different server HP to a Dell so unless you have been really really unlucky and gone from a server with a fault to a new/ different server with the same fault the only thing I can see that may be the same hardware is the underlying storage as if its on a DAS host or a SAN fabric that's what I would look at first. If its been a lift and shift e.g you have copied all the data from one server to the other then its probably not going to be a storage related issue.... he says holding his breath! 1
Theldron Posted September 13, 2016 Author Posted September 13, 2016 3S-GTECH - I have checked RAID and all seems well, I have checked the disks, temperature is fine. The new server is a different name and IP address. HPlum78 - No it was a lift and shift, I robocopied all the data over. Its not a SAN or anything, its just a file server. A Dell PowerEdge 2900 on RAID 10.
truebluesteve Posted September 13, 2016 Posted September 13, 2016 Does it ever crash overnight/weekends or only during the day when it is being accessed by users? If it was OK over the summer when I assume there were no users, then maybe there is some lock happening when files are being accessed. We had an issue with a race lock happening and causing shares to freeze and it turned out to be our AV. 1
Theldron Posted September 13, 2016 Author Posted September 13, 2016 No it only seems to happen during the day, when the files are being accessed. I was the only one in over the summer and it didnt happen then. I have disabled AV at the moment, so see how it goes. If it is the AV what would be the best way to get round it not locking the shares?
truebluesteve Posted September 13, 2016 Posted September 13, 2016 (edited) No it only seems to happen during the day, when the files are being accessed. I was the only one in over the summer and it didnt happen then. I have disabled AV at the moment, so see how it goes. If it is the AV what would be the best way to get round it not locking the shares? I'm not familiar with ESET as we use Symantec and I think that it was the network threat protection that was causing the issue. We removed that component which solved the problem and left the anti-virus element in place. If it does seem to be the cause of the problem it would be worth contacting ESET Edit - I should also say that we tried a couple of Microsoft Hotfixes which related to the SMB protocol (I think) which can cause a similar problem on Server 2012. I cant remember the numbers of the hotfixes but a quick Google might help. Edited September 13, 2016 by truebluesteve
HPlum78 Posted September 13, 2016 Posted September 13, 2016 (edited) Just out of interest how much storage and how many users (active) at anyone time have you got? how many spindles in your server? and whats the spec of the individual disks? I am still not going to rule out H/W issue here, SMB across 2k8 and 2k12 possible but would have to understand the patching methodology. AV more likely if it's bashing your file server on every read write! but I am thinking a under scoped H/W config but need the details. more so the more I think about it, only when under load! hear hoofs think horses not zebra...... should try to use my hands instead of my face to type! Edited September 13, 2016 by HPlum78 more thoughts!
Theldron Posted September 13, 2016 Author Posted September 13, 2016 (edited) I need to check on the spindle info as we were given the server from our sponsor. But we are a small school of 320 students and about 150 PCs of which about 90 will be used at any one time. We have about 2TB of disks in RAID 10 on the server and about 850GB of data. Edited September 13, 2016 by Theldron
HPlum78 Posted September 13, 2016 Posted September 13, 2016 Probably worth firing up perfmon and adding some of the disk counters to it, just to see what the profile of the disk subsystem looks like over time. I am going to assume that the OS is not on the same set of disks as the file share but I would be tempted to add all drives to the logging. If we cannot see an issue there, I would then start looking at CPU and Memory but my gut feeling here is a bottle neck around storage subsystems I think I saw further up the thread about checking out firmware on the RAID cards and individual disks would be worth looking at that as one disk with a different firmware can cause issues also check that the disks are the same. I have seen some interesting server and SAN configs in my time, I am no storage expert but with around 500TB on the windows network I have seen a lot of configs.
forkies Posted September 13, 2016 Posted September 13, 2016 I had an issue similar to this at a past school, after a lot of tracking down using Impero to see who was logged on at every event we isolated it to a end user, if I remember correctly they were working on some file that was "damaged" effectively causing the server to crash everytime they tried to go in the same folder as the file. Not sure as it was a long time ago (5-6 years) but definitely was issue relating to end user opening a file/folder on the server. We removed the folder using eraser or something and the issue never reoccurred. Wish I had more info but best I can remember from prior workplace.
Theldron Posted September 14, 2016 Author Posted September 14, 2016 Forkies - That was a thought I had had, but tracking it down is going to be a bu**er. Was wondering if something like data deduplication would help? I have just read up on it. Trying to find a damaged file is going to be fun. I am off work today but I am watching the disk activity remotely.
HPlum78 Posted September 14, 2016 Posted September 14, 2016 (edited) I would be surprised at a damaged file causing this, the fact is you have copied the data means every file on the server has been touched by a read and a write, that operation right there would suggest that the files are all intact...... (just a thought) assuming that there was no errors. oh and if you are thinking that you have issues around disk integrity then the last thing I would do is start turning on dedupe. Edited September 14, 2016 by HPlum78
zag Posted September 14, 2016 Posted September 14, 2016 (edited) Remove the RAID, buy SSD's. Also seperate your OS drive from your data storage drive. Then you know if its a file causing it or hardware easily. Edited September 14, 2016 by zag
pantscat Posted September 14, 2016 Posted September 14, 2016 Remove the RAID, buy SSD's. Being devil's advocaat here (he loves that stuff) - why? SSDs are an expensive option for a fileserver.
Theldron Posted September 14, 2016 Author Posted September 14, 2016 The OS and data are on seperate drives. HPlum - Yeah good point. I think I am just frustrated by it all, so annoying.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now