AngryTechnician Posted March 4, 2011 Posted March 4, 2011 (edited) Ever since upgrading my 2008 R2 servers to SP1, my DPM 2010 server (on Hyper-V) has been decidedly flaky. Since upgrading I haven't managed an uptime of more than 2 1/2 days before the server becomes unresponsive and spits out emails like this to the configured alert account: The following alert occurred at 04/03/2011 13:30:32. Status: Active Data source type: Data source: Mailbox Database 1849896325 Computer: exchange.angrytech.internal Description: Last 1 recovery points not created. Task is cancelled because some other task in agent is not responding on backups.angrytech.internal machine. For more information, open DPM Administrator Console and review the alert details in the Monitoring task area. The DPM server still responds to ping and appears OK at first, but I can't log in either via RDP or the Hyper-V console (it hangs on "Configuring remote session" or "Please wait for the user profile service" respectively. Remote RPC calls also time out. Both guest and host have been upgraded to SP1. No other guests are exhibiting problems. It's different data sources every time, seems to be just whatever happened to be running at the time. There's nothing in the event logs on either server that correlate with the timings of the problem, and no errors being logged recently that haven't been logged before. Is it just me or is anyone else seeing odd behaviour with a similar setup? Edited March 4, 2011 by AngryTechnician
Rozzer Posted March 4, 2011 Posted March 4, 2011 I did have that issue but just reinstalled the DPM agent on the exchange server. Seems to be ok now. Ross 1
AngryTechnician Posted March 4, 2011 Author Posted March 4, 2011 Interesting, thanks. I'll keep monitoring to see if the errors seem to mention Exchange more often than other servers. Did yours start with SP1 or was it just a problem you've had once in the past?
Theblacksheep Posted March 4, 2011 Posted March 4, 2011 Not had these errors yet, but exchange 2010 OS isn't on SP1 yet.
Bruce123 Posted March 4, 2011 Posted March 4, 2011 Out of interested what are you using for your archive backups for DPM 2010; USB HDD or tape? Bruce.
AngryTechnician Posted March 4, 2011 Author Posted March 4, 2011 Out of interested what are you using for your archive backups for DPM 2010; USB HDD or tape? Bruce. We're not taking any at present. The regular storage is an iSCSI NAS on the far side of the site, so all our backups are remote. In my entire career I've never had to restore anything that was more than a couple of weeks old or was a full system restore, and our current system covers that. We are currently discussing whether we start keeping monthly archives of our MIS data, but again, it's never been needed before. I certainly won't be using tape if we go down that route.
oxide54 Posted March 6, 2011 Posted March 6, 2011 In my entire career I've never had to restore anything that was more than a couple of weeks old or was a full system restore, and our current system covers that. how long you been at this? a week?
AngryTechnician Posted March 6, 2011 Author Posted March 6, 2011 About 10 years now, frighteningly. Maybe my users aren't as daft as yours?
plexer Posted March 6, 2011 Posted March 6, 2011 Oxide54: there was absolutely no need for your comment. Ben
oxide54 Posted March 6, 2011 Posted March 6, 2011 About 10 years now, frighteningly. Maybe my users aren't as daft as yours? my users are seriously daft. have you always been in education?
AngryTechnician Posted March 6, 2011 Author Posted March 6, 2011 Since 2005. I'm not sure I'd categorise the users I looked after in industry as any less daft though!
oxide54 Posted March 6, 2011 Posted March 6, 2011 Since 2005. I'm not sure I'd categorise the users I looked after in industry as any less daft though! no less daft no! but the SLA's are normally more stringent, and an underwriter after some policy related document tends to not let it go, whereas a teacher after a kids work will just tell them to re-do it in the end.
eejit Posted August 15, 2011 Posted August 15, 2011 Anyway, back to the issue at hand - we are having the same problem AngryTechnician. Did you ever find out what actually caused the problem? Thanks
Bruce123 Posted August 15, 2011 Posted August 15, 2011 We're not taking any at present. The regular storage is an iSCSI NAS on the far side of the site, so all our backups are remote. In my entire career I've never had to restore anything that was more than a couple of weeks old or was a full system restore, and our current system covers that. We are currently discussing whether we start keeping monthly archives of our MIS data, but again, it's never been needed before. I certainly won't be using tape if we go down that route. We are planning to implement DPM 2010 before September, and included in the project plan is to include longer term storage onto either tape or portable HDD. Main main concern about just D2NAS is if a hacker or worm hits your network, having the NAS offsite won't help. All it would take is for a worm to gain admin rights on your network, and it can potentially wipe all the HDDs on your servers, including the iSCSI devices linked to the DPM. There was a documented case where company backed up to an online NAS with no off-line backups, and this happened to them; the result? The company went under. I think it was an online-gaming company. I was at presentation by a well known and highly regarded (and rightly so) backup company; VEEAM (complete DR for VMWare hosted servers). I asked their reps the same question and it was as if it hadn't even ocurred to them. Their first response was that you can use RAID in the NAS (which wouldn't help) and then said you could always replicate the NAS to another NAS (again, wouldn't help), their third response was they you could replicate it into the cloud (might help if the provider does archive backups). I am sure they do have a solution, but this issue just seems to be off the radar generally. I was looking at either FireStreamer by a company called Cristalink.com (which allows DPM to archive to USB HDD) or an LTO5 tape drive. My main concern with Cristalink/FireStreamer is that it appears to be a company based around one product, which is mainly aimed at DPM 2010. Too many eggs on once basket So in the end we opted for a Quantum LTO5 Superloader 3 LTO5 with 8 slots (expandable to 16). With a max storage with compression of 48TB (3TB per tape). Placed the order last week It is worth nothing that during our research we discovered HP and other popular venders of backup hardware use the same drive LTO5 mechanism manufatured by Tandberg, so it probably isn't worth paying the premimum for an HP branded backup drive. We also ordered an SAS 6Gbit/s card and cable to link to the server to the tape drive. We plan to have the server, NASes and tape drive all together offsite, connected to the rest of the network with 100Mbit link. We concluded that you really can't base your tape drive/server in your main datacentre/server room as in the event of a fire all your DR infrastructure would be destroyed (even if the tapes are safe offsite it could take over 24 hours to get a new server/backup drive in place). Will try to keep people updated on how this project goes. Bruce. Leeds, UK. 1
AngryTechnician Posted August 15, 2011 Author Posted August 15, 2011 Anyway, back to the issue at hand - we are having the same problem AngryTechnician. Did you ever find out what actually caused the problem? Thanks No. At the moment I have the server on a nightly reboot which keeps me from having to intervene more than every week or so, but I am still none the wiser. I have even rebuilt the DPM server last half term to no avail. My next step (later during the summer) is to rebuild it without SP1 to see if it is SP1 on the DPM server itself that is causing the problem.
AngryTechnician Posted August 15, 2011 Author Posted August 15, 2011 Main main concern about just D2NAS is if a hacker or worm hits your network, having the NAS offsite won't help. All it would take is for a worm to gain admin rights on your network, and it can potentially wipe all the HDDs on your servers, including the iSCSI devices linked to the DPM. You are absolutely right - any online backup (onsite or offsite) is potentially susceptible to a sufficiently skilled or complex intrusion. The only system that is not susceptible to a remote attack is one where you have to be physically present to change the storage media. Really it comes down to whether the risk (and likelihood) balances out the manpower you would spend dealing with offline backups, and the additional cost if you implement both offline and online.
eejit Posted August 15, 2011 Posted August 15, 2011 No. At the moment I have the server on a nightly reboot which keeps me from having to intervene more than every week or so, but I am still none the wiser. I have even rebuilt the DPM server last half term to no avail. My next step (later during the summer) is to rebuild it without SP1 to see if it is SP1 on the DPM server itself that is causing the problem. Wow, that is frightening. We'll open up a case with MS if we can't get further and let you know what we find. Cheers.
AngryTechnician Posted September 18, 2011 Author Posted September 18, 2011 Just wondering whether you got any further with this eejit? I'm going ahead today with my rebuild onto Server 2008 R2 RTM and am wondering whether I'm still going to be on to a loser.
gshaw Posted September 19, 2011 Posted September 19, 2011 We are planning to implement DPM 2010 before September, and included in the project plan is to include longer term storage onto either tape or portable HDD. Main main concern about just D2NAS is if a hacker or worm hits your network, having the NAS offsite won't help. All it would take is for a worm to gain admin rights on your network, and it can potentially wipe all the HDDs on your servers, including the iSCSI devices linked to the DPM. There was a documented case where company backed up to an online NAS with no off-line backups, and this happened to them; the result? The company went under. I think it was an online-gaming company. I was at presentation by a well known and highly regarded (and rightly so) backup company; VEEAM (complete DR for VMWare hosted servers). I asked their reps the same question and it was as if it hadn't even ocurred to them. Their first response was that you can use RAID in the NAS (which wouldn't help) and then said you could always replicate the NAS to another NAS (again, wouldn't help), their third response was they you could replicate it into the cloud (might help if the provider does archive backups). I am sure they do have a solution, but this issue just seems to be off the radar generally. Sounds like exactly the situation I was concerned by when looking at going to a disk-only backup solution... seems a lot of people haven't considered that scenario in the rush to get away from tape. In the end we've gone for the best compromise we can make and doing Veeam to disk with 30-day retention then back up the VBK file weekly to LTO-5 tape (we're using reverse-incremental backup on Veeam so the VBK would be the latest full backup) That way at least we have something offline we could restore from. Also looked into a removable disk solution (Teralyte) but was far too pricey for what it was and prefer to have a different medium for long-term archive backups. Cloud would be great but the prices are still far too high and becomes another ongoing cost to commit to (very difficult with current budgets)
AngryTechnician Posted September 24, 2011 Author Posted September 24, 2011 One (hopefully final) update on the original issue: since rebuilding the server a second time, this time back onto Server 2008 R2 RTM instead of SP1, the server has so far been up uninterrupted for 4 days, which is significantly longer than it ever lasted when it was doing exactly the same backup tasks on SP1. I don't know what it is about my setup that causes it to choke on SP1, and at this point I no longer care. I'm going to hold off trying SP1 again until DPM 2012 arrives.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now