OOPS - Disabled NIC on a remote computer

[RabbieBurns]

I made a big mistake today with my HyperV host back in scotland. I was remote desktoped into it, and had a couple of VM windows open. I thought I was disabling the NIC on one of those VMs, however it was actually the host nic. OOPS.

 

Now, all the VMs are still up and running fine and connected to the network, its just the host I cant get into.

 

Is there any way of getting back into this machine, seeing as though its pretty much disconnected? I dont think there is!

[SYNACK]

Unless there are other adapters that the host has active or you have some kind of remote KVM ability like vPro or iLO you may have an issue.

 

Technically with enough information one could bluepill it http://en.wikipedia.org/wiki/Blue_Pill_(malware) and take over the host that way but thats well outside the definition of easy.

[RabbieBurns]

Sounds complicated heh.. I knew I should have plugged back in the KVM! And I knew I should have plugged in more than one NIC... Ah well..

 

Someone will be able to log in locally next weekend I guess I can do without it till then.. What an idiot haha

[psydii]

Before I ask this question, it is important to realise I have (often) made similar mistakes, and nowadays I'm lucky enough to be able to generally leave server management to those more obsessive about best practice and procedures. It is also a theme of mine to raise awareness of the importance of doing things properly. So anytime I see a case where the root cause appears to be 'not doing things properly' I feel that we should all look again to understand how it should have played out, and not simply accept that mistakes happen. Sure they happen, but we (the Edu Tech) community must learn and improve.

 

Consider critically how you found yourself in this situation.

Question: Why when you 'knew you should' reconnect the KVM did you not? Especially when you realised that you'd be away from the servers for an extended period?

Does this reflect on other working practices that you follow?

What step could you take to ensure that this pattern of behaviour does not present ongoing risk to your ability to manage the service to your users.

 

Consider how this converstion might feel if you were having to have it with a senior manager because the oversight/mistake had effected service.

 

Personally since I realised that I was having sort of conversation with myself far too regularly, introducing a 'supervisor' into any situation where changes to servers are taking place has reduced the incidences of operator errors. Sometimes I'm the supervisor, sometimes one of the technicians is supervising me. Either way changes are monitored and sense checked, it keeps the ship tight. So far.

 

P.

 

P.S I'm not looking for you to answer necessarily, it is more a prompt for other SA's to consider how they might fall into the same type of mistake, and consider what they must do to avoid it.

Edited February 23, 2011 by psydii

[RabbieBurns]

thanks for the reply and points are all valid

 

I must point out however this is just a machine running at my parents place back in scotland that I use for testing purposes, and they use as a fileserver..

 

Reason i never reconnected the KVM was i was only home for xmas for a week and ran out of time as was out and about.

 

The main reason I have this test setup, is so I make any mistakes on this, and not on a live mission critical system..

[psydii]

It did cross my mind that your relaxed attitude might indicate that it was a non-critical box that you'd lost access to. Glad it was so, and also thanks for the update - laughing off what appears to be a potentially critical mistake can contribute to Group-Think acceptance of bad practice, and that's something we don't want to encourage!

[RabbieBurns]

thats me back up and running..

 

lesson learnt!