Virus Problems Continue. Help!

[Gongalong]

Hi folks,

 

Our virus problems continue, and it’s getting a bit desperate.

 

As background, we saw the first signs of the infection via Sophos “threat detection” last Tuesday/Wednesday. Our backups suggest the virus hit predominantly the Friday before that (28th Nov). At least one thing the virus is doing is converting PDFs into EXE files. The EXE files presumably host the virus, but they also display the PDF if clicked on. Certainly if you browse to a directory on a PC using Sophos it triggers a “threat detection”, as does scanning any files/folders with these files in.

 

Sophos are struggling. Because this is a zero day attack their software isn’t doing a great job of cleaning the virus. Bizarrely their on-access scanner does clean it, but not their scheduled or right-click/forced scan. We are using their Source of Infection tool (Sophos Source of Infection Tool) to try and spot the problem, but needless to say are having no luck getting trapped locations to infect. Last week we were running scans continually, because it takes so long on the file servers, so inevitably things were slow but usable.

 

The bigger problem is this week. Particularly from Tuesday onwards the “network” has almost ground to a halt, specifically classroom PCs are so slow they’re unusable e.g. logging on/off, opening files from the network. I did ramp up the virus checking, so that both clients and servers are performing on-access scanning, so I have disabled the latter to see if that helps but it doesn’t seem to be. I can’t spot anything obviously wrong from a server load perspective – low CPU and RAM usage. Also we use ProCurve Manager, so I know the switches aren’t under load. It’s a bit less clear if the SAN is under load (we run a two host Hyper-V cluster, joined to a single SAN) although I’m doing some more monitoring of that. It’s possible there’s an issue there.

 

The catastrophe situation is basically if the students can’t use PCs, and that’s where we’re at.

 

Clutching at straws I’m debating whether to try a small roll out of MS Endpoint from SCCM, to see if that is better at spotting the infection. Other than that I’m stuck for ideas. Sophos should be continuing their research, but need their so called “Source of Infection”. To add to the problem I’m away on holiday (unavoidably) from tomorrow until the start of the Spring term, so we have a thirdline consultant providing cover.

 

Other than the above I’m stuck for ideas. Anyone have any suggestions?

 

TIA

[Steve21]

I know it's not particularly a fix, but if you're currently set to let exe's run from homedrives/appdata (assuming as you mentioned people clicking the "pdfs" and it showing it etc) can't you quickly put an SRP rule to deny all exe's running from these areas anyway? Not sure why they can run them currently (again assuming) but even if it's temporary while cleaning it up.

 

But on a more suggestion side, would suggest (if it is totally down currently as won't affect things), segment off the "server room" and just test with a few random computers to see if it's server side or network side that's causing the problem.

 

I'd also suggest using your "trial" of Malwarebytes to have a look locally on the fileservers too.

 

If you have found some infected files, might be worth checking the hash values on these to see if they are the same or if it's changing, as again could roll out an SRP rule for that hash currently.

 

Did you narrow down at all as to whats spreading it? Is it users clicking stuff, or is it self expanding etc?

 

Steve

[abillybob]

Do you know what the virus is called!? Sounds like something similar we had a while back now! Is it sitting on the Servers or the Client computers?

How is it running? Surely standard Users can't run files and thus would be blocking the spreading!?

 

I'd find out from Sophos where the Virus is actually located, I'd then look in that folder and see if I can manually just delete it, if that fails go to the Services or Processes and stop the .exe running, then try to delete it manually. If it's still falling take down the network for a bit (teachers will hate you but life goes on) restart the server or client machine in safe mode, while in safe mode clean out the Users profile and delete the Virus to where it shows on Sophos.

 

To temporarily stop it create a standard User and set the security permissions on the folder of where the virus is located to only be able to run on a standard User account although make sure they can't open .exe files, this will hopefully block it and disallow the user to read, run or write to it.

 

Apart from that without knowing the name of the Virus it's hard to give you a solution.

Edited December 11, 2014 by abillybob

[andydis]

can you upload an infected file to

Jotti's malware scan

once uploaded you can see what the virus is called and make a plan of action and who has the best software to remove it.

 

 

i would not believe the line "this is a zero day attack " , IMO it is sophos that is just not up to catching it / removing it.

[Gongalong]

Thanks all for the replies.

 

I know it's not particularly a fix, but if you're currently set to let exe's run from homedrives/appdata (assuming as you mentioned people clicking the "pdfs" and it showing it etc) can't you quickly put an SRP rule to deny all exe's running from these areas anyway? Not sure why they can run them currently (again assuming) but even if it's temporary while cleaning it up.

File Server Resource Manager, right? We use this to lock down EXEs for students, but I was thinking about widening it out for everything.

 

Student create EXEs when programming, but we've added in an exclusion so that files starting with a certain prefix are allowed.

But on a more suggestion side, would suggest (if it is totally down currently as won't affect things), segment off the "server room" and just test with a few random computers to see if it's server side or network side that's causing the problem.

We've killed a couple of tasks that were running on the server that may have caused a disk queue. Today the clients actually seem to be working again, so far.

I'd also suggest using your "trial" of Malwarebytes to have a look locally on the fileservers too.

Will do.

If you have found some infected files, might be worth checking the hash values on these to see if they are the same or if it's changing, as again could roll out an SRP rule for that hash currently.

OK, not sure how to do that, but I'll research.

Did you narrow down at all as to whats spreading it? Is it users clicking stuff, or is it self expanding etc?

My best guess is these infected PDFs. The virus is annoying enough to even change the executable file's icon to a PDF, so unwitting staff can easily mistake them, even though they've been told not to open PDFs.

 

(Also for abillybob) This is the virus that Sophos picks up: Detailed Analysis - W32/Agent-AKJF - Viruses and Spyware - Erkennen und Entfernen von Web-Bedrohungen, Viren und Spyware | Sophos - Threat Center - Sicherer Schutz von Mobile, Cloud, Endpoint, Encryption, Email, Web, UTM Firewall, Wi-Fi, VPN und Serv

Is it sitting on the Servers or the Client computers?

Both I suspect. Infected PDFs are in the data shares of the servers. I don't think the servers are infected, but they're hosting the files. Clearly some clients are infected, and I'm sure of those are staff PCs because of where the infection is occurring.

How is it running? Surely standard Users can't run files and thus would be blocking the spreading!?

I'd find out from Sophos where the Virus is actually located, I'd then look in that folder and see if I can manually just delete it, if that fails go to the Services or Processes and stop the .exe running, then try to delete it manually. If it's still falling take down the network for a bit (teachers will hate you but life goes on) restart the server or client machine in safe mode, while in safe mode clean out the Users profile and delete the Virus to where it shows on Sophos.

See the link above for the forms it takes, but because it's on client PCs it's basically an epic job. I'm hoping that Sophos can sort out their endpoint to properly destroy it.

can you upload an infected file to

Jotti's malware scan

once uploaded you can see what the virus is called and make a plan of action and who has the best software to remove it.

Thanks, will do that.

i would not believe the line "this is a zero day attack " , IMO it is sophos that is just not up to catching it / removing it.

The fact their on-access scanner cleans it, but their manual scanner doesn't worries me.

[difinity]

Upload an infected file to virustotal and see if it's detected by Microsoft.

I'm no virus expert, but i have spent the last 12+ years cleaning malware off my customers (and teachers when i was a school tech).

Whilst I think anti virus is ok at detecting an initial infected file from say a download, once a virus has got hold it's pretty useless at removing and dedicated malware removal software is required.

Sophos missed a Zeus infection, and malwarebytes only found it once the laptop had been put in safe mode.

However recently I have seen obvious infected files on computers, that were not detected by either malwarebytes or superantispyware and were only detected by 5 out of 56 scanners on virustotal.

With online obfuscators so easily available nowadays, it's not hard to see why they are easily missed by the scanners. And as the recent Sony hack showed, there is some nasty malware out there.

I'd turn off the network and clean the infection before the network grinds to a halt, if it's got slower since Tuesday, it seems Sophos is loosing the battle.

[Arreks]

You mentioned that your network is running 'slow', have you performed some connection and latency tests to see the actual speeds running through? have you also checked your outgoing connections? there's always the distinct possibility that the payload of the virus is to form it's own network of clients to be used for nefarious purposes! i.e a botnet, it wouldn't be very fun if your whole network is being used to DDoS something.

 

Do Staff have access to running .exe's as well? Might be worth temporarily disallowing executables for the whole school (for one day) while you get a better grasp of the situation.

[win]

If it's not on the servers, then grab some extra bodies - anyone willing to help with technical competence and start re-imaging.

 

Do as many clients as possible, and put a working AV on your servers. Sophos appears to be ineffective so change it for endpoint as you suggested. It will be a long task but at least it will get there.

[notalot]

Its a heavy handed approach but if done correctly it will work, I also don't know if its worth the time and disruption.

 

You could Isolate the network switch, clearing the Virus as you go either through manual repair or through pc reimage.

 

Once repaired the pc is allowed to reconnect to the infection clean network.

 

We had to do this when we had a infection of conflicker (in the early days before it got patched out).

 

Its a pain and will take some time, this relies on the fact that the servers are clean or can be cleaned.

[Gongalong]

Upload an infected file to virustotal and see if it's detected by Microsoft.

Some of the scanners here found a virus in the infected file, although I wonder if this is the root virus or just a side effect.

if it's got slower since Tuesday, it seems Sophos is loosing the battle.

Yes, very much so.

[abillybob]

Have you tried just stopping these processes running:

 

[h=5]Processes Created[/h]

[*=left]c:\Documents and Settings\test user\application data\softwareprotectionplatform\sppc.exe
[*=left]c:\docume~1\support\locals~1\temp\~3.tmp
[*=left]c:\docume~1\support\locals~1\temp\~4.tmp
[*=left]c:\docume~1\support\locals~1\temp\~6.tmp
[*=left]c:\docume~1\support\locals~1\temp\~8.tmp
[*=left]c:\program files\adobe\reader 8.0\reader\acrord32.exe
[*=left]c:\program files\internet explorer\iedw.exe
[*=left]c:\windows\system32\wsauth.exe
[*=left]c:\windows\temp\~5.tmp
[*=left]c:\windows\temp\~7.tmp

[Gongalong]

Unfortunately I'm now effectively on holiday, and not at the school. Are you referring to looking for them in Task Manager?

[Gongalong]

You mentioned that your network is running 'slow', have you performed some connection and latency tests to see the actual speeds running through? have you also checked your outgoing connections? there's always the distinct possibility that the payload of the virus is to form it's own network of clients to be used for nefarious purposes! i.e a botnet, it wouldn't be very fun if your whole network is being used to DDoS something.

ProCurve Manager isn't showing any high bandwidth usage across the switches. In terms of connection latency... Ping tests, or something more?

Do Staff have access to running .exe's as well? Might be worth temporarily disallowing executables for the whole school (for one day) while you get a better grasp of the situation.

They did to create on network shares. Today I added file screens across all shares to block the creation of any form of executable file.

 

Not sure how I block them from running EXEs?

If it's not on the servers, then grab some extra bodies - anyone willing to help with technical competence and start re-imaging.

 

Do as many clients as possible, and put a working AV on your servers. Sophos appears to be ineffective so change it for endpoint as you suggested. It will be a long task but at least it will get there.

Yep, we'll need to find something that works ahead of reimaging, otherwise my worry is the clients will just get reinfected.

Its a heavy handed approach but if done correctly it will work, I also don't know if its worth the time and disruption.

 

You could Isolate the network switch, clearing the Virus as you go either through manual repair or through pc reimage.

 

Once repaired the pc is allowed to reconnect to the infection clean network.

 

We had to do this when we had a infection of conflicker (in the early days before it got patched out).

 

Its a pain and will take some time, this relies on the fact that the servers are clean or can be cleaned.

I did wonder about this. We could unplug all the fibres from our core switch and work around the school, although it still means finding working AV.

[abillybob]

Unfortunately I'm now effectively on holiday, and not at the school. Are you referring to looking for them in Task Manager?

 

Yup. See how they're running first, have a look at the properties of the process, see how much resources it's using up and that it's definitely that making your network seem slower and not another issue.

Then stop the process see if the PC improves and make sure it's not set to run at Startup. Can you rename the extension on the PDF files that are infected back to .pdf? If so what happens does it just revert back to .exe again?

 

If once you have stopped the processes and everything seems ok again I'm sure there is somewhere you can stop those processes from running on your network, Group Policy maybe!? I'll be honest I'm not 100% sure with this but I'm sure someone else can confirm if you can or not. Then once it's all temporarily disabled you can go back and properly research how to delete the virus, try different Anti-Virus software and eventually get it removed all while it's doing no harm to your network.

 

Hope this helps.

[Arthur]

I would try doing an offline virus scan on each server using a bootable AV disc like ESET SysRescue Live, Avira Rescue System or BitDefender's Rescue CD. The virus/trojan won't be able to interfere with the cleanup process then.

[timbo343]

Sophos might have not detected it but try a-nother.. how about Avast, Avira to see if they are able to help - you never know. We had to install AVG on a standalone machine when conflicker hit to clean USB sticks as Sophos didn't pick anything up.

[synaesthesia]

Sorry, wanted to reply to this sooner but forgot. I concur with one important piece of advice given above; yank the network and disinfect from the ground up. If individual machines can be easily rebuilt (i.e. you use SCCM, CC4 etc) then don't worry about disinfecting those, just pull them from the network and worry about rebuilding them when the server is cleaned. That way you won't have to worry about reinfection. It is a hard approach but often that's the best way to do it.

[JJonas]

Have you tried Sophos technical support? Sometimes AV companies have specialist disinfection tools that can help clean up specific outbreaks. What was their advice on how to deal with it?

 

I would block exe's and PDF's on any network shares using FSRM then look at disinfecting the servers first with whatever tools will do the job.

 

For the labs find a virus rescue disk that will clean it and burn a couple of dozen copies of it. Use that to disinfect a lab. Install an antivirus that will keep it clean. Reconnect Lab to network. Repeat for rest of school.

[Gongalong]

Yes, been liaising with Sophos support since it was first found. I had hope initially, but they say they can only help if we use their Source of Infection tool to find it, and that's not working.

 

EXEs are now blocked, but I haven't blocked PDFs yet.

 

I agree re. the individual machine clean, I think that's the only way at this point.

[LiamH]

We once got hit by a zero day, it spread via network shares and we had to isolate the network (take down the switch's) and did a complete reimage in sections. We ended up setting up a few machines just to clone 5 HDD at a time and just went and swapped out hard drives. One of us would clone 30 hard drives while others were out replacing them, rinse and repeat.

 

Our main problem was if we missed one it would reinfect the network till Microsoft patched the exploit.

 

How are your backups? hopefully your PDF files are safe, i know we had a few bits of educational software that just pulled up PDF's, are they all okay?

 

if all of your runnable executable's follow a file name like you said maybe running a search on network shares for *.exe but excluding your allowed file name could help you get rid of a chunk of them.

[Gongalong]

We use DPM, so it's based on retention periods, but I've recovered from two dates to a temporary location away from the network just in case. I've checked the restores and they appear clean.

 

I've run a search on the network shares and there are around 40,000 EXEs. A large number are virus'd PDFs, but there's also a lot of other things that teachers are using.

[LiamH]

How about running a search for .pdf files and removing write access? may slow down the spread of infection even if it will be a pain to reset permissions after it is all over.

 

It might just mean you end up having the original .pdf along side a .exe with the same name...

Edited December 11, 2014 by LiamH

[DMcCoy]

Don't forget managed firewall rules for clients. I had it so machine/machine communication was blocked, only to/from the server. This will help stop reinfection if the servers can be cleaned and the clients are unable to spread it amongst themselves.

[speckytecky]

This sounds like the sort of happening where it would pay Sophos to send some of their technical people into your establishment and make a case study of sorting the problem out for you. It would enhance their reputation no end if they achieved that and be resources well deployed.

[jmak]

This is intended to be helpful (and not saying I'd easily take the advice I'm offering):

 

You are on holiday for something very important, otherwise you wouldn't be off for a long period. Give the problem to someone else. The holiday is authorised, so someone else needs to take responsibility. You almost certainly care more and would do a better job, but the school could find someone else to do this.