Logfile analysis

[iking]

When we needed to find out what a pupil has been viewing on the web, we used a marvelous tool written by Andrew Virnuls (I hope I have spelled his name corrrectly) called Proxy. It was designed to take MS proxy server logs and search for usernames or "bad" words. Unfortunately we now have logs which are in Extended Squid format, and although I found Webalizer, it seems geared to analysis of server traffic, not spotting what naughty pupils have been up to. I've temporarily set up a simple Access database which converts unixtime to "normal" time, and finds entries by uername IP address etc, but it is a bit clunky and certainly not very user friendly. Does anyone know of a tool to do this job (or know if Andrew has written anything similar which will deal with this format?

Ian

[Geoff]

http://squidalyser.sourceforge.net/

You'll need a webserver that understands PHP and has MySQL. A LAMP solution is probably your best bet there.

[leuven]

We too use squid
We used to use something called Squidalyser (or similar) but found it very processor intensive. It automatically updated a MySQL database every day, and allowed searches by username. Also, you can then choose to see all sites per username, all traffic, or just images. You can also set up groups of users or keywords to search for. I guess it would work best with a dedicated machine to run the database - running it on the same machine as Squid got us lots of 'slow internet' reports.
Right now, I've got a couple of enthusiastic uni students manually checking the logs in their spare time, using 'grep' on linux Works well. Also, I have been invading ICT lessons to do a demo of how we check the logs, and how we can see exactly what they did. No problems recorded since!
I'm thinking of getting the head of ICT to build my demo into the curriculum. Will save me lots of time if the kids take it on board.

[iking]

We are running all this on an RM Connect system with NT servers. The machine which creates the logs is running linux, but we can't install anything on it, so squidalizer isn't an option for us. Anyone know of a windows alternative?

[Geoff]

Apache + PHP + MySQL will run on Windows too.

[DMcCoy]

Apache + PHP + MySQL will run on Windows too.

So does Squid 8)

[Alex]

Apache + PHP + MySQL will run on Windows too.

So does Squid 8)

off topic

/me looks at Avatar

What corp/char?

[DMcCoy]

Apache + PHP + MySQL will run on Windows too.

So does Squid 8)

off topic

/me looks at Avatar

What corp/char?

off topic reply (it involves computers I guess)

Didn't know if anyone would recognise the EVE avatars

my in game name is Windle Poons and I'm part of PIE Inc.


On topic

I just use Wingrep on the ISA logs.

[Alex]

Apache + PHP + MySQL will run on Windows too.

So does Squid 8)

off topic

/me looks at Avatar

What corp/char?

off topic reply (it involves computers I guess)

Didn't know if anyone would recognise the EVE avatars

my in game name is Windle Poons and I'm part of PIE Inc.


On topic

I just use Wingrep on the ISA logs.

Woah PIE rock man

Was talking to one of your guys at the fanfest, which reitterated this further.

<< Gunstar Zero, Reikoku, BoB.

too much to drinkies ofo to bedos

[iking]

Back on topic again.... I'm still struggling with this - I have the source - code for an ms proxy server log analysis program, but so far I have to convert extended squid format log files to csv format before it will read them. (The fields are in a different order of course but I've managed to fix that.) I need an example ms proxy server log file to play around with. Can anyone supply me with one?

[Geoff]

MS Proxy was dead and buried years ago I thought. Do you mean ISA server?

[Dos_Box]

It might be one of those 'Fraken-networks' where you have to power them up at midnight and spend hours scratching your head as the kids run rings around the 8 year old security model.

[Ric_]

@iking: Do you mean ISA or the MS Proxy log format (which you can select somewhere in ISA IIRC)

[iking]

Thanks - I'm looking for a n example log in ms proxy format

[Ric_]

@iking: If you setup your proxy to produce reports in this format your problem should be solved. Pester (PM) me when I'm at work and I'll have a look for the setting - off the top of my head I can only remember that it's in the same place as where you can tell it to log to an SQL database - something that would be more powerful if you ask me.