+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 21
How do you do....it? Thread, Logfile analysis in Technical; When we needed to find out what a pupil has been viewing on the web, we used a marvelous tool ...
  1. #1
    iking's Avatar
    Join Date
    Jul 2005
    Location
    Kelso, Scotland
    Posts
    26
    Thank Post
    1
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Logfile analysis

    When we needed to find out what a pupil has been viewing on the web, we used a marvelous tool written by Andrew Virnuls (I hope I have spelled his name corrrectly) called Proxy. It was designed to take MS proxy server logs and search for usernames or "bad" words. Unfortunately we now have logs which are in Extended Squid format, and although I found Webalizer, it seems geared to analysis of server traffic, not spotting what naughty pupils have been up to. I've temporarily set up a simple Access database which converts unixtime to "normal" time, and finds entries by uername IP address etc, but it is a bit clunky and certainly not very user friendly. Does anyone know of a tool to do this job (or know if Andrew has written anything similar which will deal with this format?

    Ian

  2. #2

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Logfile analysis

    http://squidalyser.sourceforge.net/

    You'll need a webserver that understands PHP and has MySQL. A LAMP solution is probably your best bet there.

  3. #3

    Join Date
    Oct 2005
    Location
    Midlands
    Posts
    8
    Thank Post
    0
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Re: Logfile analysis

    We too use squid
    We used to use something called Squidalyser (or similar) but found it very processor intensive. It automatically updated a MySQL database every day, and allowed searches by username. Also, you can then choose to see all sites per username, all traffic, or just images. You can also set up groups of users or keywords to search for. I guess it would work best with a dedicated machine to run the database - running it on the same machine as Squid got us lots of 'slow internet' reports.
    Right now, I've got a couple of enthusiastic uni students manually checking the logs in their spare time, using 'grep' on linux Works well. Also, I have been invading ICT lessons to do a demo of how we check the logs, and how we can see exactly what they did. No problems recorded since!
    I'm thinking of getting the head of ICT to build my demo into the curriculum. Will save me lots of time if the kids take it on board.

  4. #4
    iking's Avatar
    Join Date
    Jul 2005
    Location
    Kelso, Scotland
    Posts
    26
    Thank Post
    1
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Re: Logfile analysis

    We are running all this on an RM Connect system with NT servers. The machine which creates the logs is running linux, but we can't install anything on it, so squidalizer isn't an option for us. Anyone know of a windows alternative?

  5. #5

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Logfile analysis

    Apache + PHP + MySQL will run on Windows too.

  6. #6
    DMcCoy's Avatar
    Join Date
    Oct 2005
    Location
    Isle of Wight
    Posts
    3,424
    Thank Post
    10
    Thanked 488 Times in 428 Posts
    Rep Power
    111

    Re: Logfile analysis

    Quote Originally Posted by Geoff
    Apache + PHP + MySQL will run on Windows too.
    So does Squid 8)

  7. #7

    Join Date
    Jun 2005
    Location
    Leeds
    Posts
    113
    Thank Post
    0
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Re: Logfile analysis

    Quote Originally Posted by DMcCoy
    Quote Originally Posted by Geoff
    Apache + PHP + MySQL will run on Windows too.
    So does Squid 8)
    off topic

    /me looks at Avatar

    What corp/char?

  8. #8
    DMcCoy's Avatar
    Join Date
    Oct 2005
    Location
    Isle of Wight
    Posts
    3,424
    Thank Post
    10
    Thanked 488 Times in 428 Posts
    Rep Power
    111

    Re: Logfile analysis

    Quote Originally Posted by Alex
    Quote Originally Posted by DMcCoy
    Quote Originally Posted by Geoff
    Apache + PHP + MySQL will run on Windows too.
    So does Squid 8)
    off topic

    /me looks at Avatar

    What corp/char?
    off topic reply (it involves computers I guess)

    Didn't know if anyone would recognise the EVE avatars

    my in game name is Windle Poons and I'm part of PIE Inc.


    On topic

    I just use Wingrep on the ISA logs.

  9. #9

    Join Date
    Jun 2005
    Location
    Leeds
    Posts
    113
    Thank Post
    0
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Re: Logfile analysis

    Quote Originally Posted by DMcCoy
    Quote Originally Posted by Alex
    Quote Originally Posted by DMcCoy
    Quote Originally Posted by Geoff
    Apache + PHP + MySQL will run on Windows too.
    So does Squid 8)
    off topic

    /me looks at Avatar

    What corp/char?
    off topic reply (it involves computers I guess)

    Didn't know if anyone would recognise the EVE avatars

    my in game name is Windle Poons and I'm part of PIE Inc.


    On topic

    I just use Wingrep on the ISA logs.
    Woah PIE rock man

    Was talking to one of your guys at the fanfest, which reitterated this further.

    << Gunstar Zero, Reikoku, BoB.

    too much to drinkies ofo to bedos

  10. #10
    iking's Avatar
    Join Date
    Jul 2005
    Location
    Kelso, Scotland
    Posts
    26
    Thank Post
    1
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Re: Logfile analysis

    Back on topic again.... I'm still struggling with this - I have the source - code for an ms proxy server log analysis program, but so far I have to convert extended squid format log files to csv format before it will read them. (The fields are in a different order of course but I've managed to fix that.) I need an example ms proxy server log file to play around with. Can anyone supply me with one?

  11. #11

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Logfile analysis

    MS Proxy was dead and buried years ago I thought. Do you mean ISA server?

  12. #12

    Dos_Box's Avatar
    Join Date
    Jun 2005
    Location
    Preston, Lancashire
    Posts
    9,824
    Thank Post
    580
    Thanked 2,161 Times in 986 Posts
    Blog Entries
    23
    Rep Power
    627

    Re: Logfile analysis

    It might be one of those 'Fraken-networks' where you have to power them up at midnight and spend hours scratching your head as the kids run rings around the 8 year old security model.

  13. #13

    Ric_'s Avatar
    Join Date
    Jun 2005
    Location
    London
    Posts
    7,590
    Thank Post
    109
    Thanked 762 Times in 593 Posts
    Rep Power
    180

    Re: Logfile analysis

    @iking: Do you mean ISA or the MS Proxy log format (which you can select somewhere in ISA IIRC)

  14. #14
    iking's Avatar
    Join Date
    Jul 2005
    Location
    Kelso, Scotland
    Posts
    26
    Thank Post
    1
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Re: Logfile analysis

    Thanks - I'm looking for a n example log in ms proxy format

  15. #15

    Ric_'s Avatar
    Join Date
    Jun 2005
    Location
    London
    Posts
    7,590
    Thank Post
    109
    Thanked 762 Times in 593 Posts
    Rep Power
    180

    Re: Logfile analysis

    @iking: If you setup your proxy to produce reports in this format your problem should be solved. Pester (PM) me when I'm at work and I'll have a look for the setting - off the top of my head I can only remember that it's in the same place as where you can tell it to log to an SQL database - something that would be more powerful if you ask me.

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. IRIS Behaviour Management and Analysis Tool
    By Sylv3r in forum MIS Systems
    Replies: 15
    Last Post: 14th July 2009, 01:24 PM
  2. Free MS Tool Change Analysis Diagnostic
    By thom in forum Windows
    Replies: 2
    Last Post: 20th February 2009, 08:59 AM
  3. Peformance analysis
    By lancaster.david in forum MIS Systems
    Replies: 0
    Last Post: 24th May 2007, 01:17 PM
  4. CMIS Analysis
    By Kained in forum MIS Systems
    Replies: 0
    Last Post: 25th April 2007, 09:41 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •