Performance Monitoring Tool

[pcstru]

We are having a lot of problems with time taken for web pages to load (users say things like "browser crashed" but while that isn't true - we can see very slow times for some pages). We need to monitor loading times in some way which replicates what a browser does (in terms of multiple request/response). I'm looking at producing some script (perl/python/bash - probably one of the first two) which drives wget and records the time to complete. Any other ideas for techniques/tools or potential problems in using wget?

[Dos_Box]

Are these local (intranet) pages or from the web in general? Thgere was a web page benchmarking site published here just recently which should be able to help you out.

[pcstru]

Are these local (intranet) pages or from the web in general? Thgere was a web page benchmarking site published here just recently which should be able to help you out.

 

It's internet pages that are slow, pages from servers on the local network are generally fine.

[bossman]

@pcstru:

 

One would presume that your school has the LA ISP for its internet connection?

Then it would most probably be their upstream proxy cache servers which may be adding to the increase in slow web-page loading.

They often don't get reset so they tend to lock up which causes a bottleneck for all the other schools who use those services.

 

We have had several accounts of this in the past with our LA ISP.

 

Check out your proxy cache first to make sure your proxy is not at fault.

[pcstru]

@pcstru:

 

One would presume that your school has the LA ISP for its internet connection?

Then it would most probably be their upstream proxy cache servers which may be adding to the increase in slow web-page loading.

They often don't get reset so they tend to lock up which causes a bottleneck for all the other schools who use those services.

 

We have had several accounts of this in the past with our LA ISP.

 

Check out your proxy cache first to make sure your proxy is not at fault.

 

We have checked as best we can. Our ISA server shows no particular issues and isn't stretched either processor or memory wise. We have one internal site which can hit the ISA server before coming back in and it's generally fine.

 

Unfortunately the LA also say they have checked their end and there are no problems there. I need something to break the impasse or give us all a productive way forward.

[m25man]

Where does the Layer2 connection from your LEA ISP Router/Switch connect to your LAN?

A simple topology diagram might help..

Is the ISA in a two NIC Firewall mode or a single NIC sideways Proxy?

 

In general the fewer switch hops between your gateway device and your core/distribution switch the better.

[pcstru]

Where does the Layer2 connection from your LEA ISP Router/Switch connect to your LAN?

A simple topology diagram might help..

Is the ISA in a two NIC Firewall mode or a single NIC sideways Proxy?

 

Our ISA server operates 2 NIC's one is the internal proxy address for all internet traffic, the second connects into the LA provided box which connects the paper cups and string to the other paper cup and string at the LA end ...

 

Response can be miserably slow during the school day. Edugeek used to be particularly bad - I think because of the multiple request/response (which is why I want whole page loads rather than ping etc) ... until ... I put it in the ... white list. Now it's OK(ish). That suggested to me that the content filtering was a problem but the LA have just come back with some data which they say shows the response from their box to be OK (i.e. it's our problem not theirs).

[annieneustar]

Edited March 29, 2011 by Domino

[pcstru]

 

Thanks but I prefer an open public discussion for a reason...

Edited March 29, 2011 by vikpaw
<Removed quoted unauthorized advertising>

[annieneustar]

We are having a lot of problems with time taken for web pages to load (users say things like "browser crashed" but while that isn't true - we can see very slow times for some pages). We need to monitor loading times in some way which replicates what a browser does (in terms of multiple request/response).

 

I would suggest some third party tools that can be considered as best practices: perhaps external performance monitoring - which monitors your website from outside the firewall (through a true browser), that way you can identify errors, and at the same time, monitor the availability and performance of your website. Most importantly-- obtain an end users perspective (staging environment rather than production environment)

 

In addition, load testing prior to launch is also considered a best practice where you are able to test capacity and id bottlenecks.

[pcstru]

Ah - sorry for not being entirely clear; the problem is not with our website but somewhere in the infrastructure between a users (student/staff) browser and the website they are visiting. So when a user accesses (say) BBC - Homepage, the page they want may take 10 seconds (or more!) to load or, when a class of 30 are directed by teacher to go to IANA — Example domains at about the same time, the site can take so long to load that people think their browser has crashed.

 

For progress : I've set up nagios and am using check_http as well as some custom wget scripts. However, none of these is really capable of doing everything that a browser might do.

[annieneustar]

based on your information (without knowing your IT environment)...it can be quite a few things.... might be your DNS, CDN, bandwidth..but its sounds like Internal DNS...

[bossman]

@pcstru:

 

Have you tried plugging a stand alone laptop directly into your external access route bypassing your ISA entirely. This would go directly to the internet and then you would be able to determine if it is your network LAN or the ISPs WAN.

 

Then if you find out its your LAN you could do some diagnostics by hiring a fluke tester and having some fun.

[tom_newton]

I use a few wget --mirror commands.. but that would be a bit sequential... threads in perl are messy, and LWP doesnt like threading.. so either perl, and fork wget commands, or python and its internal webget bits. nile_c might be able to give you a pointer there.

[pcstru]

@pcstru:

 

Have you tried plugging a stand alone laptop directly into your external access route bypassing your ISA entirely. This would go directly to the internet and then you would be able to determine if it is your network LAN or the ISPs WAN.

 

Excellent suggestion. Thanks. Did exactly that late friday and this mornings accumulated data clearly shows spikes and regular time outs on the LA side.