Outage Planned Tues 10th March 6am-8am

[SimpleSi]

Westfield status page says outage planned 6am to 8am , Tues 10th March.

So cancel anything that needs internet for at least the rest of the morning if not the whole day/week/.....

regards

Simon

[Geoff]

Sorry I should of posted about this when it came up on Schools Portal. But the information there is slightly more extensive. There is also an outage on the 3rd.

There will be some disruption to the CLEO Broadband Network on Tuesday 3rd March 2009 and Tuesday 10th March 2009 between 06:00 and 08:00

1) Tuesday 3rd March 2009 Routine Maintenance - replacement of faulty part.

Replacement of faulty routing engines at Lillyhall, Coniston & Whitehaven CLEO backbone sites. This will affect all sites fed from these core sites (maximum 15 - 30 minutes outage per site). It will occurr prior to school opening times to minimise the impact.

2) Tuesday 10 March 2009 CLEO Backbone rollout - Installation of the new backbone router in Lancaster as part of the new CLEO network re-design.

All CLEO sites are likely to experience some outages during this network at risk period. Work will be carried out prior to school opening times to minimise the impact.

CLEO have identified the risk level of impact to you as Low - Medium.

CLEO would like to apologise in advance for any inconvenience caused by the outages and thank you for your support while we carry out this essential work to maintain and upgrade the CLEO network. In the event that you continue to experience technical problems with your CLEO connection after these periods please report these to your usual CLEO support and fault reporting service:

The Westfield Centre

[SimpleSi]

I saw that on a school noticeboard yesterday - I think the one next week isn't designed to hurt us southerners - just them in the artic north

I've tried getting myself on the portal, head emailed them authoristation - got an email with user name - then tried applying for a password and computer said no

Didn't regconise my user name or email address - that was despite having emailed me my user name

maybe I've forgotton how to tupe - I mean type

regards

Simon

PS
Westfield contacted me yesterday and said both webservers have been fully rebuilt (said they'd moved up to using WinMe instead of Win98SE) so we can look forward to years of uninterupted service (might have made up part of this post )

[Geoff]

Westfield contacted me yesterday and said both webservers have been fully rebuilt (said they'd moved up to using WinMe instead of Win98SE) so we can look forward to years of uninterupted service

And I'm away this weekend, I'm going shooting wild flying pigs.

[pete]

Not in Lancs myself - but why aren't they doing this earlier in the morning to give themselves more elbow room for cockups?

[kirk]

We are constrained by the JANET national network at risk period for centain sites connected via the CLEO network. Planned maintenance can occur between 7am and 9am on Tuesdays, unless there are special circumstances. JANET have allowed us to potentially start work at 6am if needs be.

The outages yesterday affected some of these sites, so the engineers involved set out from Lancaster between 5.30am and 6.00am with the actual work starting from 7.30am. Due to the work being routine in nature and the planning involved we could confidently say the work would be finished within 30 minutes and not much elbow room was needed.

The outage on Tuesday 10th is grander in scale. Therefore the actual outage will commence from 6am but the engineers involved we actually be starting preparation and pre-outage work from 3am.

All outages announced are worst case and therefore have breathing room built in. Actual downtime to the network is always estimated not to fill the whole period announced. We also try to give two weeks notice for any planned outage.

Though sometimes because of unforeseen circumstances and conditions there are outages on the CLEO Network, the engineers responsible for designing and maintaining the network are passionate about what they do and do everything within their power to keep the network usable 24/7 while still constantly upgrading and improving it.

[bossman]

@kirk:
I find this interesting to say the least as our County is looking at a similar setup to CLEOs, what I find titillating is why do they have to do it in term time and not during school holidays which would have the least impact or is it business that drives the changes?

Just asking the question :-)

[Geoff]

I can't speak for LUNS or CLEO but certainly for LancsNGFL cost has been cited as the issue as to why maintenance isn't done overnight/weekends.

[SimpleSi]

Due to the work being routine in nature and the planning involved we could confidently say the work would be finished within 30 minutes and not much elbow room was needed.

And it was just an unfortunate co-incidence that LancsNGFL websites were slow to the point of unusability afterwards

(Don't mean to start an argument with my service provider but you want to try being on the receiving end of everybodys upstream service from schools not quite 100% availability - the outages multiply up )

I used to work for a big broadcasting company and I've been up cold mountains on Christmas Eve in order to keep programmes getting to viewers so a 6am start isn't that impressive

I wonder what JANET do during the night that is more important than teaching during the day

regards

Simon

[sinusoidal]

And it was just an unfortunate co-incidence that LancsNGFL websites were slow to the point of unusability afterwards

(Don't mean to start an argument with my service provider but you want to try being on the receiving end of everybodys upstream service from schools not quite 100% availability - the outages multiply up )

I used to work for a big broadcasting company and I've been up cold mountains on Christmas Eve in order to keep programmes getting to viewers so a 6am start isn't that impressive

I wonder what JANET do during the night that is more important than teaching during the day
Simon

I can confirm that those looking after CLEO are also regularly called out in the middle of the night to go up mountains to core radio sites and replace router and radio parts to fix outages that are never seen by schools the following morning.

There was a seperate radio power failure in Kendal on Tuesday morning which may have caused some slowness in that area, but there were no other problems following the planned outages; though I can't speak for the ngfl server farm, but I haven't heard of anything.

Believe me, we do always have the users in mind and we aren't just single mindedly doing as we please. This is partly why Janet enforce the outage window rigidly. I have worked in schools before and been on the end of outages and I understand the problems and disruption they bring. The network is being heavily upgraded at the moment in its entirety, and there is never a good time to have a planned outage. I for one would happily conduct network at risk work earlier, but we aren't permitted to. It's worth noting that a lot of pre-work did actually intentionally occur towards the end of last summer holidays - even this caused problems as there are a fair number of sites which are in over the holidays.

These upgrades are mainly to improve the core links and routers so that they can support all high schools having 100Mb and primaries 10Mb If you think about the 1200+ schools the network supports, that adds up to a fair amount of traffic and managing it isn't trivial. Indeed, part of these upgrades is to increase the amount of resilience in the network so that failure cannot cause such disruption.

We are gradually trying to address the problems identified by schools, and I hope we can work with you rather than be seen to be working against you with this. Unfortunately with all the shared requirements of the network (some sites have greater technical skillsets than others/some want different things from their connection - and I believe we must start to take that into account), changes and decisions can take a while to reach fruition.

[SimpleSi]

Thank you for an interesting post

The one thing I'd like to ask is (As it makes no sense to me) - why do JANET not let you start earlier than 6/7am?

Who is more important than us that needs the network running overnight?

Our antenna people normally started work at midnight and stopped before 6am - railway engineers work on Sundays wherever possible etc.

I didn't think that your work up in the north caused the effect we saw down here but at the end of the day - tuesday was a washout as far as web services go in Lancs - couldn't even access the status page to see what was broken

"At risk Tuesday am" days should be over - I'd suggest sometime Friday evening to 9am Sunday should be the "at risk period"

But - unless you are the head of whoever provides our "service" - thanks for your efforts - i used to be an engineer once upon a time

regards

Simon

[Geoff]

Yes, it's nice to hear the point of view of the CLEO/LUNS/etc engineer on the ground. It's not a voice we usually get to listen too.

[SimpleSi]

....well - lets see what happens in 9 hours time

:praying:

Si

[SimpleSi]

Looks like the work went OK (touching wood) - either that or it was cancelled

Simon