Cloud Hosted Solution Outages

[osmani45]

I am looking to advise a MAT on moving some of their services to a cloud hosted solution and wanted to know about any outages anyone may have faced in the past 12 months on any of their current hosted services i.e. on Microsoft Azure, Google Cloud or Amazon AWS.

Detail on what, when and how long the outage was would be very useful.

[Edu-IT]

Only thing we’ve had is broadband loss, so, get that part right.

[mjk]

I'm pretty sure they are all designed upon zero outage - even if an entire continent goes down the service will just flip over to another datacentre.

I agree with @Edu-IT that you need to concentrate on your connections to the internet. ie you need to have at least one redundant connection that connects to a different service provider and via a different exchange.

[Wave9_Lee]

I'm pretty sure they are all designed upon zero outage - even if an entire continent goes down the service will just flip over to another datacentre.

I agree with @Edu-IT that you need to concentrate on your connections to the internet. ie you need to have at least one redundant connection that connects to a different service provider and via a different exchange.

 

In terms of 'zero outage', this comes down to the redundancy you pay for, plus logical design. Also you need to understand the context of your applications - mail for instance, you would probably never know there was an outage.

 

In addition, many outages have occurred in Azure in the last day or two as forced patches are undertaken (resulting in VMs being taken down) as a result of the Intel chip vulnerability discovered this week.

 

Hosted platforms are great, but you lose some element of control and they're not always as cheap as anticipated. Quite often the benefits come down to reduced costs elsewhere (IT tech/more uptime/better performance/freedom to do other things) and predictable spend.

[SimonHooker]

We use AWS heavily. We've only had detectable outages during planned maintenance / update periods. No downtime in the last few years has been down to Amazon. The only time we ever really get downtime is usually down to shortcomings in how our cluster handles nodes restarting.

 

Relating to the latest security issues, in the end this even only resulted in having to reboot a single EC2 instance and then not a vital one.

 

As mentioned by Wave9_Lee, the key issue is that the cost can easily creep up. We benefit from the flexibility and ability to scale up more easily than if we provisioned our own hardware combined with being able to use their various products to reduce how much time we spend on looking after infrastructure. It does mean keeping an eye on the bills though and making sure we aren't just scaling up for the sake of it.

 

Compared to something self-hosted I would say that a decent cloud hosted solution would be more reliable however if looking at something primarily aimed at internal use that does rely on having a stable internet connection of course as already mentioned. But that's 1 moving part to have to worry about instead of having to worry about that on top of the infrastructure.

[ITGURU]

The more that is moved to the cloud, the less need for technical staff on-site. Then comes the "i cannot fix it as it's on the internet' response and no idea when things will be back online.

At least with services in-house if anything goes wrong you can jump on it straight away and resolve.

 

If i was relying on eveything on the cloud the job would be quite boring!

[CloudGeek]

I'm pretty sure they are all designed upon zero outage - even if an entire continent goes down the service will just flip over to another datacentre.

I agree with @Edu-IT that you need to concentrate on your connections to the internet. ie you need to have at least one redundant connection that connects to a different service provider and via a different exchange.

 

Sounds too good to believe.

 

When you search for “UK cloud outages” you get numerous outages over the past 12 months - some links are here:

 

https://istheservicedown.co.uk/status/amazon-web-services-aws/history

 

The 10 Biggest Cloud Outages of 2017 (So Far) - Page: 1 | CRN

 

Are you saying in UK schools using AWS / Azure were not affected in any of these outages?

 

Am I missing something?

[mjk]

Sounds too good to believe.

 

When you search for “UK cloud outages” you get numerous outages over the past 12 months - some links are here:

 

https://istheservicedown.co.uk/status/amazon-web-services-aws/history

 

The 10 Biggest Cloud Outages of 2017 (So Far) - Page: 1 | CRN

 

Are you saying in UK schools using AWS / Azure were not affected in any of these outages?

 

Am I missing something?

 

I've only used GCE and not AWS or Azure. Whenever there has been a problem the service just failed over so I've not known anything about those problems.

Just for the record; I didn't state that anything could based is completely bullet proof - but it is just more reliable than running certain services locally because I don't have to worry about things like power failure, some idiot site staff unplugging something, the building burning down etc.

[djm968]

Moving my MAT Schools to Azure and have not had any major service disruption in the 18 months I have been working with the Trust. Connectivity from the local site to any cloud service is always the weakest link.

[bennz]

Sounds too good to believe.

When you search for “UK cloud outages” you get numerous outages over the past 12 months - some links are here:

https://istheservicedown.co.uk/status/amazon-web-services-aws/history

The 10 Biggest Cloud Outages of 2017 (So Far) - Page: 1 | CRN

Are you saying in UK schools using AWS / Azure were not affected in any of these outages?

Am I missing something?

 

The first link contains dates and times without any supporting information, for accurate information on AWS service status and histroy go to https://status.aws.amazon.com/, AWS i believe is the most transparent company out there when it come to outages.

 

The best example of this is the s3 outage of feb 28th if you read the post mortem from it they go into extreme detail on the cause and what they have done to make sure it doesnt happens again, whats surprising from someone like me who uses AWS everyday is how many big names that were effected, the outage only effected 1 AWS region, s3 has built in region replication features to prevent this very eventuality.

 

AWS is also the only company i have come across where cost savings is one of their principles when building systems, however this does require understanding of how pricing works including the cost of things like data transfer. I do think its well worth it and you will save time and money if you do it right dont just lift and shift you have to do it right or you wont see the benefits.

[CloudGeek]

Moving my MAT Schools to Azure and have not had any major service disruption in the 18 months I have been working with the Trust. Connectivity from the local site to any cloud service is always the weakest link.

 

Sounds encouraging.

 

Did you also move school MIS and school finance to Azure? Which ones did you opt to? Any problems with performance?

[john]

One thing we see with our MS Cloud Services is service issues so things may not be hard down but impacted, so we have had mail-flow delayed due to issues with the cloud not processing efficiently or something going wrong in it for example. So MS don't call it an outage as such yet it impacts business for us. I don't think a week goes by without my O365 Admin App going ping with a service alert for something not working 100% right.

[CloudGeek]

One thing we see with our MS Cloud Services is service issues so things may not be hard down but impacted, so we have had mail-flow delayed due to issues with the cloud not processing efficiently or something going wrong in it for example. So MS don't call it an outage as such yet it impacts business for us. I don't think a week goes by without my O365 Admin App going ping with a service alert for something not working 100% right.

 

OK, if a week goes by without your O365 Admin App going ping with a service alert for something not working 100% right, how do think your school MIS reliability will look like?

[mjk]

OK, if a week goes by without your O365 Admin App going ping with a service alert for something not working 100% right, how do think your school MIS reliability will look like?

I suspect it depends on how it's designed.

Azure, AWS and GCE are all capable of running docker containers, so it is conceivable that a well designed MIS could just failover to another cloud provider.

Our services are designed to failover from our server room to a cloud service. We don't failover to another cloud provider because of the cost is too high but I don't see why a company like Capita wouldn't design their MIS to do that, but I do see why 0365 might choose not to.

[djm968]

Apologies just seen this.... most Schools are running cloud based PSF and hosted SIMS.

[john]

 

That is the report from O365 for the last 7 days.... you will note that some items are from beyond 7 days as they are still fixing them.... but nothing is "down" just degraded but the degradation could be an issue for you if you use those areas / items etc.. so I am certainly not saying don't use it just ensure you have good alternative options for if a degradation hits you and stops a process.

[mjk]

^ that's pretty shocking. I guess part of their strategy is to make you still reliant on an Office local install so making sure everything is 100% reliable in the cloud would almost be a negative thing for Microsoft. The Google up-times look a lot better in comparison, but of course it's more problematic when it does go wrong. https://www.google.co.uk/appsstatus#hl=en-GB&v=status

[SwedishChef]

We've been using Google here for the last year or so, have to say not had any issues at all (some issues have happend whilst we've been asleep, but would not have had much impact anyways). It's been a joy to manage and has enabled an already shrinking IT team to focus on other priorities and be more proactive. I would say a stable internet is essential (for that reason we moved away from our grid for learning, we are using Chromebooks which are capable of working on google documents offline, if ever needed.

 

To the IT staff who "hold back" their schools by not embracing the cloud, worried about their jobs. I can only say embrace it you might find it a lot less stressful and staff and students more appreciateive that they dont' just see you around the school when there is a problem, as well as learning new skills and helping their students acheive more. No students means no job!

(climbed down of my soap box now)

 

I've always said if the internet was down we have a problem if we were in the cloud or not, as most teaching resources are online!

 

Hope that helps.

[chaplic]

I think that report is a little misleading, Office365 is huge and these issues reported probably won't affect you at all, or are degradation rather than complete outages (Microsoft should do a better job in localising the reports)

 

You don't need 100% uptime and such a solution is too expensive for just about everyone.

[john]

I think that report is a little misleading, Office365 is huge and these issues reported probably won't affect you at all, or are degradation rather than complete outages (Microsoft should do a better job in localising the reports)

 

You don't need 100% uptime and such a solution is too expensive for just about everyone.

 

Afraid to say I disagree, I did indicate in an earlier post in this thread it was degradation rather than total outage, however we have users hit by some of these, it has a column to indicate when they can identify directly impacted users but its masked for confidentiality on my image, but what are those users to do if its access to OneDrive where all the files are stored for example? If that onedrive is the shared area hosting the snow chain perhaps and staff didn't download it.....

 

A number of the items in that image are general issues with the service (and it varies region to region so someone else looking at that report may see differently as they may not be on the same DC and Farm that we are), what I'm getting at is it is all about planning, the cloud for sure has a place to be in peoples strategy but just needs to be planned and ensure options for those what if situations occur as I suspect they will be straight at IT's door going emails down how do I email the notices out or how do I make a phone call if you use a cloud PBX etc..

[chaplic]

I think I broadly agree with you, my view on the usefulness of service status comes from looking after a few tenants, one with 50k seats. I gave up proactively checking the service status first thing, as there is usually some issue and then to find out no reports from users/ all OK our end. It's a bit like if you look at all train delays you'd think the trains never run at all. I've not done it yet but some agent doing synthetic transactions may be a better monitoring solution

 

Cloud is not a panacea, it does move the game on but in many cases simply changes the questions.