[ipad] iPad Updates Queuing/Throttled

[Michael]

Hello all,

 

Since Apple have released iOS8, they have introduced a queuing system for iOS devices - probably due to the large number of devices hammering Apple's servers, every time they release an new OS or update.

 

I can re-create this problem at multiple BGfL schools, and I'm curious to know if other schools be it in Birmingham or not are experiencing the same/similar issues.

 

For example, I had 16 x iPads running iOS8.1.1, with the 8.1.2 update averaging about 20MB. Out of the 16, 8 downloaded/installed the update and the remaining 8 failed. I tried reboots, disconnecting/re-connecting to the wireless network and still, it wouldn't download the update - it just sits there requesting the update.

 

If I then tether a problematic iPad to my phone using 3G, the update is downloaded and installed immediately. I can't find any documentation online on how Apple have implemented their queuing system, but it may simply be based on the school's external IP.

 

Has anyone else experienced these same issues? The BGfL have been helpful in trying to resolve this issue, but we're all still unclear as to how Apple have implemented their queuing system. All traffic is based on SSL, so it's difficult to inspect and to identify how Apple are restricting the flow of updates.

 

I can only conclude it isn't based on the device or its mac address, but rather too many requests made from one external IP (in other words, the broadband connection in question), say every 24 hours for example.

 

Any suggestions would be welcome!

Edited January 15, 2015 by elsiegee40

[Edu-IT]

Not going to be much help on how the queue works, but, could you add your own caching server or use Config Manager to push out iOS releases?

 

I have managed to update about 30 over WiFi without being blocked / failing.

[Michael]

I use offline *.ipsw for major OS releases (which makes sense), but when Apple release small updates like 8.1.1 to 8.1.2, it makes more sense to do it over wireless, updating multiple devices in one go.

 

I seem to keep hitting a ceiling and with no logs or anything to go on, it's frustrating to know whether it's something I can change say internally or if it's simply an Apple implementation.

[Michael]

I guess I like to plan ahead - most of the devices I support are mostly running iOS8.1.2 now, with a handful of iOS8.1.1's, but we all know as soon as you have them updated, sods law Apple will release iOS8.2 and I'm then back to square one!

 

One possibility of course is maybe the queuing system isn't proxy friendly, but I cannot prove this with no solid evidence!

[e4e]

Probably logged your IP address and seeing multiple devices downloading the same update and locked you out some how. Had the same problem with some of ours, but just done them as and when . .

[Michael]

Probably logged your IP address and seeing multiple devices downloading the same update and locked you out some how. Had the same problem with some of ours, but just done them as and when . .

 

So is there someone we can contact at Apple regarding this? Maybe to whitelist our external IP or register as a corporate client?

 

Just seems ridiculous that Apple appear to have ignored customers who have multiple iOS type devices!? Businesses, schools etc... there must be loads surely?

[Brimstone]

Hello all,

 

Since Apple have released iOS8, they have introduced a queuing system for iOS devices - probably due to the large number of devices hammering Apple's servers, every time they release an new OS or update.

 

I can re-create this problem at multiple BGfL schools, and I'm curious to know if other schools be it in Birmingham or not are experiencing the same/similar issues.

 

For example, I had 16 x iPads running iOS8.1.1, with the 8.1.2 update averaging about 20MB. Out of the 16, 8 downloaded/installed the update and the remaining 8 failed. I tried reboots, disconnecting/re-connecting to the wireless network and still, it wouldn't download the update - it just sits there requesting the update.

 

If I then tether a problematic iPad to my phone using 3G, the update is downloaded and installed immediately. I can't find any documentation online on how Apple have implemented their queuing system, but it may simply be based on the school's external IP.

 

Has anyone else experienced these same issues? The BGfL have been helpful in trying to resolve this issue, but we're all still unclear as to how Apple have implemented their queuing system. All traffic is based on SSL, so it's difficult to inspect and to identify how Apple are restricting the flow of updates.

 

I can only conclude it isn't based on the device or its mac address, but rather too many requests made from one external IP (in other words, the broadband connection in question), say every 24 hours for example.

 

Any suggestions would be welcome!

 

I have plenty of schools all around the West Midlands all downloading iOS updates without any problems, it's all down to your LA throttling the bandwidth when your device is connected through the Link2ICT proxy.

 

They are telling you porky pies as there is no queueing system for iOS Updates....However if someone else has a badly configured Caching Server that may cause you problems with all iOS and App updates when connected through the Link2ICT Wan.

[e4e]

So is there someone we can contact at Apple regarding this? Maybe to whitelist our external IP or register as a corporate client?

 

Just seems ridiculous that Apple appear to have ignored customers who have multiple iOS type devices!? Businesses, schools etc... there must be loads surely?

 

What I have found particularly when registering apple id's you can only do between 8-10 per day per ip. This seems to be known by several apple installers I have spoken to. One did mention you can get white listed for a time but never let on/knew how to get this done.

 

But the last techie I worked with from a apple accredited company commented about download speeds being better for users having a caching server onsite using the apple OS X server over other MDM solutions here they seemed to get slower speeds. But this was a general observation other then hard fact

[Michael]

I have plenty of schools all around the West Midlands all downloading iOS updates without any problems, it's all down to your LA throttling the bandwidth when your device is connected through the Link2ICT proxy.

 

They are telling you porky pies as there is no queueing system for iOS Updates....However if someone else has a badly configured Caching Server that may cause you problems with all iOS and App updates when connected through the Link2ICT Wan.

 

The Apple queuing system is discussed online, such as here or here so I know it exists as previously the 'requested update' message did not appear.

 

A badly configured caching server I agree is one possibility, but all schools are separated through VLANs, so it should be impossible for one school configuration to affect another - hence the whole point of VLANs to properly manage traffic through the BGfL.

 

I'm not so convinced about throttling bandwidth being the issue - even if they did, it would still download, just at a slower rate. It's just odd how about 8 can download without issues, then it's as if a block is applied for 24 hours.

[Michael]

What I have found particularly when registering apple id's you can only do between 8-10 per day per ip. This seems to be known by several apple installers I have spoken to. One did mention you can get white listed for a time but never let on/knew how to get this done.

 

But the last techie I worked with from a apple accredited company commented about download speeds being better for users having a caching server onsite using the apple OS X server over other MDM solutions here they seemed to get slower speeds. But this was a general observation other then hard fact

 

Well I am using the same Apple ID across iPads, so this isn't the issue - but that's still good to know for reference! I agree a caching server should in theory be better utilising your internal LAN and not your WAN to obtain updates. It generally hasn't been a problem in the past and using offline *.ipsw files has worked well - it's just unfortunate that small 20MB post updates seem to be struggling when they shouldn't.

[Brimstone]

The Apple queuing system is discussed online, such as here or here so I know it exists as previously the 'requested update' message did not appear.

 

A badly configured caching server I agree is one possibility, but all schools are separated through VLANs, so it should be impossible for one school configuration to affect another - hence the whole point of VLANs to properly manage traffic through the BGfL.

 

I'm not so convinced about throttling bandwidth being the issue - even if they did, it would still download, just at a slower rate. It's just odd how about 8 can download without issues, then it's as if a block is applied for 24 hours.

 

 

Those 2 threads relate directly to when iOS8 was first available to download, so you have millions of users across the world all trying to download at the same time, which would mean long waiting times. I'll say it again there is no queueing system for iOS updates

 

Vlans make no difference here....

All schools under BGfL are sharing a small pool of public I.P. addresses (in fact at the last count this was only 10). You need your own static public I.P. address for your own caching server otherwise when other schools try to update their devices and are given the same I.P. as your schools they will point to your caching server and not update, example below....

 

**This is what is happening with your devices**

 

Most LEAs tend to have a round robin type installation of proxy which means that they have a pool of multiple external addresses.

 

School 1 places a caching service on its network at 192.168.1.10. It goes online and gets the public IP 194.15.16.23. It registers this IP address with Apple’s Servers.

 

School 2 has 10 iPads that go online. 5 use 194.15.16.23 and are told that a caching service exists on their network and to use it. However they cannot connect since it is a different network entirely.

 

The other 5 are OK since no caching service exists on any other public IP at this time.

 

But if the caching service was granted a different public IP the next time it went online then the caching service would update Apple with the new IP and then the affected devices would differ.

 

 

 

Apple UK senior engineers have been into link2ICT to explain what they need to do to help schools, however if a school has been sold a Mac Mini Server and have turned on the Caching Server without knowing what this does this is going to randomly affect your updates without Link2ICT knowing about it.

 

In the last week I've been to see 3 schools and turned off caching services in those schools that just decided without any thought to turn this service on.

 

Lastly..

 

Using the same Apple ID across all the iPads is bad practice and you get into a real mess sooner or later

[Michael]

That's very interesting to read and I did often wonder do external sources (servers/websites) see one of the centralised proxies IPs or the school's own external IP.

 

By logic then there's only two solutions - either the BGfL have their 10 external IPs whitelisted or the BGfL would have to be totally re-designed, which just isn't going to happen.

 

Also if a caching server was mis-configured at the actual school it was hosted at, surely they'd notice it wasn't working as intended?

 

In your experience (if you don't mind me asking) how are Network Managers mis-configuring their Apple caching servers? Is it a specific setting or could it be one of many?

[Michael]

Also - like WSUS I suppose, is it possible for someone like Link2ICT to host a parent Apple caching server, then allow child caching servers in schools that specifically request it?

[Brimstone]

Also - like WSUS I suppose, is it possible for someone like Link2ICT to host a parent Apple caching server, then allow child caching servers in schools that specifically request it?

 

 

Schools are buying caching servers to eliminate the bottlenecks on their bandwidth when it comes to either;

 

delivering apps over their wifi (iMovie is over 600MB so trying to deliver this to 100 iPads in your school at once, forget it)

 

or updating apps and iOS.

 

The problem herein is that LEA Networks are set up as a WAN, with a pool of public I.P.'s that they randomly allocate to schools whenever a computer is accessing the internet, your router will request this I.P. from the WAN.

 

If a school buys Mac OS X Server and turns on the caching service that service then tells Apple it is available as a caching server and registers it's public I.P. on Apple's servers, if the I.P. changes then the caching server will register itself again with Apple's servers.

 

As before Link2ICT don't use a static set of public I.P. addresses for every school, it would be extremely expensive for one thing. If this public I.P. address is not static (which most schools have not done) then other schools in another LA area may be allocated that same I.P. when they try and update the iPad, therefore experiencing the same problem you are having. The problem can be completely random depending on whether your iPad is trying to access updates via the I.P. address of another caching server on the same network (which you are all on)

 

Link2ict would have no knowledge of a caching server on a school network and schools and lots of network admins I have met don't even know what this feature is for so I have turned it off to stop this problem.

 

To summarise;

Without a static public I.P. your caching server may work fine for you, but other schools without one will find getting updates for iOS a headache.

 

Apple can not permanently whitelist public I.P.'s they will only whitelist schools that are rolling out large numbers of iPads so that IT admins can create in bulk Apple I.D's from a single domain or I.P. address (Apple prefer a domain) You will only get a service like this from an Apple Education Solutions Expert that has direct links to Apple Engineers.

 

Yes while it is possible for link2ICT for have their own caching service which would work just like WSUS bear in mind that they are likely to need more than one caching server with lots of storage for every school.

 

Apple have updated caching server which now allows for you to specify multiple or specific public I.P. ranges and other networks by using MX Record and DNS.

 

https://help.apple.com/serverapp/mac/4.0/#/apdC36C9994-1533-4DCB-9CFF-870CB0FADCDB