smarties11 Posted October 4, 2019 Posted October 4, 2019 (edited) Hi All, We migrated to Exchange 2016 from Exchange 2010 over the summer. I've noticed recently that the daily IIS logs are HUGE! They started at around 120MB per day (Exchange 2010 was about 60MB per day) - they've been increasing steadily in size and are now 2-3GB per day, the largest one being 5GB! I've looked at the logs and we have some ActiveSync devices (we allow all staff and Y11-13 students to use ActiveSync) that are creating ~300,000 lines per day in the log file! The vast majority of these are using the iOS/Android Outlook app, so this seems to be a factor at play. I've called 4 of the worst offending devices into the office and on three of them I've removed the Outlook profile and recreated it, and on the fourth I have removed the Outlook profile and set up the e-mail account in the native e-mail app. In all cases, the log file entries per day have decreased down to 3,000ish; although this is still much higher than the average device which is around 300 entries. I've looked at the iOS and Android Outlook app options and there appears to be no way of controlling the frequency of sync. I thought about blocking the Outlook app using a device access rule in Exchange, but then I wondered if it was possible to throttle the connection at the Exchange server instead. I've done some research on throttling policies but there are literally hundreds of options and I'm not sure which ones I need. I don't want to negatively impact on the user performance nor affect other clients e.g. Outlook desktop on our domain machines. Can anyone point me in the right direction with the relevant commands and best practices? I like to keep IIS logs for 1 year so that we are able to look back in time in the event of a breach, but there's no way we can afford the ~1TB of storage this would require! Thanks! Edited October 4, 2019 by smarties11
chaplic Posted October 4, 2019 Posted October 4, 2019 I've worked with many very, very security conscious clients whose security and event log correlation systems cost literally millions. I am fairly sure none of them are importing wholesale IIS logfiles for activesync. I'd focus your energy on ensuring security auditing is useful, and I dont think verbose active sync logs are the key part of this. If you disagree then such storage in azure on cool storage would cost a couple of beers.
smarties11 Posted October 7, 2019 Author Posted October 7, 2019 It's not the security I am concerned about. It's the fact that our logs have increased in size by 100x with no apparent reason. Clearly this will have a performance implication and cost for our server and out clients! I'm sorry but the answer to this is not 'just feed it some cloud storage'...
k-strider Posted October 7, 2019 Posted October 7, 2019 im seeing the same issue... i do wonder given its all text wheter NTFS file compression on the folder would help??? ive not tried it just airing it as a possibility as i bet the raitio would be really high..
smarties11 Posted October 7, 2019 Author Posted October 7, 2019 (edited) im seeing the same issue... i do wonder given its all text wheter NTFS file compression on the folder would help??? ive not tried it just airing it as a possibility as i bet the raitio would be really high.. Yes I'm sure it would help in terms of the storage space required. I might try zipping one too. I've done some more digging today and this is 100% an Outlook mobile app issue. On Saturday (our log file was 5GB), we had 3,300,517 IIS hits on our Exchange server. 3,102,193 (94%) of these came from the user-agent Outlook-iOS-Android/1.0. 2,917,697 (88%) of these came from the 5 worst offenders! I think we're going to have to block Outlook-iOS-Android/1.0 for now unless someone can suggest the best way to throttle these server side? We've got 121 users using the Outlook App so it won't be a popular decision if we block it! Edited October 7, 2019 by smarties11
Max_Power Posted October 7, 2019 Posted October 7, 2019 I've got same issue, gone from around 800MB a day to well over 10GB a day! Got NTFS compression setup on the folders and it doesn't help at all for me i'm sorry to say. Got the logs set to clear out daily, but its still puzzling and troubling. Sundays Log was 9GBs, today it already at 19GBs and its only 11am No server changes have been made, but...... around the time the logs started to increase in size (around the 25th September I noticed they started to get steadily larger) saw the release of iOS13!!! Could be utter pure coincidence but had issues with latest versions of iOS and Exchange in the past, so its not impossible i suppose. As more devices have upgraded it could see the size of the logs increase hence why the increase has been steady?
k-strider Posted October 7, 2019 Posted October 7, 2019 mine was 20GB on friday... it didn't fill the disk but did stop exchange accepting emial from my Postfix server with this message status=deferred (host xxx.xxx.xxx.xxx[xxx.xxx.xxx.xxx] said: 452 4.3.1 Insufficient system resources (SystemMemory we have 3 years with iPads so stopping the Outlook iOS/Andriod app isnt going to be an option :-/
smarties11 Posted October 7, 2019 Author Posted October 7, 2019 mine was 20GB on friday... it didn't fill the disk but did stop exchange accepting emial from my Postfix server with this message status=deferred (host xxx.xxx.xxx.xxx[xxx.xxx.xxx.xxx] said: 452 4.3.1 Insufficient system resources (SystemMemory we have 3 years with iPads so stopping the Outlook iOS/Andriod app isnt going to be an option :-/ You can use the built in iOS/Android mail app instead or GMail? No reason to specifically use Outlook!
smarties11 Posted October 11, 2019 Author Posted October 11, 2019 Just to confirm that the issue is definitely with the Outlook app. I took the decision to block the Outlook-iOS-Android/1.0 user agent in Exchange on Wednesday. My IIS log file from yesterday is only 300mb (vs 3-5Gb for previous days). I have also enabled NTFS compression on the IIS logs folder as suggested by k-strider. This helps with disk space and seems to be compressing the logs to around about 45% of their original size without any noticeable impact on performance. I have also posted a message on the MSDN forums (https://social.technet.microsoft.com/Forums/office/en-US/4132cc73-a67a-4fae-a998-b98d9ac55904/throttling-outlook-mobile-clients?forum=Exch2016CM). I have had a response identifying the ActiveSync (EAS) settings available and their default values but I'm still not sure what would be a good setting in a production environment.
Max_Power Posted October 11, 2019 Posted October 11, 2019 (edited) How have your users taken to not being able to use the Outlook app on their devices? What are you asking them to use instead? Dont particularly want to disable the functionality as we have 400 devices using Outlook app at the moment. Edited October 11, 2019 by Max_Power
smarties11 Posted October 11, 2019 Author Posted October 11, 2019 How have your users taken to not being able to use the Outlook app on their devices? Yeah fine. I e-mailed them first to warn them it was happening and asked them to use a different app (suggested either built-in Mail app or GMail). We've had a few pop in to ask for help setting it up but other than that it's been fine. We had approx. 120 users using Outlook.
Max_Power Posted October 11, 2019 Posted October 11, 2019 Just looking on iOS store now and there is an updated version of the outlook app released yesterday (4.6.0), fixes performances issues - installing that and will check logs to see if it reduces the number of requests.
smarties11 Posted October 11, 2019 Author Posted October 11, 2019 Just looking on iOS store now and there is an updated version of the outlook app released yesterday (4.6.0), fixes performances issues - installing that and will check logs to see if it reduces the number of requests. Interesting - let me know your findings!
Max_Power Posted October 15, 2019 Posted October 15, 2019 Interesting - let me know your findings! Another upgrade to the Outlook app (on iOS yesterday), now on version 4.7.0 and log files are back to normal sizes. Not getting as many errors in the logs, so hoping devices are updating to the latest version and that's resolving the issue. Made no other changes to the exchange server but since 4.6.0 release last week log files have been steadily reducing in size day on day! 1
smarties11 Posted October 15, 2019 Author Posted October 15, 2019 Another upgrade to the Outlook app (on iOS yesterday), now on version 4.7.0 and log files are back to normal sizes. Not getting as many errors in the logs, so hoping devices are updating to the latest version and that's resolving the issue. Made no other changes to the exchange server but since 4.6.0 release last week log files have been steadily reducing in size day on day! That's great news - I'll give it a week or so for everyone to update and then enable the app on our Exchange server again and see what happens!
smarties11 Posted October 30, 2019 Author Posted October 30, 2019 Just to add to this again, I've allowed the Outlook mobile app again and let users know. Quite a number of students have started using it again. The logs have increased in size, but nowhere near as drastic last time. With Outlook mobile blocked, we were averaging ~300mb per day. This week, on Monday it was 462mb and yesterday it was 526mb. Analysing the logs shows that some Outlook mobile clients are hitting our server ~11,000 times per day which is still excessive, but better than the 300,000 times before the app bug fix. Will keep an eye on it as other clients reconnect. Are you seeing any improvement @Max_Power and @k-strider?
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now