SIMS Summer Update - Broken Wonde, Groupcall

[Tefters]

One member of staff (teacher) is still having same problem, so not convinced the patch solved whatever is the background problem.

 

 

Its possible that the patch has become incompatible due to other patches released since.

 

As mentioned previously, code in the patch just makes the DB ignore deadlocks. Since having the patch we have not had any deadlocks but the patch was made for us back in August.

[gumbygaz]

We're still having problems. ESS remoted on Friday to get a copy of the errors and sims database but have been silent since.

Still having the issues, edulink reports "Sims Critical: Requests are taking too long to process" - Very frustrating!

[Space_Munkey]

Just chiming in that we also have deadlock issues appearing. We've not yet installed CWP2-4, but I've seen on ess forums that a others have these installed and are still seeing it.

 

https://customer.support-ess.com/csm?id=community_question&sys_id=e975c32bfb205e185a34ff40aeefdc7a

 

The ess forum links if anybody is interested (not sure if links are allow, so sorry if not!)

[haci]

I'm a little bit behind on this one, our return in september has been a mixture of trying to get projects over the line that should have been completed in the summer, our SAN failing and trying to get servers restored to loaned equipment as well as all the other usual september stuff

 

I've had various reports since then but just haven't had the time/manpower to look at it until now.

I've scheduled the 30133 patch this evening, has that solved this for everyone or are any of the workstation patches required also?

 

Sorry should have been more specific, this is the deadlock issue not the wonde/group call integration thing

 

Cheers

Edited October 7, 2024 by haci

[Space_Munkey]

I have just actually had a response to my ticket I put in with our SIMS provider.

 

 

They have said that the 30113 & 30133 patches will not fix the errors, therefore they are recommending we hold out until an official statement is released from ESS.

 

Some staff are getting the error 3 times a day, but it's not all staff and it's random...

 

Hopefully it doesn't get too noisy and we're able to wait out the fix

[Tefters]

I have just actually had a response to my ticket I put in with our SIMS provider.

 

 

They have said that the 30113 & 30133 patches will not fix the errors, therefore they are recommending we hold out until an official statement is released from ESS.

 

Some staff are getting the error 3 times a day, but it's not all staff and it's random...

 

Hopefully it doesn't get too noisy and we're able to wait out the fix

 

Be warned that if this happens during registers that your teachers will need to go back in and make sure marks are correct again. This issue is a lot more serious than ESS seem to be taking it as a whole.

 

Funnily enough my receptionist just had a deadlock this morning for the first time since we were given the original fix just before summer and will investigate it in a moment.

Edited October 7, 2024 by Tefters

[Space_Munkey]

Yea good thing to consider - we're going to put out some correspondence to alert staff to this fact.

[Tefters]

So I just looked through my SQL deadlock report setup against the SIMS database and my receptionist got booted out trying to save Lesson Marks.

 

Also still seeing a large amount of deadlocks behind the scenes but luckily not affecting end users, mainly its failures in the DEX service which is used for SIMS Online / NextGen.

 

If I find some time in the next few hours I might write up some instructions on how you can setup an SQL report for Deadlocks to see how many you are getting and what area they are related to.

[coolhands]

So it's going to be something to do with that SIMS ID thing isn't it. Which has always been weird in my experience - you know the whole options online registration nightmare, where students can log in using google or MS etc as long as they click the correct obscure button on first use, but if they enter their school email address then it will create a different sign in method for them (SIMS ID) and anyone trying to help the kids log in ends up in the mire forevermore.

 

Pants

[Tefters]

So it's going to be something to do with that SIMS ID thing isn't it. Which has always been weird in my experience - you know the whole options online registration nightmare, where students can log in using google or MS etc as long as they click the correct obscure button on first use, but if they enter their school email address then it will create a different sign in method for them (SIMS ID) and anyone trying to help the kids log in ends up in the mire forevermore.

 

Pants

Pretty much.

 

Sims ID / NextGen is all intertwined behind the scenes through Azure.

 

Last time I bothered to mess around with NextGen which was about a year ago I managed to slip into Sims ID from the NextGen setup and end up in a rabbit hole of test resources, messed up redirects and all sorts.

 

Dex the service that manages all of this uses SMS (Sims Service Manager) and has been unstable for a long time.

They moved it from syncing every 20 minutes to pretty much live and with write back. Throw into the mix of other services that write back live such as Class charts etc... and you now have a very outdated SQL DB model being battered with transactions left right and center.

 

The Summer release definitely made big changes behind the scenes that have made deadlocks a big problem or at least brought them to the surface more (end user client crashes).

 

Our case was not taken very seriously at first and then escalated into weeks and months before we were thrown a patch. Many excuses ranging from isolated incident to patch can't be created due to cloud strike issues.

 

I've mentioned before but looking at the release notes in the SQL code of the patches majority if not all the coding is now outsourced across the pond to a 3rd party so add that into the equation when reporting, monitoring, troubleshooting along with all the other intertwined write back and sync systems both with ESS and from 3rd parties.

 

Now that more schools have finally run the summer patch, albeit a little late, it would appear that there are bigger issues a foot underneath and now the Sims forums are starting to light up like a Christmas tree too and ESS finally caving in to having a major case open.

 

Problem is now that Sims on-prem is going through a bad patch and NextGen is in no way looking like a finished product anytime soon. I'm still shocked they are getting away with having all this data online available through a simple login method with no MFA, not even Microsoft account linked when it's all sitting in Azure!

 

It's ok though because ESS asked schools to commit to 3 or 5 year contract renewals right. [emoji849]

Edited October 7, 2024 by Tefters

[lmgtfy]

So I just looked through my SQL deadlock report setup against the SIMS database and my receptionist got booted out trying to save Lesson Marks.

 

Also still seeing a large amount of deadlocks behind the scenes but luckily not affecting end users, mainly its failures in the DEX service which is used for SIMS Online / NextGen.

 

If I find some time in the next few hours I might write up some instructions on how you can setup an SQL report for Deadlocks to see how many you are getting and what area they are related to.

If you could when you get a chance I would really appreciate this. Cheers

[Tefters]

Instructions for DB deadlock reporting:

 

1. Open SQL Management Studio against your SIMS server.

2. Left hand menu, go to Management > Extended Events and then right click on the Sessions folder and select New Session.

3. General Tab - Give it a name, Eg. "Deadlocks" > Events Tab - Use the search events field and look for "database_xml_deadlock_report" and push it to the right column > Data Storage - Choose event file as the target and specify a location, ours is set at maximum 1GB and rollover 5 files > Advance tab - Change nothing.

4. Once made you should see your created session under the sessions folder, right click on it and start.

5. Sit back and wait a day or 2 and let the deadlock events roll in, its handy to have your end users report deadlock client crashes to you and keep a spreadsheet with date, time, what they were doing in sims and error message (ID in particular).

6. Come back and check you session report by expanding the session name and you will see something underneath it that you can double click, has the wording of "package0.event.blablabla".

7. A list of deadlocks will open on the right side of SQL Management Studio, you left click once on each one and at the bottom will be a details pain of each event, the last field says "xml_report", double click that to get the real nitty gritty details in XML format. You can read the first major line "process id=" and scroll along to see the process ID, user logon, date and time. You can also scroll down through the details to see what part the process was in when it deadlocked to help marry up with end users.

 

As per my previous post and screenshot you can then use this to count how many you are really getting a day. Yesterday I had 12 but no end users reported anything, I think mine have given up reporting to be honest so from our perspective its fixed but who knows...

 

EDIT: Added screenshot for guidance.

Edited October 9, 2024 by Tefters

[Tefters]

Let me know how bad yours looks, curious to see what its like on others.

[haci]

Let me know how bad yours looks, curious to see what its like on others.

 

I did install the NOLOCK patch but its seemingly done nothing to reduce them.

I've had 12 events in the last hour, again I think staff have got to the point they are no longer reporting.

[Tefters]

I did install the NOLOCK patch but its seemingly done nothing to reduce them.

I've had 12 events in the last hour, again I think staff have got to the point they are no longer reporting.

 

When i had the original problem before summer (i applied the Summer patch a few days after release), I was getting in the region of 60+ a day, mainly with Attendance Marks and Registers during tutor and afternoon as that's the main AM/PM register.

After doing all the hoops with ESS and waiting an extortionate amount of time the NOLOCK patch was created specifically for us and we ran it in and have only had 2 complaints since and the XML logging shows roughly 10 a day now but as mentioned I do not know if the users are getting booted out of the client and just logging back in or not as teachers get bored or reporting things very quickly and expect you to crystal ball everything.

 

Seems a little strange that it worked for me and 1 or 2 other users who reported on here that it helped them but then not others but looking at time lines and people posting on here it would seem they ran it after a few consolidated workstation patches were released after the summer update so wonder if something with them caused to patch to invalidate?!

 

At this point its just guessing as we don't know their coding etc... but seeing as their own forums are lighting up and they have announced a major case if now open it will be an interesting follow. #popcorn

[lmgtfy]

If it helps at all we applied the NOLOCK patch during September after the Sims summer update. We have applied the consolidated workstation patch 2 but I have been too scared to apply consolidated workstation patch 4, and consolidated database patch 2 out of fear these deadlocks will return.

 

Since applying the NOLOCK patch we have gone from reports of around 20 a day to the odd one if at all a week. I will put in place the report that @Tefters has kindly written up to see how many we're getting in the background still.

 

Thanks

[Space_Munkey]

We've just had word that Consolidated Workstation Patch 5 is now available. We're going to be running it overnight (along with CWSP2-4 as we're a little behind).

[psydii]

For what its worth, we have lots of stuff syncing to/from sims and have had no reported issues with deadlocks on CSWP4.

 

We went to the summer release very late, so all the third party sync products were updated well in advance.

 

We are a fairly large school, and registers are taken in a third party system and sync'd back to sims in near realtime-ish.

 

The upgrade to CSWP4 was the smoothest I have ever seen, but that is probably because it came out so quickly after we'd completed the main upgrade. Hopefully patch 5 will be as easy.

[broadld]

We are on workstation patch 4 and we are still having the deadlock issue. Its seems to mainly happen when taking registers.

I can't even think of a time when we have got to workstation patch 5! We seem to be forever updating SIMS.

[haci]

Just got this update on my ticket

 

Hi Colleagues

I've spoken to the department involved with dealing with the deadlock problems and they've provided me with the following information.

Patch 30113 was withdrawn as it did not deal with some of the deadlock issues linked with the SIMS Summer Release 2024. The new patch is expected to be available this week or early next week (w/b 14/10/2024). The new patch is based on Patch 30113, but should be more successful in dealing with deadlock problems.

Jim

[Space_Munkey]

We've been with CWSP5 for a week and seeing no improvement.

 

Anybody else had any luck or news?

[haci]

We've been with CWSP5 for a week and seeing no improvement.

 

Anybody else had any luck or news?

 

Hadn't actually noticed that patch, Doh!

Scheduled for later today but with half term now I won't know until they return if its helped or not.

as the release date is 1st oct I'm assuming it wasn't the patch that Jim on the Capta forums was referring too so hopefully there's something further to come.

 

I've had no update on my ticket.

[coolhands]

They need to bloody well get on with it. I’ve got various teachers still have various deadlock / crashing them out problems, it’s ridiculous. Who else in their job could take a few weeks to solve a problem? Yes sorry boss, I’ll get back to you….in a month or two

[davinderkundi]

I've definitely had some trouble with the latest SIMS update, too. It really messed up the Wonde and Groupcall links, which I rely on for syncing data with other systems. I was caught off guard since I expected everything to be ready after the August release, but when they pushed it out early, it left me scrambling to find workarounds.

[eddyc]

Further to the original patch, Monday [emoji637][emoji640]th ESS sent us a “testing” patch to try and resolve the deadlock issues and general poor performance. The patch made no difference to anything.