[acs] Eportal - Register/Login slowdown crawl

[chaft]

Each morning begins the same with frantic emails telling me that staff can't access or get into registers and some waiting ages for screens to load. Then after the registration time the MIS comes to life and is usable again.

I've used process explorer to monitor the server but the cpu barely peaks above 70% , the network and disk don't particularly go mad at registration times and yet the performance greatly suffers.

 

We installed 15.1 of eportal/facility recently in hope that it resolves the performance but instead brought new problems of events being blank whilst keeping the same performance issues.

We're not exactly a huge school but does anyone have eportal not slowdown to absolute crawl at registration time?

I've followed their tech sheet for the problems:

...................................................................................................................................................................................................................................................................................................

 

 

Below are links to guides to enhance the performance of ePortal. Please run through each of these articles one by one if you are experiencing performance problems.

Before making any modifications please ensure that you have taken a back up of Facility Admin and made a copy of the ‘ePortal’ folder (usually located in C:\Facility\ePortal\Webapps, assuming a default install) to ensure you have a back up copy of the current configuration settings.

Checking the Server Details

 

• Check the current server specification - intel xeon 1.9ghz 4gb ram
  

 

• Available disk space on server -10gigs on all disks free
  

 

• Size of audittrail and transaction Log -cleared at half term
  

 

• Check for multiple installs of ePortal -none
  

 

• Check for open ports -nothing has changed
  

 

• Conflicts with other software -no new software except eportal updates
  

 

• Check the regional settings -england gmt
  

 

• Scheduled task for an overnight stop/start of eportal -yup we have 2 batch files
  

 

• Check network communication between SQL and ePortal servers (if separate) -they aren't separate
  

 

• Force network speed for adapters and switches -haven't tried yet, not sure if applicable
  

 

• Monitor disk usage -yes and it doesn't seem intense but what measurements should i compare too?
  

 

• Disabling Microsoft Group Policy to troubleshoot poor ePortal performance -really don't want to have to do that
  

 

• SQL server priority boost setting -not done yet
  

 

ePortal Configuration Settings

 

• Check the contents of the webapps folder -removed old files today
  

 

• Changing configuration files for roll call/registration -done
  

 

• Comment out alerts and behaviour bitlets
  

Updating/Removing Data in Admin

 

• Reduce number of Admin users -not sure if we have too many but we try to restrict this
  

 

• Check number of users with BMFULL access
  

 

• Set behaviour alerts to expire -i am looking into this at the moment, but we are debating whether we will lose important information
  

 

• Remove all expired messages
  

Regards

Paul

 

.........................................................................................................................................................................................................................................

 

 

Is anyone else having Eportal issues of slowdown? have you resolved yours? if so please help me.

[Oaktech]

Does it make any difference if you restart the machine it runs on?

 

We had a similar issue - except ours stayed slow, and the answer for us was to set a scheduled task to reboot the server at about 1am and set the services to delayed start. It wasn't pretty, but it made it much more reliable.

 

We were pointed to restarting the eportal services, but we found that just restarting them didn't fix it. Sometimes restarting the IIS worker process for eportal helped, but the foolproof, worked every time solution was a bounce of the whole box.

Edited May 18, 2015 by Oaktech

[tomearnshaw]

We have Facility on one server and then two other servers with just ePortal on them. Teachers then have a link to a simple static HTML page with links to each server. If one server has issues, they simply use the other. We even have a link on the page that opens a helpdesk ticket so they can report if a server has fallen over. This means we can also update one server and push everybody over to the other one. Also means we don't have to kick everybody out of Facility when ePortal falls over and won't get going again without a reboot.

[chaft]

I'll try the restarting the pc option. Thanks. I'll let you know if it improves the situation

[Achandler]

Are they connecting via the IP or via a DNS name, if a DNS name they are going in and out are them?

[chaft]

A DNS name

[Achandler]

When you ping the dns name, does it reply with the internal IP?

[chaft]

well I assume dns as they don't type a IP address in. They have website address but it's destination on one of our servers ( http://chsmis/eportal/index.jsp)

i don't get an ip address from pinging this. It's an internal file.

[Achandler]

yeah its DNS name, you can change the chsmis to the server IP and see if it is any quicker.

 

You should also be able to ping chsmis.domain.local

[chaft]

yes i can get an ip http://(local ip)/eportal/index.jsp

will this help with performance of the site if i point people to this address instead?

[Achandler]

No harm in getting some people to try it, will prove if it is a DNS issue or not.

[chaft]

rightio have sent the ipaddress out to some staff and will see that helps or not

[JJonas]

I talked to our Network manager about this. He said...

 

Make sure your temporary internet files are set to check for new versions of the page every time you visit the site.

 

and

 

When was the last time you shrunk your SQL transaction log in the SQL management studio?

[chaft]

my boss says easter but is there a way i can double check whether the transactional logs were done or not and when?

 

the temporary internet files option is a group policy thing the users don't have access to change themselves. I'll have to think about whether that impacts anything else but it has been recommended for other eportal events issues. Not ruling it out though

[Michael]

The only thing that sticks out (that no one's mentioned) is the lack of RAM - 4GB is tiny. Most MIS servers I support running Facility and e-Portal (on the same server) have 16GB of memory and hovers around 6GB-9GB on average. The high CPU could be an indication the swap file is being heavily used.

 

Another thing to check is Write Caching is enabled on your RAID array. Without it enabled, many requests can bring the server to a crawl.

 

Unless you're still running a 32Bit OS, then there should be no reason why you can't increase the RAM. It's relatively cheap to do.

[chaft]

The ram is a little low, but most of the time the 'available' reserved memory is 500mb and using process explorer i've never seen physical memory peak to it's maximum. The PF hardly changes from 4gigs and I have set it so it can grow to 7gig. I'd admit the 4gb is low for such a MIS but it's an old server and has for years been fine at the role of doing eportal. The server is exclusive to this role. The cpu isn't really taxed that hard in my opinion. I rarely see it peak to 70% of it's full potential. mostly hovers between 5-60%. There is plenty of idle time too.

 

I've suggested adding more ram but this is an old server and adding ram is a risky maneuver even if it seems the obvious thing to do for performance. I've seen servers die after hardware changes. Long term we are moving the server to a visualized environment but now it's about keeping stability and lasting it out to the summer but resolving the performance as best as we can with what we have. It went from working fine to being slow recently.

 

update-- i've had a few respond positively to the IP address link.. will update later if it holds throughout the next series of periods

[JJonas]

my boss says easter but is there a way i can double check whether the transactional logs were done or not and when?

 

You can see the size of the log file under "currently allocated space" if its too big, shrink it.

[Achandler]

You can move ePortal in about 45 minutes, I say move, I just mean install from scratch on a new server.

 

15.1 is a pile of garbage, full of bugs and all sorts.

 

With regards to events being blank, are we talking about behaviour events not showing? Theres a bug with them as well that I think is fixable.

[Michael]

To be honest I don't think this is a DNS issue - specifying the IP rather than the FQDN may process the request fractionally quicker, but it's the actual resource (in this case e-Portal), which is slow.

 

There's still 7-8 weeks or so left until the summer, so I still think increasing the RAM is advisable just to keep things working. Windows Server will never use the full 4GB of available memory, so it does very much sound to me the swap file is being used to compensate - be it badly. If you are running a 64Bit OS, I would increase it to 8GB. I think this would definitely ease the situation.

[Banjo]

Things may have changed, but eportal used to hammer a single core. Overall it looks like the multi core processor is not doing much, but one of the cores may be at or close to 100%.

 

When we used to install facility the app and SQL server used to be memory heavy (16GB+) and the eportal servers - fast processors speeds >(3Ghz).

 

Some schools always experienced slow downs as the dataset grew throughout the year.

[chaft]

Memory is lower than we would like for an MIS server but it's an old server and adding memory would be ideal but at the moment we don't have the budget to as we are moving to a new system in the summer. Also it's a little risky as any hardware change could cause stability issues. That aside the available reserved memory is about 500mb and the page file never fills up more than half way. The cpu isn't too high, it peaks at 70% at times but balances out between 5% and 50% most of the time with plenty of idle time too. Again upgrading that is ideal and in the summer we will move to a visualized server. For now I'm just trying to get the best of the system without too much risk or expense. This system did work for the last 7 years fine. Only recently it hasn't

 

update on the ip address link---- It's working well. I can't explain or don't understand why it would work better. Does anyone have any idea why as I'd like to learn why it might impact performance.

[chaft]

I will be able to know for sure by friday if the ip link resolves the problem but so far we've had positive responses.

[chaft]

we also now restart the services an hour before registration each day in the morning. although we had the resets in the evening i was concerned that i heard it often needs two resets on the services to actually have an effect. hopefully this too will help. the test will be whether the registers get slower in the afternoon periods

[Trojan]

It was memory for us here.. moved facility and ePortal from a VM with 4gb ram to one with 16gb and it made a world of difference.

 

15.1 is full of bugs though, but that's another story.

[chaft]

15.1 is a whole other thread . Hopefully i won't get assigned those problems. I'm hiding under a rug from that atm. Monitoring registers is my top priority at the mo Edited May 20, 2015 by chaft
spelling