Application fails for all users on All flows/Services

Check ServiceManager (SM) health

• Login to SM. If fails, try to restart ServiceManager
• Click on Services section and Application section, and verify that all services/applications are there. If not, restore from last backup.
• Go to SM Admin section, License Properties - check License Expiration, and Licensed Server.
• Go to SM Admin section, click on Server Information (at the bottom):

1. Check "Used/Total Memory". If it's too high, restart ServiceManager.

2. Check "Used/Total Disk Space". If it's too low, allocate disk space on this machine

• If your environment is configured with Load Balancer, check all SM instances and verify your Load Balancer it working as expected.
• Perform requests on the application, and validate that you see the requests that comes from the application in the SM Log. If you do not see the requests, so the request are probably blocked prior to reaching AuraPlayer server.

Check Forms/EBS system health and connectivity

• Open a browser, login to your Forms/EBS system and perform a basic flow.

if fails, contact your system admin.

• Enter the ServiceManager machine (RDP or SSH), and perform 'GET' to the Forms/EBS system from the SM machine (to verify connectivity) - either by browser or by wget/curl

If fails, contact your system admin.

• Login to ServiceManager from your browser, Run one of the basic services (for example, login service)

If fails, check the SM logs

• Open the application from your browser, and see if loads the login page and if you can login to the EBS system.

If fails, check SM logs

Application fails for specific flow - investigate at run time

Verify the flow on the underlaying system (Form/EBS)

Open your Forms/EBS application and do the exact flow, using the same credentials and the same inputs. Verify that it works as expected (no error messages and no changes from the original flow).

If the flow or the inputs are complicated or not clear, you can do the following:

• Open the browser DevTools/network tab and see the services that are invoked for the application UI flow.
• Open SM log and see for each service what input values were sent for that service.
• If you are not sure regarding the flow that a service is playing in the Forms/EBS system, open the SM Services section. Find the service and look at the service “Scenario View” section.

If the flow on EBS also has issue, contact your system administrator. else:

Look for a service that failed in the application

• In the browser DevTools/Network find the exact service that fails and look for the returned error message.
  -If the error message not clear or not detailed enough to understand the issue you may need to search in the SM logץ 
  -If HTML service failed, a 'screen snapshot' would be created. to get it go to Admin->Server Files->temp directory, and find that snapshot (HtmlDump_<service name>)
• Verify in the service definition (or in the log) that the service was invoked for the correct Forms/EBS URL.
• Invoke the service from the SM service tester directly (only possible for full services). If it works from the tester look for the issue in the application UI.
• Debug the application using Chrome DevTools. Break before calling the service and validate that all input values are correct.
  * If input parameters are not sent correctly try to clean the browser cache (or run in incognito mode), meta data may not be sent correctly due to version update.

Get a snapshot from underlaying application just before the failure (HTML Services)

If the flow works correctly on the underlaying system and the returned status or the logs are not detailed enough to reveal the problem, try to get a snapshot of the underlaying system (EBS) just before the failure. Many times the snapshots will reveal an unexpected situation (error message, unexpected page, etc.)

To get a snapshot do the following:

• on a laptop/desktop open a chrome browser.
• Open the application
• Login with correct credentials and get as close as possible to point right before the failure
• Open Chrome dev tools (press f12)
• On Console line run the following command: Storage.set("screenshots", "true")
• Perform in the UI the action that leads to the failure
• On Console line run the following command: Storage.remove("screenshots")
• Go to SM temp directory (admin/server files) and look for files with a name pattern of “<service_name>_after_command_n”. These files contains snapshots of the underlaying system after each playback command performed by the service.
  If you have more than one SM server you might need to check on all of them
• Download the files to a local directory and open the one with "<service_name>_after_command_1". This would be the first snapshot. On the top right you should find "Next/prev" to move between the snapshots.
  This will show us the actual UI responses SM gets from EBS.

Use the AuraPlayer visualizer Debugger

In some cases, it's not possible to use the Chrome dev tools (e.g. mobile devices). Instead, it's possible to use our dedicated Debugger tool. 

In order to activate the Debugger -

1. Open the sidebar menu by clicking on the hamburger menu icon

2. Tap or click three times in the area at the bottom (under the links to the various pages)

3. The Debugger buttons should be visible now

You can use the Debugger tool to monitor services and generate snapshots.

Monitoring Services - 

1. After the Debugger is activated reproduce the problematic flow

2. Click on "show log"

3. You'll see a list of all of the services that were running after you activated the Debugger

For each service, you can find all of the the arguments that were sent and the response from the server.

You can either download or copy an HTML version of the list (which the user can paste into an email etc.).

Get logs of specific time

Service Manager is using a rotating log mechanism.  When looking aא the log window on "Admin -> View log", you are actually viewing the recent log messages saved on the latest log file. By default, SM preserves logs for about a month (depends on the size of the logs) on multiple files.

To get the log messages for specific time range you can do the following:

• Go to Admin -> Statistics
• On the "Period List" field choose "Custom Dates" option
• On "Statistics Range" field, press the "Edit" button and select the requested time range 
• Press the "Admin Log" button
  This will download a log file with the log messages for this current range (will be collected form all existing log files).

Timeout issues

Timeout means an operation didn't finish within a reasonable time period. Timeouts may be acceptable when happening occasionally due to external network slowness but can also suggest on a real problem when happening too often.

In SM application you may encounter 3 different types of timeouts:

Application side timeout

Application side timeout means that the application sent a request to the AuraPlayer server and waited too much time for the server response. By default the SM application is configured to wait for 60 seconds and then cancel the request.

When this happens the user may get a "Timeout reached, no response" error message or a custom application error of the application saying that the service request has failed. On the network tab in Chrome DevTools you would see the request marked as (canceled) and colored in red.

Application side timeouts should not happen. 60 seconds are more than enough time for getting a response.  If the application gets these timeouts on a frequent basis you must check your SM and EBS/Forms health and connectivity as described in the first section. 

If timeouts happens only for specific service/s - check the flow - if it should take more than 60 seconds by design (for example creating a report) then the application timeout should be extended.

otherwise, it is suggested that these services should be re-designed. 

Connection Timeout

Connection timeout will occur when the ServiceManager fails to establish connection to the EBS/Forms system, getting timeout error. When this happens the application will display an error of  "Connection wait time exceeded".

When this happens it means there are issues with SM connectivity to the EBS/Forms system. Please refer to the first section for checking environment general health.

Service Timeout

Service timeout may occur when a service that interacts with the EBS/Forms system does not get a response after more than the timeouts defined as the system defaults. To see your system defaults for timeouts goto Admin -> System Properties. In the properties section search for "Timeouts" section. If these timeouts are too short for you EBS/Form response times you may change it.

When service get a timeout error, the application will usually display a general error message. When this happens you will probably follow the instruction on the "Application fails for specific flow - investigate at run time" section above and find the timeout error in the SM logs. There are few options to resolve this issue:

• Increase default timeouts on the SM (if makes sense)
• increase the timeout of that specific service
• Check the EBS/Forms system to figure out why the response is that slow
• Re-design the service to use a faster flow

*Note: timeout error may also occur when a service flow fails for some reason (e.g. a field has changed, data caused unexpected change in the flow, etc). In this case the issue must be investigated on R&D level and may cause code changes.

General Analysis

If you want to get a general view over time of your system usage, errors and performance, you may use the Service Manager Statistics page. To see Service Manager statistics go to Admin -> Statistics. 

On the Service Usage pane you will see a list of all your system services with various information on usage, errors and performance. 

If you notice an abnormal behavior of some services (e.g. too slow, too many failures), you may drill to this specific service and get on the bottom pane a detailed graph view of this service behavior over time.

You also have the option to press the "Execution log" button and download a detailed reports of all service executions in the selected time range to get better understanding on the problematic services.