CCS Service Interruption: FRS ongoing issue

Incident Report for U of G IT System

Resolved

No new issues reported today after the fix went live yesterday.
FRS is up and running with PDF Functionality restored.

Part of our investigation is to prevent similar service outages in the future.

We appreciate your patience and support during this service interruption.
Posted Mar 12, 2020 - 16:48 EDT

Update

We have deployed the fix successfully to FRS.
PDF functionality is now available.

We will keep monitoring tomorrow and keeping this incident open for now.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 11, 2020 - 17:05 EDT

Update

Fix is scheduled for deployment at 4.30PM to bring back PDF Generation functionality.
FRS remains up and running.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 11, 2020 - 15:11 EDT

Update

Client Testing completed successfully.
Deployment of fix to be scheduled.
FRS remains up and running.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 10, 2020 - 17:08 EDT

Update

Client Testing scheduled to begin tomorrow (Tuesday, March 10, 2020).
FRS remains up and running.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 09, 2020 - 16:48 EDT

Monitoring

The fix is now in the Testing phase.
FRS remains up and running.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 06, 2020 - 10:18 EST

Update

We now have a fix for PDF Generation in FRS that will be entering test phase tomorrow.
FRS remains up and running.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 05, 2020 - 17:20 EST

Update

FRS is now available with Feb-19/20 now final. Mar-19/20 transactions are being posted and will show in FRS tomorrow. All PDF related features in FRS have been disabled.

Many scheduled reports including Month End Reports are impacted. FRS will not be able to send any reports via email however all data in FRS can be accessed by running reports immediately and viewing on screen. The options to "Print" or export to "Excel" continue to be available and working properly.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 05, 2020 - 14:28 EST

Identified

We have identified the cause of the issue with PDF Generation; however, services still remain impacted.
We are looking into bringing FRS back online without PDF Generation while we continue working on the issue.

More updates to follow soon.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 05, 2020 - 11:05 EST

Update

We now have a potential root cause that we will continue investigating to confirm.
FRS remains disabled at the moment.

We have also logged a Service Request with Oracle.

We appreciate your patience as we work to resolve this issue.
Posted Mar 04, 2020 - 18:46 EST

Update

Investigation is still ongoing at the moment and we have engaged extra resources.
FRS remains disabled at the moment.

We appreciate your patience as we work to resolve this issue.
Posted Mar 04, 2020 - 16:28 EST

Update

Investigation is ongoing. We will continue to post updates here.
FRS remains disabled at the moment.

We appreciate your patience as we work to resolve this issue.
Posted Mar 04, 2020 - 13:39 EST

Update

We continue to experience issues with FRS and the service has been disabled as of approximately 11:50 AM today, Wednesday, March 4, 2020.

We appreciate your patience as we work to resolve this issue.
Posted Mar 04, 2020 - 12:00 EST

Investigating

We were experiencing an issue with FRS and the service was disabled as of approximately Tuesday Mar 3rd, 2020 at 3.00PM to Wednesday Mar 4th, 2020 at 10.00AM.

FRS is currently up and running. We will post additional service interruptions here.

Should you require assistance during this time, please contact the CCS Help Centre at 519-824-4120 x58888 or IThelp@uoguelph.ca. If after hours, please leave a message and your call will be returned on the next business day.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 04, 2020 - 10:18 EST