CCS Service Interruption: VPN connection issues
Incident Report for University of Guelph IT System
Resolved
Services have been stable for a week. While we are still working with the vendor on analysis of root cause, we have put fail-over mechanisms in place. They should ensure the availability of the service, even if the same problem is encountered again until we have a permanent fix in place.
Posted Mar 30, 2020 - 22:01 EDT
Update
In order to stabilize services we had configured two methods of leasing DHCP addresses to a remote computer when logging in to VPN. We are currently experiencing an issue with the primary method and are investigating.

VPN connections will continue to work, but the login process can take longer than usual (up to 2 minutes in the worst case). This will only affect how long it takes to login, once authenticated clients can work normally.

We have re-engaged the vendor in order to further investigate and apologize for any inconvenience this may cause.
Posted Mar 24, 2020 - 12:13 EDT
Update
We have seen a good and solid increase of VPN connections throughout the morning and confirmed that these connections use the newly allocated address pool.

While there were still a couple of individual reports of having trouble connecting to the service, they have been isolated incidents.

As a gentle reminder: A VPN connection is valid for 48 hours. After this time, you will be disconnected and asked to log in again.

We will continue to monitor services throughout today, but evidence so far indicates that the capacity related VPN issues from yesterday have been resolved.

If you are encountering an issue, please contact the helpdesk and provide your username and the time you encountered the problem in the ticket.
Posted Mar 18, 2020 - 11:31 EDT
Update
We have been working with vendor support throughout the evening/morning. The error messages we have seen earlier have stopped. We should be able to leverage the additional range now. Due to low utilization at night, we will confirm in the morning when VPN numbers are expected to climb. The support case with the vendor remains open and we will re-engage immediately, if we see any additional problems.

Should you encounter problems when trying to connect to VPN, please contact the CCS Help Centre at IThelp@uoguelph.ca.
Posted Mar 18, 2020 - 02:48 EDT
Update
We have completed the maintenance to add additional address space for VPN services. Unfortunately we have encountered some issues that still prevent us from leveraging the added resources. This has been escalated with the vendor and the team is actively working with support engineers find a fix.

We appreciate everybody's patience and support and will provide updates as they become available.
Posted Mar 17, 2020 - 21:08 EDT
Update
The previously mentioned increase of capacity seems to be working now. Users should be able to connect to VPN. We are planning to add more capacity at the end of the business day to further support our clients working remotely. CCS will continue to monitor the service to ensure it remains stable after the additional capacity is added tonight.
Posted Mar 17, 2020 - 14:23 EDT
Update
We are still receiving complaints that some clients are unable to connect via VPN and continue to investigate. Our focus is to make the service stable as soon as we can and will provide additional updates as they become available.
Posted Mar 17, 2020 - 11:21 EDT
Monitoring
We have added additional resources to the VPN service which should resolve the issue. Clients that were previously unable to connect should try to connect again. We will continue to monitor the service to see if further adjustments are required.

We apologize for any inconvenience this has caused.
Posted Mar 17, 2020 - 10:40 EDT
Investigating
We have received reports from clients stating they are unable to connect to VPN or have been kicked out. We are actively looking into this issue with the highest priority and hope to have the problem resolved quickly.

Will provide updates on here.

Should you require assistance, please contact the CCS Help Centre at 519-824-4120 x58888 or IThelp@uoguelph.ca. If after hours, please leave a message and your call will be returned on the next business day.

We will provide updates on this page until resolution of the issue. Next update will be at approximately 11 am.

We appreciate your patience and support as we work to resolve this issue.
Posted Mar 17, 2020 - 10:16 EDT