Slow Application Response
Incident Report for MyPass Credential Manager
Postmortem

Background

Over the past two days MyPass Password Manager experienced is first real performance and response challenge since the service was launched. As a result of this we felt it necessary to provide some feedback as to the cause as well as what steps have (and will be taken) to mitigate such occurrences in the future.

MyPass utilizes a proxy gateway on customer sites to in order to facilitate interaction with customer password repositories. These gateways offer customers security and flexibility in the integration and operation of MyPass.

Cause

Over the weekend, one of the large customer tenants on the MyPassplatform experience performance and configuration issues with their gateway. This created a situation whereby web services calls from the MyPassplatform to the customer gateway stayed open for the maximum timeout period. Due to the size of the tenants and the length of the timeout period, this in effect triggered a cascading scenario that meant large amounts of network connections were being consumed. The ultimate effect of this resulted in connection queuing that adversely impacted the response and performance of other tenants on the platform.

At the core of the challenge was the fact that the traffic was valid customer connections, rather that invalid malicious traffic from something like an external DDOS attack. The result was the only traffic throttling (and not blocking) could be used to manage the incident.

Action / Mitigation

As of this afternoon, the issue at the customer site has been re-mediated and the following actions have been taken to mitigate such an occurrence again:

  • Core connection limits on the MyPass platform has been increased

  • Resources on the core gateway infrastructure has been increased

  • Throttling has been imposed on the amount of EEC (Enrollment and Enforcement Client) transactions that will be allowed per minute

  • Connection timeouts have been adjusted

Additionally, we will be adding additional servers to the core gateway pool in the coming weeks to bolster capacity.

As always, the entire team here at Integralis is committed to providing our customers with accessible, secure and convenient password management across any system. As such we will continue to monitor the situation. If there are any questions or concerns regarding this outage, please do not hesitate to contact us at info@mypass.co.za.

Thank you for your patience in this regard as well as you continued support in building the MyPass story.

** - Team Integralis -**

Posted Oct 10, 2017 - 15:33 SAST

Resolved
All client and core platform issues has been resolved.
Posted Oct 10, 2017 - 15:06 SAST
Monitoring
Slow response and timeouts have been resolved. We are continuing to monitor access time. Thank you for your patience. If there are any further issues, please contact us at servicedesk@iitcon.co.za.
Posted Oct 09, 2017 - 11:46 SAST
Investigating
Please note that we are currently experiencing slow or intermittent response on the MyPass Web Client. We are investigating the issue and will revert shortly.
Posted Oct 09, 2017 - 06:43 SAST