Over the past two days MyPass Password Manager experienced is first real performance and response challenge since the service was launched. As a result of this we felt it necessary to provide some feedback as to the cause as well as what steps have (and will be taken) to mitigate such occurrences in the future.
MyPass utilizes a proxy gateway on customer sites to in order to facilitate interaction with customer password repositories. These gateways offer customers security and flexibility in the integration and operation of MyPass.
Over the weekend, one of the large customer tenants on the MyPassplatform experience performance and configuration issues with their gateway. This created a situation whereby web services calls from the MyPassplatform to the customer gateway stayed open for the maximum timeout period. Due to the size of the tenants and the length of the timeout period, this in effect triggered a cascading scenario that meant large amounts of network connections were being consumed. The ultimate effect of this resulted in connection queuing that adversely impacted the response and performance of other tenants on the platform.
At the core of the challenge was the fact that the traffic was valid customer connections, rather that invalid malicious traffic from something like an external DDOS attack. The result was the only traffic throttling (and not blocking) could be used to manage the incident.
As of this afternoon, the issue at the customer site has been re-mediated and the following actions have been taken to mitigate such an occurrence again:
Core connection limits on the MyPass platform has been increased
Resources on the core gateway infrastructure has been increased
Throttling has been imposed on the amount of EEC (Enrollment and Enforcement Client) transactions that will be allowed per minute
Connection timeouts have been adjusted
Additionally, we will be adding additional servers to the core gateway pool in the coming weeks to bolster capacity.
As always, the entire team here at Integralis is committed to providing our customers with accessible, secure and convenient password management across any system. As such we will continue to monitor the situation. If there are any questions or concerns regarding this outage, please do not hesitate to contact us at info@mypass.co.za.
Thank you for your patience in this regard as well as you continued support in building the MyPass story.
** - Team Integralis -**