tag:status.editionguard.com,2005:/historyEditionGuard Status - Incident History2024-03-28T22:23:59-07:00EditionGuardtag:status.editionguard.com,2005:Incident/30896772019-10-10T06:30:00-07:002019-10-10T11:02:13-07:00Adobe DRM Server Cluster Outage<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>06:30</var> PDT</small><br><strong>Resolved</strong> - Our Adobe DRM server cluster had degraded performance and finally an outage due to a massive number of large files being delivered for encryption, resulting in out of memory errors on all servers.
<br />
<br />To address this issue, we have come up with short term and long term plans.
<br />
<br />In the short term, we're increasing the number of server instances we have in commission, as well as their memory amount and cpu power. This will mean substantially increased redundancy and failover, which will in turn minimize the risk of similar outages in the future.
<br />
<br />In the long term, we will begin processing files in an asynchronous manner instead of doing so instantly, which will allow us to manage server resources better. More information on this will be forthcoming in the coming weeks as we come up with the development and deployment roadmap.</p>tag:status.editionguard.com,2005:Incident/28584552019-08-13T14:23:37-07:002019-08-13T14:23:47-07:00Adobe DRM Server Cluster Unresponsive<p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>14:23</var> PDT</small><br><strong>Resolved</strong> - Our DRM server cluster is back up. An in-depth investigation is now underway to determine the root cause of the fault.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>14:07</var> PDT</small><br><strong>Investigating</strong> - Server cluster became unresponsive at 14:07 PM PST. We are working to resolve asap.</p>tag:status.editionguard.com,2005:Incident/26926222019-07-11T12:45:16-07:002019-07-11T12:45:16-07:00Adobe DRM Cluster Fulfillment Service Outage<p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>12:45</var> PDT</small><br><strong>Resolved</strong> - There was a temporary network issue on our Adobe DRM Server cluster, where the Adobe activation servers weren't being resolved correctly. The issue has now been resolved. However, a few end-users may have experienced fulfillment errors within the 30 minutes preceding this notice. The fulfillment services are now working properly and retrying previously failed fulfillments should clear any remaining issues.</p>tag:status.editionguard.com,2005:Incident/23528782019-05-03T11:30:49-07:002019-05-03T11:30:49-07:00Adobe eBook Activation/Licensing Services - Planned Production Server<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>11:30</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>11:15</var> PDT</small><br><strong>Scheduled</strong> - From Adobe: Adobe IT team will be applying various Operating system, Network and Database related patches/updates to the Adobe production servers in the coming days.<br /><br />Database related patches/updates will be applied on Adobe eBook Activation and Licensing Servers.<br /><br />Adobe Licensing Service(nasigningservice.adobe.com) will be redirected to other available nodes during patching and we don’t expect any outages due to this.<br /><br />Adobe Activation Service(adeactivate.adobe.com) will have an outage and service unavailability for 2 hours during this activity. Due to this new device activations will not work. Users will be able use their existing activated device and read their existing purchased books during this time.<br /><br />EditionGuard Devops Team Note: The 2 hour downtime in Adobe's activation services may lead to customer inquiries regarding new device activations and adobe id creations. We recommend telling them it's a temporary situation that will be resolved in a matter of hours.</p>tag:status.editionguard.com,2005:Incident/21704002019-02-06T19:26:57-08:002019-02-06T19:26:57-08:00Web Application Database Maintenance<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>19:26</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>19:26</var> PST</small><br><strong>Update</strong> - We have now completed the upgrade process and all systems are back up. Thank you for your patience.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>19:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>16:05</var> PST</small><br><strong>Scheduled</strong> - We will be performing maintenance on our web application database servers to upgrade their capacity in anticipation for gradual increase of load going forward. We anticipate a maximum of 30 minutes downtime for several components during this maintenance.</p>tag:status.editionguard.com,2005:Incident/21801822019-01-30T09:20:53-08:002019-01-30T09:20:54-08:00Dashboard Page Outage<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>09:20</var> PST</small><br><strong>Resolved</strong> - Our app dashboard was unavailable for 3 minutes due to a failure in one of the third party APIs we utilize. It is now back up and we've taken measures to ensure our dashboard stays up regardless of failures in this API. No other components were affected.</p>tag:status.editionguard.com,2005:Incident/20176172018-11-05T15:45:39-08:002018-11-05T15:45:40-08:00Branding Update (Zero Downtime)<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>15:45</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>15:15</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>15:05</var> PST</small><br><strong>Scheduled</strong> - We'll be deploying our newly branded marketing website and dashboard front-ends. These will not affect any of our services, as the changes are strictly cosmetic. If you experience any difficulties using your account pages, please empty your browser caches and reload/refresh to resolve the issues.</p>tag:status.editionguard.com,2005:Incident/19479582018-10-01T21:59:09-07:002018-10-03T19:26:31-07:00Adobe Activation Server Maintenance<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>21:59</var> PDT</small><br><strong>Completed</strong> - Adobe completed maintenance on their activation servers which caused a 30 minute downtime for end-user license fulfillments.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>21:30</var> PDT</small><br><strong>Update</strong> - Adobe's maintenance of their activation servers have begun.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>21:30</var> PDT</small><br><strong>Scheduled</strong> - Adobe will be performing maintenance on their activation servers which will cause a 30 minute downtime for end-user license fulfillments.</p>tag:status.editionguard.com,2005:Incident/19479702018-09-04T12:36:13-07:002018-10-03T19:10:46-07:00Adobe DRM Server Cluster Outage<p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>12:36</var> PDT</small><br><strong>Resolved</strong> - Our Adobe DRM Server Cluster experienced a down-time of 28 minutes after a rolling software deployment process caused erroneous configuration code to be activated. The whole cluster was rolled back to the previous version of the software, restoring the service fully. End-user fulfillments were delayed during this outage and resumed normally upon resolution.</p><p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>12:08</var> PDT</small><br><strong>Investigating</strong> - Adobe DRM web services are currently down after a rolling update. We're now investigating.</p>