tag:status.adeptcloud.net,2005:/historyAdeptcore, Inc. Status - Incident History2024-03-27T21:53:00-05:00Adeptcore, Inc.tag:status.adeptcloud.net,2005:Incident/203331602024-03-22T18:22:47-05:002024-03-22T18:22:47-05:00ELK Datacenter Network Failover<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>18:22</var> CDT</small><br><strong>Monitoring</strong> - We have received and confirmed reports of a network failover event at our ELK datacenter. Approximately 2 minutes and 47 seconds of downtime happened while our firewall interfaces changed primary routes, due to an upstream provider failover. We are monitoring this situation And do not expect any further disruptions.</p>tag:status.adeptcloud.net,2005:Incident/197442992024-01-18T16:11:54-06:002024-01-18T16:11:54-06:00PHX Major Outage<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>16:11</var> CST</small><br><strong>Resolved</strong> - We are marking the incident as resolved at this time. We are waiting for our peering provider at PHX to provide an after-action report. They estimate their investigation into the issue and a complete report is going to take between seven to fifteen workings days. Once this is done, they will release the information over to us. We have, however, requested an escalation to push up this timeline so we can provide our clients with the exact details of what occurred in a timelier manner. A Postmortem report will be published on here as soon as we receive the after-action report.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>20:33</var> CST</small><br><strong>Update</strong> - Monitoring has just alerted us to another potential disruption. This one seems to have lasted approximately 3 minutes and all services are back online. This is part of the restoration process while the routes are propagating out. We can confirm this is related to the first outage with the global routes. Currently users are able to work but monitoring software may trigger some alerts as routes are propagating and establishing. <br /><br />We are fully operational at this time and are awaiting the after action report as we continuously monitor the global routes to PHX.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>18:25</var> CST</small><br><strong>Update</strong> - All monitoring is showing that all services, including Horizon, are currently back online. MSP tools such as RMM, ScreenConnect and other may take a few minutes to reestablish connectivity.<br /><br /> - If you are using an uptime monitor you may see multiple connect/reconnect attempts while the services are actually all still online.<br /> - We have confirmed that all VPN tunnels and connectivity have been restored.<br /><br />We are awaiting on an after incident report to see what caused the bad routing/networking issues in the first place. From our preliminary troubleshooting, the global routing tables lost all routes to PHX IP's. We will post an update within 24 hours specifying when to expect an after incident report from us and any further steps being taken.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>18:07</var> CST</small><br><strong>Monitoring</strong> - UPDATE: Our status alerts are starting to alert that routing is resolved and that services are coming back online. We will post our next update as soon as we verify that all routing is resolved and all services are back online (ETA 15 minutes).</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>18:05</var> CST</small><br><strong>Update</strong> - UPDATE: We are seeing routes propagating and beginning to work right now. Please hold tight as we believe this issue is about to be resolved.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>17:56</var> CST</small><br><strong>Update</strong> - UPDATE: We are currently seeing an issue that is affecting all IPv4 routing at the PHX datacenter with about 156k IPs affected. We are still working with our datacenter NOC as this issue is affecting all peering.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>17:26</var> CST</small><br><strong>Update</strong> - We are seeing that the BGP routes to all PHX networks are currently down. This issue is affecting all IP space and not just Adeptcloud, we are working with the providers to get this resolved. We are seeing entire 66.85.128.0/18 being down at the moment.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>17:09</var> CST</small><br><strong>Identified</strong> - The cause has been identified as an issue with the datacenter WAN connection. We are currently working with the datacenter engineers to resolve this issue as quickly as possible. We will keep you updated with more information as it becomes available.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>16:57</var> CST</small><br><strong>Investigating</strong> - We are aware of reports of servers appearing offline at our PHX datacenter. Our engineers are currently investigating this issue to get it resolved as quickly as possible. We will update this post as more information becomes available.</p>tag:status.adeptcloud.net,2005:Incident/174918992023-06-07T17:32:17-05:002023-06-07T17:39:04-05:00PHX Datacenter I/O latency<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>17:32</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>16:31</var> CDT</small><br><strong>Monitoring</strong> - we have received confirmation that the latency issue has been resolved, we are leaving this in monitoring status for 24 hours, we have requested a root cause analysis from HPE and are awaiting confirmation of all details related to this event, we will post another update within 24 hours for our plan of action</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>16:00</var> CDT</small><br><strong>Update</strong> - all datastores on nimble storage 2 have returned to pre-issue latency numbers we are awaiting confirmation of root cause as well as confirmation that the issue will not reoccur</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>15:41</var> CDT</small><br><strong>Update</strong> - We are currently detecting another spike and are still working with datacenter engineers and vendor (HPE) on a resolution.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>15:27</var> CDT</small><br><strong>Update</strong> - we are seeing latency has been stable since the last spike, DC Engineering and Vendor (HPE) are working on resolving the issue with Nimble San 2, majority of VMs should be fine, we are noticing if a VM froze up due to a storage latency spike that CPU utilization goes up while it is trying to catch up on the missed I/O</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>15:03</var> CDT</small><br><strong>Update</strong> - datacenter engineering has escalated the issue to the vendor "HPE" engineering team and both are actively working on resolving the issue.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>14:57</var> CDT</small><br><strong>Update</strong> - We are seeing latency spikes continue to drop and back to normal utilization as of right now. It seems the latency spikes have been coming in frequencies of 1 hour. We are continuing to work with the datacenter engineers to resolve this issue and our indicators are pointing to a read cache issue leading to read latency on NIMBLE-SAN 2 which is the cause for the issues.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>14:30</var> CDT</small><br><strong>Update</strong> - 1/5 datastores on the second nimble array are still continuing to show higher than normal spikes, we are working with storage engineers at our data center to determine root cause and resolve the issue</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>12:53</var> CDT</small><br><strong>Update</strong> - latency is once again subsiding in the last 15 min it has stabilized to normal levels again, and VMs might take a couple of moments to catch up. We are running migration to a different nimble array but doing it carefully so as to not add to the latency problem</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>12:28</var> CDT</small><br><strong>Identified</strong> - We are seeing increased storage latency at a portion of our PHX datacenter. The latency has subsided and we are migrating VM's which is taking longer than usual due to the latency. VM's may experience higher than normal queue depths which will present as slow or degraded performance in most cases. We will update you as we get more information</p>tag:status.adeptcloud.net,2005:Incident/174901302023-06-07T13:11:17-05:002023-10-13T13:11:43-05:00Elk Datacenter Service Degredation<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>13:11</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>11:04</var> CDT</small><br><strong>Monitoring</strong> - all servers appear to be back online and the issue appears to be resolved. If you exprience any continued issues, let us know by submitting a ticket to support@adeptcloud.net or calling us at 847-595-0343.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>09:50</var> CDT</small><br><strong>Identified</strong> - The issue has been resolved and all affected servers should be back online momentarily.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>09:15</var> CDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>09:14</var> CDT</small><br><strong>Investigating</strong> - We are currently getting reports of service degredation for some servers at our elk datacenter. We are investigating this to find the cause and resolution as quickly as possible. We will update you as we get more information.</p>tag:status.adeptcloud.net,2005:Incident/173771042023-06-06T09:15:01-05:002023-06-06T09:15:01-05:00Elk Datacenter Outage<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>09:15</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>08:13</var> CDT</small><br><strong>Monitoring</strong> - All servers appear to be back online at this point and time. Please try to log in or have your clients try to log in and confirm there are no residual issues. If you experience issues, please call us at 847-595-0343 or email us at support.adeptcloud.net. We will continue to monitor and post a post-mortem within the next week detailing what happened and plans for the future.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>07:50</var> CDT</small><br><strong>Identified</strong> - We have identified the issue as an outage for our active directory cluster of hosts and are working on bringing them back up currently.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>06:56</var> CDT</small><br><strong>Investigating</strong> - We are investigating reports of outages at our ELK datacenter. We are working with engineers to identify and resolve this issue as quickly as possible. We will update you once we have more information.</p>tag:status.adeptcloud.net,2005:Incident/172830162023-05-19T11:20:03-05:002023-05-19T11:20:03-05:00PHX Packet Loss<p><small>May <var data-var='date'>19</var>, <var data-var='time'>11:20</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>15:34</var> CDT</small><br><strong>Monitoring</strong> - Packet loss across the affected providers has subsided, and no further clients have been affected. We will continue to monitor the network and provide an update if the situation changes.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>08:09</var> CDT</small><br><strong>Update</strong> - we are getting reports that some of the network issues have been resolved, however we do have a few continued reports of network issues. These issues seem to be affecting about 2% of our total traffic, If you are experiancing these issues, please contact support by submitting an email to support@adeptcloud.net or calling us at 847-595-0343</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>15:59</var> CDT</small><br><strong>Identified</strong> - We have received some reports of packet loss and degraded connectivity to our PHX datacenter specifically from clients utilizing charter networks. We are actively working on this issue with networking engineers and will update you as we get more information.</p>tag:status.adeptcloud.net,2005:Incident/166209322023-05-17T15:54:23-05:002023-05-17T15:54:23-05:00Elk Datacenter Issues<p><small>May <var data-var='date'>17</var>, <var data-var='time'>15:54</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>09:49</var> CDT</small><br><strong>Monitoring</strong> - The storage issue was resolved around 30 minutes ago and all virtual machines appear to be functioning correctly. We are continuing to monitor the infrastructure and are working with the storage engineers to determine the exact cause and to prevent this issue from reoccurring in the future.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>08:57</var> CDT</small><br><strong>Identified</strong> - We have received additional reports regarding virtual machines showing offline at our ELK datacenter. The storage engineers have confirmed the issue is related to storage and are working on resolving it now.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>13:12</var> CDT</small><br><strong>Monitoring</strong> - It appears this issue has been resolved once again. All servers at this time appear to be back online and functional. If you have any continued issues, please email us at support@adeptcloud.net and we can take a look. We are still investigating the root cause and will update this status once we determine the cause of the issues.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>12:38</var> CDT</small><br><strong>Update</strong> - It appears that the storage is continuing to experience issues. Storage engineers are currently looking at the cluster and this issue is actively being worked on.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>12:25</var> CDT</small><br><strong>Identified</strong> - We have identified the issue to be storage related and have resolved the issue. We are working on bringing all effected VM's back online currently.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>11:58</var> CDT</small><br><strong>Investigating</strong> - We are aware of reports about some VM's at our ELK data center and are looking into the issue. We will update you as we get more information.</p>tag:status.adeptcloud.net,2005:Incident/169759672023-05-17T15:54:11-05:002023-05-17T15:54:11-05:00PHX Datacenter outage<p><small>May <var data-var='date'>17</var>, <var data-var='time'>15:54</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>19:02</var> CDT</small><br><strong>Update</strong> - We have finished booting up the last of the servers. At this time, all servers should be online or coming online within the next couple of minutes. If you continue to see your servers offline or have troubles logging in, please submit an email to emergency@adeptcloud.net for issues that require immediate action or submit a ticket to support@adeptcloud.net for issues that can be resolved during normal business hours.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>18:00</var> CDT</small><br><strong>Monitoring</strong> - We are seeing networking being restored and are working on ensuring everything is brought back up. We will post another update once everything appears to be back up and running.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>17:43</var> CDT</small><br><strong>Identified</strong> - The PHX Datacenter engineers have informed us that the issue is a networking issue affecting their SAN network. We are actively engaged with datacenter Engineers to resolve this issue. We will update this as more information and updates become available.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>17:27</var> CDT</small><br><strong>Investigating</strong> - We are aware of reports of VM's not responding and showing offline at our PHX Datacenter. We are investigating this and will let you know once we have more information</p>tag:status.adeptcloud.net,2005:Incident/118473022023-03-23T12:07:55-05:002023-03-23T12:07:55-05:00PHX Datacenter Outage<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>12:07</var> CDT</small><br><strong>Resolved</strong> - This has been resolved with no incident for several months after an overhaul of our power infrastructure at the datacenter.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>12:47</var> CDT</small><br><strong>Update</strong> - Updates from datacenter visit that occurred on 10/20/2022 and 10/21/2022.<br /><br />Below is an update for everyone from Denis' trip to the datacenter on October 20th and 21st. <br /><br />- ISCSI connectivity was confirmed to be happening across two NICs on each host and across two separate PCI network cards.<br />- ISCSI connectivity was tested and confirmed with the Datacenter team. We tested each switch and network link across the entire ISCSI path from our network racks back to the datacenter provided Nimble storage arrays. We have re-routed our ISCSI connections with the datacenter to occur across two separate network paths. Our drops come into the Datacenter storage cage from both the East and the West sides now. <br />- We performed in-rack testing of ISCSI network redundancy across the switching fabric. <br />- ISCSI is configured in MPIO, should any portions of the paths go down, the traffic is rerouted via other paths.<br />- The power has been upgraded to 208v 30AMP circuits for both the primary and secondary circuits.<br />- New PDU's were installed into our network racks.<br />- We've tested the redundancy of each hosts' power, network connectivity and SAN connectivity. <br />- We've tested the redundancy of each switch, firewall, network connectivity and their power inputs.<br />- We've performed thorough testing of the PDU's in each rack. During this testing, we shut down A-side PDU's in each rack and confirmed a seamless transition to B-side PDU's without any interruptions or outages occurring. This was tested and confirmed three times overnight.<br /> > The same tests were performed on the B-side PDU's in which we confirmed a seamless transition to the A-side PDU's without interruptions or outages occurring.<br /> > Additional testing was performed on each PDU Bank and breaker. <br />- All the power cords inside of our racks were replaced with Z-lock locking power cords. <br />- We have spoken with and established a policy with the datacenter provider for two specific remote hand technicians for the time being. Both of these technicians were involved in some of the testing and received in-person information and walkthrough in regards to our configuration, infrastructure and specific needs. They were amazing in helping us during testing and with getting everything replaced within our established timeframe.<br /><br />Additional updates on future upgrades and changes to our infrastructure will be provided on December 23rd, 2022 in line with our original timeframe.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>15:34</var> CDT</small><br><strong>Update</strong> - Adeptcloud Outage:<br /><br />We experienced a loss of our primary SAN network on 10/11/2022 around 4:10 PM CST. The issue was determined to have occurred as a result of an incorrect datacenter networking configuration that was applied shortly before this occurred. Our primary SAN network connection was restored at 4:55 PM CST. Upon restoration of the datacenter networking configuration, the virtual machine bootup took longer than expected due to insufficient availability in IOPs on the datacenter Nimble storage arrays. As a result of this outage, we will be making the below changes to our infrastructure. <br /><br /><br />- Immediate Action Plan:<br /><br />Denis will be arriving at the PHX DC on Thursday Morning (10/20/2022) to perform the following tasks:<br /><br />Review networking and storage network redundancy from Adeptcloud equipment to upstream datacenter switches and storage arrays.<br />Review networking and power redundancy in Adeptcloud environment between the hosts and other rack devices.<br />Replacement of all power equipment in the network racks with 208V 30AMP circuits; followed by the replacement of all power cables with locking cables.<br />Power will be upgraded for storage upgrades that will occur around December.<br />Redundancy testing will not cause any downtime as hosts will be cleared of all client virtual machines at the time of testing.<br /><br />> Important Note<br />We will be posting another update here on 10/24/2022 in regards to this week's datacenter visit and the items addressed in the immediate action plan.<br /><br />- Short Term Action Plan<br /><br />Following this visit, the following will occur in the next sixty days:<br /><br />Denis will be arriving at the PHX DC again to setup new storage upgrades.<br />An installation of storage servers and a subsequent migration from datacenter Nimble environment which will help us ensure guaranteed IOPS; specifically for boot storms and failover situations.<br />Storage failover testing (to occur while empty and with no customer virtual machines running on storage).<br />New environmental monitors being installed into the network racks for continued optimal operating temperatures and humidity.<br /><br />> Important Note<br />We will be posting another update here on 12/23/2022 in regards to the subsequent datacenter visit and the items addressed in the short term action plan.<br /><br />- Continuous Action Plan<br /><br />Subsequently, the following measures will be implemented in the next 90 days: <br /><br />Adeptcloud staff will schedule visits to the PHX DC on a quarterly basis to test redundancy and failover plans.<br />We will be limiting DC Access to our network rack and equipment to emergency use only by remote hand technicians.<br />All future hardware upgrades and server installations will be performed by Adeptcloud staff or with our staff present during any such installations performed by the datacenter.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>19:43</var> CDT</small><br><strong>Monitoring</strong> - At this time all servers appear to be back online and functionality has been restored. A full after-action report and post-mortem will be published and made available by the end of the week.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>17:35</var> CDT</small><br><strong>Update</strong> - We are currently monitoring the virtual machines coming up. Approximately 30% of virtual machines are back online at this time. Configuration has been restored by NETOPS for SAN Network. We are working on verifying all virtual machines are coming up.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>16:45</var> CDT</small><br><strong>Identified</strong> - We have identified the issue to be related to our storage network. Primary SAN network went down, DC team is now investigating the issue and restoring the network config. We expect this to be completed in the next 15 minutes and for VM's to start coming back online.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>16:13</var> CDT</small><br><strong>Investigating</strong> - We are aware of reports of issues with some servers on the PHX data center and are currently investigating the cause. We will post updates here as we gather more information.</p>tag:status.adeptcloud.net,2005:Incident/157789692023-03-23T12:06:59-05:002023-03-23T12:06:59-05:00PHX Datacenter Host Outage<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>12:06</var> CDT</small><br><strong>Resolved</strong> - host was brought back online and was confirmed to be faulty hardware in the host. This hardware has been replaced and the host has been operating without incident</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>13:10</var> CST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>10:41</var> CST</small><br><strong>Update</strong> - The last of the VM's have been brought online and are being migrated to different hosts to allow for investigation as to the cause of this issue. We will have a report as to the cause in the coming days. If your VM's still show offline, please give them a few minutes before checking again. If you continue to have issues please submit a ticket by emailing support@adeptcloud.net</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>10:12</var> CST</small><br><strong>Monitoring</strong> - We have powered on the host and are currently working on bringing the VM's online. We will update once this process has been completed. If you have submitted a ticket you will also be notified in the ticket once your machines are online.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>10:02</var> CST</small><br><strong>Investigating</strong> - We are currently experiencing issues with a single host at our PHX datacenter that experienced an outage. This host is currently coming back up. We expect the host and all VM's to be back online shortly and will provide an update in 15 minutes.</p>tag:status.adeptcloud.net,2005:Incident/161663132023-03-23T11:59:36-05:002023-03-23T11:59:36-05:00Elk Datacenter Network Issue<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>11:59</var> CDT</small><br><strong>Resolved</strong> - issue was due to a kernel panic on the primary switch. Failover worked as expected to bring services back online as quickly as possible.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>14:33</var> CST</small><br><strong>Monitoring</strong> - At this time we have confirmed that all services are back up and running from the outage. We are continuing to investigate the cause and will update once we have identified and resolved the issue.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>10:53</var> CST</small><br><strong>Identified</strong> - We are aware of network issues regarding one of the firewalls at our Elk Datacenter. It appears that the primary firewall went down and failed over to the secondary firewall as expected. We are already seeing devices come back online with full connectivity restored. We are currently investigating what caused the primary firewall to go offline and will update this as we get more information.</p>tag:status.adeptcloud.net,2005:Incident/127335202022-11-07T08:48:51-06:002022-11-07T08:48:51-06:00PHX Datacenter WAN Issues<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>08:48</var> CST</small><br><strong>Resolved</strong> - Sunday morning around 5 AM CST, our Phoenix data center's Lumen network links experienced a 300Gbps DDOS (Distributed Denial-of-service attack). The affected network was re-routed and the datacenter's DDOS mitigation and scrubbing services began clearing the traffic.<br /><br />Between 5 AM and 11 AM CST, additional DDOS attacks were mitigated at which point a small amount of packet loss was experienced. Packet loss and network stability was 100% since 11 AM and has been throughout the day into Monday morning (today). <br /><br />We are continuing to monitor this situation but have not seen any service disruptions since the aforementioned time frame and will be marking this incident as resolved.<br /><br />An RFO will be provided within the next seven days in regards to this issue.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>06:56</var> CST</small><br><strong>Monitoring</strong> - The datacenter has made peering changes to adjust the broken routes and we are no longer seeing packet loss to our IP space. <br /><br />From the looks of it, the last five minutes have been steady in regards to latency and packet loss. <br /><br />We are continuing to monitor this. If clients are using ScreenConnect or RMM solutions, we are seeing most of those services starting to come back online. <br /><br />We will provide an update if we detect any more issues with the network throughout the day or when we find out exact cause of Level 3 Networks/CenturyLink issues today.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>06:12</var> CST</small><br><strong>Update</strong> - The datacenter networking team is seeing routing related issues when going through Level3. Testing with Windstream has shown no packet loss occurring. As such, routing changes are currently being implemented at the PHX datacenter to eliminate Level3. We will provide an additional update once the routing changes have been confirmed.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>05:56</var> CST</small><br><strong>Identified</strong> - We have identified the issue to be packet loss we are seeing an unusually high number of packets headed to us being dropped. Datacenter NOC sees the same issue and is working on resolution of the network. <br /><br />We have confirmed all customer VM's, services, storage, address space and power are all online but may be experiencing packet loss when trying to reach out IP Space.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>05:47</var> CST</small><br><strong>Investigating</strong> - We are investigating reports of a public WAN interruption. At this time we have confirmed that all equipment is online and we are working to resolve the WAN issues.</p>tag:status.adeptcloud.net,2005:Incident/111534342022-09-16T13:33:42-05:002022-09-16T13:33:42-05:00Incident at Illinois Datacenter<p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>13:33</var> CDT</small><br><strong>Resolved</strong> - At this time, all virtual machines and servers have been online for over two hours. We have not identified any additional on-going issues as a result of the outage. <br /><br />We will continue to monitor the network and servers throughout the day, however, we will be marking this incident as resolved at this time.</p><p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>11:07</var> CDT</small><br><strong>Monitoring</strong> - All of the affected customer virtual machines appear to be back online now. If you run into any issues logging in, please give us a call.<br /><br />We are continuing to monitor the environment at this time.</p><p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>10:44</var> CDT</small><br><strong>Identified</strong> - One of the ESXI hosts at the datacenter experienced a critical issue and rebooted. <br /><br />The affected host is back online now and virtual machines that are located on it are powering back on.<br /><br />We are currently monitoring this process.<br /><br />We will provide another status update in around 20 minutes, however, some virtual machines may start to show back up online for the customers at this datacenter before then.</p><p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>10:28</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an incident at our Illinois datacenter that has partially affected the Adeptcloud environment. <br /><br />We will be updating this blog post in the next 15 minutes with additional information once some additional information has been gathered.</p>tag:status.adeptcloud.net,2005:Incident/105374202022-07-13T15:23:22-05:002022-07-13T15:25:36-05:00Outage at Primary data center<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>15:23</var> CDT</small><br><strong>Resolved</strong> - On July 11th, 2022 Adeptcloud experienced a critical outage at our Phoenix datacenter around 8:00 AM CST. At this time, we have determined the cause to be a faulty secondary PDU inside our cabinet. The irregular power issues caused by the faulty secondary PDU caused the primary PDU to trip and caused a brief power outage. Datacenter technicians confirmed that equipment was powering on when they arrived at the cabinet and we regained access to our out-of-band management network (provided through our Cisco high-availability firewalls) shortly thereafter. After ensuring all management virtual machines were powered on and operating correctly, we slowly began to power on customer environments in a staggered fashion while fielding partner calls and emails regarding the issue. We attempted to do our best to bring customer environments back online as fast as we could without also creating additional sudden power issues or overloading Nimble storage arrays. We are providing a timeline of that day as it relates to the outage below. <br /><br />• 8:01 AM – Alerts and reports received for total connectivity loss at the Phoenix Datacenter.<br />• 8:10 AM – Total connectivity loss confirmed by Adeptcloud staff.<br />• 8:15 AM – Datacenter team ticket and call in regards to the ongoing issue.<br />• 8:25 AM – Datacenter technicians dispatched to the cabinet with issues.<br />• 8:30 AM – Additional details gathered by Adeptcloud staff, signs pointing to power loss issue.<br />• 8:40 AM – Datacenter team provided details that all equipment is powered on but that a secondary PDU breaker was tripped. <br />• 8:55 AM – One of the two Cisco firewalls (in HA mode) online and accessible.<br />• 8:57 AM – Secondary Cisco firewall (in HA mode) online and accessible.<br />• 9:08 AM – Management virtual machines powering on across multiple ESXi hosts.<br />• 9:25 AM – Virtual firewalls powering on across multiple ESXi hosts.<br />• 9:30 AM – Tenant Active Directory servers being powered on in a staggered fashion.<br />• 9:45 AM – Datacenter team continuing to investigate original cause of power loss.<br />• 10:00 AM – Tenant Horizon servers being powered on in a staggered fashion.<br />• 10:30 AM – Linux based virtual machines being powered on.<br />• 10:35 AM – Manual confirmation of network connectivity to tenant horizon servers.<br />• 11:00 AM – Selective powering on of remote desktop servers and custom LOB servers.<br />• 11:20 AM – Nearly 90% of all customer environments back online and functional.<br />• 12:10 PM – All 100% of customer environments back online and functional. <br />• 4:00 PM – Receipt of further alerts regarding PSU switchovers from the servers. <br />• 4:20 PM – Review of datacenter documents, PDU power draws, server power draw statistics.<br />• 7:05 PM – Replacement of B-side PDU by datacenter staff.<br /><br />We are continuing to monitor the power usage across our environment and are working closely with datacenter staff to ensure a future PDU fault does not cause a total power loss scenario. The secondary (B-side) PDU in our cabinet was replaced and we are actively exploring available options to prevent such an issue in the future.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>12:12</var> CDT</small><br><strong>Monitoring</strong> - We've confirmed that all customer servers are online and operational. We are continuing to monitor the infrastructure and are continuing to work with the datacenter team. <br /><br />We will be publishing a post-mortem on this situation within the next 48 hours once we have gathered all relevant details.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>11:21</var> CDT</small><br><strong>Update</strong> - The vast majority of customer servers have been brought back online. Some Remote Desktop and File servers are currently still being brought back online. A final update will be posted once all of the affected servers have finished powering on. <br /><br />The datacenter team is working to determine the reason for the original power loss scenario at this time.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>10:28</var> CDT</small><br><strong>Update</strong> - We are still in the process of bringing customer environments back online. We are bringing them online in a staggered fashion, and as a result, some customers may see some servers show up online before others. We will continue to post updates as we have them.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>09:31</var> CDT</small><br><strong>Update</strong> - Customer environments are currently in the process of coming back online. We will provide another update once all the environments should be operational.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>09:19</var> CDT</small><br><strong>Update</strong> - The environment is currently slowly coming back online and customer environments will follow suit once we have verified all network and infrastructure components are operating correctly.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>08:53</var> CDT</small><br><strong>Update</strong> - Identified issue has been isolated to a power event at PHX DC, back end infrastructure has been restored and we are working on restoring customer services, next update to follow in in 20 minutes.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>08:40</var> CDT</small><br><strong>Identified</strong> - we have identified the issue working with DC OPS and are working on restoring services right now next update to follow in 20 minutes.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>08:19</var> CDT</small><br><strong>Investigating</strong> - We are aware of an ongoing issue affecting customers at our primary data center and are working on a resolution. Our engineers are working to identify, isolate, and resolve this issue as quickly as possible. We will update this page as soon as we have any updates.</p>tag:status.adeptcloud.net,2005:Incident/97701162022-04-12T17:00:15-05:002022-06-01T15:57:36-05:00Adeptcloud ELK-1 DC Network Outage<p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>17:00</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>13:30</var> CDT</small><br><strong>Identified</strong> - Adeptcloud ELK Datacenter experienced a network outage today. The primary firewall crashed and failed over to our secondary. Outage lasted from 1:13PM CST to 1:20PM CST. We are looking into the issue and will update here once the primary firewall is back in working order.</p>tag:status.adeptcloud.net,2005:Incident/90667992022-01-17T09:32:49-06:002022-02-07T09:33:10-06:00Issue with Node04<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>09:32</var> CST</small><br><strong>Resolved</strong> - This incident occurred as a result of a host memory failure. The datacenter team confirmed that memory modules are all properly working. This incident is being marked as resolved at this time.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:20</var> CST</small><br><strong>Monitoring</strong> - All the virtual machines on Node04 have booted back up and we are confirming log-on functionality for all affected clients/tenants on this node. After this is complete, we will be investigating the exact cause of the purple screen of death.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>14:53</var> CST</small><br><strong>Identified</strong> - We have identified an issue with Node04. The ESXi host received a purple screen of death and is in the process of rebooting. Virtual machines will automatically come up once all services have started. We will then investigate the exact cause and share our findings via this status page.</p>tag:status.adeptcloud.net,2005:Incident/90345552022-01-10T09:03:13-06:002022-01-10T09:03:13-06:00Issue with Node04 at Chicago Datacenter<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>09:03</var> CST</small><br><strong>Resolved</strong> - This incident is being marked as resolved. We have not seen any new issues come up during our monitoring after the affected RAM module was replaced. We will continue to monitor the infrastructure.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>15:37</var> CST</small><br><strong>Monitoring</strong> - All services appear to be in working order on the virtual machines that were migrated. We've also just received word that the failed memory module has been replaced by datacenter technicians. We are currently confirming that the affected node is back to full working order and are monitoring the rest of the environment. Once we've confirmed that this is the case, we will be moving some virtual machines back to this node. No downtime will occur as a result of any of these tasks.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>14:53</var> CST</small><br><strong>Identified</strong> - All virtual machines have been migrated to other nodes and powered on. We are currently confirming all necessary services are running on the affected servers. The failed RAM module on the affected node is scheduled to be replaced at 3PM CST by datacenter staff. No downtime will occur as a result of this memory replacement as all virtual machines have been migrated.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>14:46</var> CST</small><br><strong>Update</strong> - We've determined that the cause of the issues on Node04 was a failed stick of RAM which in turn caused a purple screen of death. We have powered the affected node back on and are in the process of migrating virtual machines to other hosts. This will prepare Node04 for the necessary maintenance and memory replacement by datacenter staff. As it stands, 30+ virtual machines have been moved and services for affected clients/tenants are being restored.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>14:10</var> CST</small><br><strong>Investigating</strong> - We're currently investigating an issue with Node04 at our Chicago Datacenter. <br /><br />Some customers may experience issues logging in if their virtual machines are located on this node. <br /><br />More information to follow as we investigate and resolve this issue.</p>tag:status.adeptcloud.net,2005:Incident/84319382021-11-10T08:04:26-06:002021-11-10T08:04:41-06:00Country-wide Comcast Issues<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>08:04</var> CST</small><br><strong>Resolved</strong> - At this time, we have not seen any additional reports of any widespread Comcast/Xfinity issues and will be marking this incident as Resolved.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>09:04</var> CST</small><br><strong>Monitoring</strong> - The issue with Comcast was occurring across the Chicago area as well as a minimum of six other states in the United States. It has been classified as a major outage and affected internet, television and telephone service. At this point, it has been more than 30 minutes since we have seen any sizeable packet loss for our Chicago-area customers (residential and business). As such, we are changing the status of this Incident to 'Monitoring'.<br /><br />We will continue to monitor this situation and provide updates should the issue return. If no further widespread internet issues relating to Comcast occur then this incident will be marked as 'Resolved' at the end of today.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>08:24</var> CST</small><br><strong>Investigating</strong> - We have been made aware of issues with the Comcast/Xfinity network that are occurring country-wide. We have confirmed this issue with some of our customers that utilize Comcast Residential as well as Comcast Business internet services. Comcast experienced issues across several markets around midnight and started experiencing issues again around 7:10 AM CST. <br />At this time, we are not experiencing any issues with our datacenter networks but customers may see issues with connecting, staying connected or experience freezing due to the ongoing problems with their respective ISP. <br /><br />We will be updating this incident as more news comes in regarding the current status of the Xfinity/Comcast outage.</p>tag:status.adeptcloud.net,2005:Incident/72559612021-06-26T08:27:51-05:002021-11-10T08:05:34-06:00Reported Network Connection Issues<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>08:27</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>11:14</var> CDT</small><br><strong>Update</strong> - We'd like to provide everyone with an update regarding the recent DDoS attacks that impacted Adeptcloud services at one of our datacenter locations.<br /><br />- The network has remained stable and there are currently no active attacks.<br />- There was a brief attempt at at a DDOS attack on June 19th around 3:00 AM CST, however, the datacenter provider was able to quickly mitigate this attack and no customers were impacted during this event.<br /><br />No other major DDoS events have occurred since our last update and network traffic continues to remain segmented in order to utilize scrubbing services. This has been effective in thwarting attacks and preventing latency issues.<br /><br />A root cause document (RFO) is forthcoming from the datacenter provider within the coming weeks. Once we receive it, we will be able to share some more technical details regarding these attacks and steps that were taken to mitigate them.<br /><br />We'd like to once again apologize for the inconvenience caused by these attacks. The network remains operational at this time with no impact to our partners or customers.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>10:33</var> CDT</small><br><strong>Update</strong> - DDOS Update:<br /><br />Last night about 7:06PM CST and 7:21PM CST 2 DDOS attacks were mitigated in under 2 minutes. <br /><br />We have received an update that our Datacenter has implement an additional layer of Defense with a scrubbing service from one of the upstream providers. While this does not stop the attack entirely it does make disruption of the network limited as seen with the 2 events last night.<br /><br />Some important information:<br /><br />What happened:<br />This DDOS is targeted at a colocation client located in the Chicago datacenter unrelated to Adeptcloud or our customers. The attack saturated bandwidth on blended routes causing the network disruptions you all have seen over the last few days.<br /><br />Will it happen again?<br />While the possibility of more network disruptions today does exist, the scrubbing service employed should make the impact minimal.<br /><br />When will it end?<br />DDOS is not something that has a time frame unfortunately, the bad actors at some point will give up the battle once the attacks are proven to not cause impact. With that said we wont have an update saying the attack is over as another one could be started at any moment.<br /><br />What is being done about future threats?<br />We do have more calls scheduled today with DC NetOps and Engineering and will provide updates as data comes in. Adeptcloud does have a plan in action to mitigate future impact and distributing workloads, we are working with carriers and datacenters to prevent future impacts and to build defenses against any further disruptions. We are continuously monitoring the network and We will provide updates as they become available.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>11:58</var> CDT</small><br><strong>Update</strong> - We have received official confirmation that Telia has been removed from our blended bandwidth and none of our traffic is currently utilizing Telia as a carrier at this time. All circuits have been increased to 100GB (UDP rate limits) by the provider and the datacenter is in the process of implementing this change on specific ports to help mitigate future attacks. These circuit upgrades are being implemented with providers such as Cogent, Comcast, Telia, CenturyLink and others. Any malicious traffic is still being routed to the Telia carrier at this time. Given that Telia is no longer being utilized by us for any inbound or outbound traffic, we're not expecting to see any issues with latency at this time. The last attack/partial disruption that occurred today was due to the attackers changing their plan of attack and attacking random IP subnets (in a wide-spread manner) in an attempt to disrupt network connectivity. They were partially successful with this attack as multiple customers, including Adeptcloud, experienced temporary loss of network connectivity until the attack was mitigated by the provider.<br /><br />The DDOS attack was occurring via UDP and the estimated size (bandwidth) of the attack varied between 100Gbps to 150Gbps, however, we're still working to determine whether the 100-150Gbps numbers were calculated from malicious traffic as a whole or from individual attacks on random IP's. We're sure more details will be forthcoming once the situation has calmed down and NOC staff at the datacenter has a chance to put together a more concise report.<br /><br />Network connectivity and traffic appears to have stabilized at this time and we are continuing to monitor inbound and outbound connectivity for any future issues.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>11:00</var> CDT</small><br><strong>Update</strong> - Additional attacks have been identified and mitigated this late morning by the datacenter provider. We're still working with the provider to ensure none of our inbound and outbound traffic utilizes Telia at this time while these attacks continue to come in. As mentioned in our earlier update, the malicious traffic is being routed to Telia by the provider so separating our network away from this carrier will help tremendously in ensuring our network is not affected by these incoming attacks. We're continuing to monitor this situation and are working on gathering more technical details to put out regarding the scope of this attack, the average bandwidth associated with it and the type of attack. At the time of this update, the network appears to be stable, however, we cannot guarantee that this issue has been completely resolved at this time. We apologize for any inconvenience caused thus far and are working diligently on resolving this issue.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>09:05</var> CDT</small><br><strong>Update</strong> - Current status of DDoS:<br /><br />Our Datacenter has reported that a client of theirs was and continues to be a target of a DDOS attack. At this time, traffic destined to this customer has been isolated to one carrier (Telia) in order to separate other customer networks and to handle the scrubbing of malicious traffic. This change effectively takes Telia out of our blended bandwidth in order to reduce the impact of this attack on our networks.<br /><br />We are continuing to monitor this situation and are in constant communication with our NetOps and datacenter teams to determine how this can be prevented more effectively in the future. <br /><br />We will provide an update as soon as more information is known.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>12:48</var> CDT</small><br><strong>Update</strong> - We have received notice that another attack was just mitigated by the datacenter. We saw a brief (three minute) period in which latency spiked and some amount of traffic was not getting through until the attack was mitigated. At the time of writing this update, ping times are back to normal and connections seem stable. We're still awaiting confirmations of this issue being completely resolved and are working with the datacenter provider on a more technical update in regards to these issues.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>16:22</var> CDT</small><br><strong>Update</strong> - Our datacenter provider has informed us that they are currently mitigating a large DDoS attack that is impacting multiple customers in the Chicago datacenter. They have mitigated some of these attacks but new ones continue to come in causing some networking related stability issues. The network engineers are still working on mitigating these new attacks that are coming in. Given that this is impacting several customers, cabinets and switches, the attacks must be occurring on some of the core routing/switching infrastructure and are not targeting us or any of our customers specifically. We'll continue to post updates as this situation develops.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>15:26</var> CDT</small><br><strong>Monitoring</strong> - Networking appears to have stabilized at this point and we have not seen networking related issues occur in the past 20 minutes. No network disruptions have been reported. We are still investigating the situation with our Datacenter. In the past hour, networking was stable for 10 minutes and would go down for 2-3 minutes. Currently we have no confirmation that network issues are fully resolved. We will provide an update as soon as possible based on the information that we have.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>14:25</var> CDT</small><br><strong>Identified</strong> - We can confirm current network stability is impacted datacenter networking is looking into the issue.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>13:59</var> CDT</small><br><strong>Investigating</strong> - We are receiving a few reports of network connectivity issues happening in CHI3 Datacenter. We are currently investigating the issue.</p>tag:status.adeptcloud.net,2005:Incident/51249062020-09-19T23:00:18-05:002020-09-25T09:01:30-05:00NODE06 Status Offline<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>23:00</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>16:33</var> CDT</small><br><strong>Update</strong> - All systems appear to be operational at this time and we will continue monitoring performance throughout the weekend. We are still investigating the exact cause of the server lock up.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>15:49</var> CDT</small><br><strong>Update</strong> - All virtual machines are coming back online now. We are confirming accessibility and functionality of the virtual machines that are coming online. We have a preliminary diagnosis of a Kernel Panic that occurred due to one of the CPU's. We are currently monitoring the environment and will be sending out a full synopsis once the environment is confirmed to be fully functional.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>15:35</var> CDT</small><br><strong>Monitoring</strong> - Server is back online now we are slowly bringing up all clients affected by this.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>15:11</var> CDT</small><br><strong>Update</strong> - Affected host is currently being rebooted as ESXI crash occurred, current ETA for restoration is 30 minutes.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>14:44</var> CDT</small><br><strong>Identified</strong> - We have identified, the issue at this time, DCOPS is currently working on the server. We will provide an update in 15 minutes on status and restoration.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>14:30</var> CDT</small><br><strong>Investigating</strong> - We just received alerts of NODE06 going offline. Adeptcloud Engineers and INAP engineers are investigating the issue.</p>tag:status.adeptcloud.net,2005:Incident/42284952020-06-01T08:51:37-05:002020-06-01T08:51:37-05:00SSL Expiration Notice for https://cloud-hosted.com Only.<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>08:51</var> CDT</small><br><strong>Resolved</strong> - SSL certificate was renewed and tested Saturday night at 10PM CST. We have confirmed operation and connectivity for https://cloud-hosted.com tenants.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>15:52</var> CDT</small><br><strong>Monitoring</strong> - https://cloud-hosted.com SSL certificate will be renewed tomorrow on Saturday, May 30th at 10:00PM CST. Services only for clients utilizing the https://cloud-hosted.com portal will be disrupted for about 15-30 minutes while the certificate is replaced and servers rebooted.</p>tag:status.adeptcloud.net,2005:Incident/38058492020-03-30T17:00:07-05:002020-04-13T16:50:42-05:00Network Latency<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>17:00</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>13:23</var> CDT</small><br><strong>Monitoring</strong> - We have identified routing issues and are working with our upstream provider on the issues detected. Currently, all services are back to normal operation. We are continuing to monitor the situation.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>13:10</var> CDT</small><br><strong>Investigating</strong> - We are currently receiving reports of network latency from certain ISP's. We are investigating the root cause of this and will update you as soon as possible.</p>tag:status.adeptcloud.net,2005:Incident/37851562020-03-27T08:35:04-05:002020-03-27T08:35:04-05:00Network routing issues detected<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>08:35</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>10:36</var> CDT</small><br><strong>Monitoring</strong> - We have reviewed the issues from this morning and confirmed the environment is still running steadily without any newly reported issues. Our data center operations have confirmed they save 2 blips this morning related to networking infrastructure. As of this time, no new cases have been reported and troubleshooting has confirmed all equipment and networks are performing as expected. We are monitoring the situation at this time. We will resolve this incident at the end of the day if no other reports come in related to network performance.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>08:24</var> CDT</small><br><strong>Investigating</strong> - We have received reports this morning of some networking related disconnects and routing issues. We are investigating it at the current moment. All services are currently operational. Users might have been disconnected from there sessions and VPNs might have reset between 7AM CST this morning and 8:15AM CST this morning.</p>tag:status.adeptcloud.net,2005:Incident/36218602020-03-16T20:30:51-05:002020-03-26T08:22:29-05:00Network Migration - Impacting Access<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>20:30</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>01:08</var> CDT</small><br><strong>Verifying</strong> - All services are now up and online. Hardware has been upgraded and we resolved an issue with a firewall firmware version. At this time we are aware of a few VPN connectivity issues. Engineering is reviewing them right now and we should have a fix tomorrow morning.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>21:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>13:33</var> CST</small><br><strong>Update</strong> - Maintenance: is scheduled from March, 14th, 2020 at 9PM CST to March 15th, 2020, 12AMCST</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>13:18</var> CST</small><br><strong>Scheduled</strong> - On March 14th, 2020 at 9PM CST Adeptcloud will be making changes to its network infrastructure. We are working in conjunction with our datacenter provider and firewall vendors to migrate firewall platforms. We will be migrating the entire infrastructure from Cisco firewalls to Palo Alto firewalls. In our preparation for this change we have tested and confirmed various details to ensure the migration goes smoothly. <br /><br />We are expecting up to 10 minutes of downtime during this period.<br /><br />As with all maintenance this time period may be extended and include more than 10 minutes of downtime.<br /><br />We will update this status once the maintenance tasks have been finished.<br /><br />Maintenance: is scheduled from March, 14th, 2020 at 9PM CST to March 15th, 2020, 12AMCST</p>tag:status.adeptcloud.net,2005:Incident/30784992019-10-10T15:34:47-05:002019-10-10T15:34:47-05:00Network Incident<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>15:34</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>17:17</var> CDT</small><br><strong>Monitoring</strong> - We are back online and marking the incident into monitoring status. With 2 outages in the day we are working on a permanent resolution at this time.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>16:21</var> CDT</small><br><strong>Identified</strong> - We are aware of a second incident with INAP networking and working on resolving this as soon as possible.</p>