Duration: 18 minutes

Affected Components: Client Dashboard

Mar 12, 13:40:06 GMT+0
Monitoring - Client Dashboard and Management API are currently being upgraded..

Mar 12, 13:57:38 GMT+0
Resolved - Upgrades have been completed..

Duration: 55 minutes

Affected Components: Nearest Roads API, Trace Attributes API, , Elevation API, Isochrone API, Optimized Routing API, Matrix Routing API, Map Matching API, Standard Routing API, Routing →

Mar 9, 02:52:32 GMT+0
Investigating - We are currently investigating an outage with routing and elevation APIs in the Asia/Pacific region..

Mar 9, 03:21:39 GMT+0
Identified - We have identified the issue and rolled out a mitigation to route around the affected datacenter. Performance may be slightly degraded..

Mar 9, 03:47:25 GMT+0
Resolved - This issue has been resolved..

Duration: 1 day, 6 hours and 4 minutes

Affected Components: Static Maps API

Jan 26, 08:30:00 GMT+0
Investigating - We are aware of sporadic, elevated errors and timeouts in the Static Maps API. The root cause is a coordinated DDOS-style attack. We have implemented multiple mitigations, and observed corresponding improvements in the impact, but we have yet to fully contain the attack due to its complexity. We are actively working to contain it, and expect to see improvements within the next hour..

Jan 27, 11:41:13 GMT+0
Monitoring - We have implemented further mitigations, and we are observing complete restoration of services. We will continue monitoring for any continued impact..

Jan 27, 14:33:35 GMT+0
Resolved - The impact has been entirely mitigated. This incident has been resolved..

Duration: 1 hour and 12 minutes

Affected Components: Satellite Map Styles (Raster Tile and Static Map APIs)

Oct 30, 09:52:12 GMT+0
Investigating - We are investigating an emerging issue with imagery basemaps. This is currently impacting less than 10% of requests. We will provide updates as we learn more..

Oct 30, 09:59:58 GMT+0
Identified - We have identified the cause of the issue to be an upstream vendor returning spurious authentication errors. We are working with the vendor to correct the issue..

Oct 30, 10:42:00 GMT+0
Monitoring - The vendor has implemented a fix, and the errors have entirely resolved. We will continue monitoring to ensure errors do not reoccur. If you have any questions or concerns about this incident, please reach out to support@stadiamaps.com, and we will be happy to answer your questions..

Oct 30, 11:03:51 GMT+0
Resolved - Our upstream vendor has informed us the issue is entirely resolved on their side, and no more errors have occurred during our monitoring period. This incident has been resolved..

Duration: 1 hour and 1 minute

Affected Components: Static Maps API

Oct 12, 13:42:53 GMT+0
Investigating - We have identified elevated errors with static maps. We are currently investigating this incident..

Oct 12, 14:10:06 GMT+0
Monitoring - We implemented a emergency fix and are currently monitoring the result. We are continuing to work on identifying any root causes and implementing permanent fixes..

Oct 12, 14:43:29 GMT+0
Resolved - This incident has been resolved. We have identified the cause to be a latent bug in a recent package upgrade. We have rolled this back, and will further investigate in the upcoming week..

Duration: 1 day, 7 hours and 26 minutes

Affected Components: Standard Map Styles, Reverse Geocoding API, Stamen Map Styles, Vector Map Tile API, , Static Maps API, Third Party: Linode → Regions → BR-GRU (Sao Paulo), Third Party: Linode → Regions → US-ORD (Chicago), , Forward Geocoding API, Structured Geocoding API, Place Details API (v2), Third Party: Linode → Regions → US-East (Newark), Third Party: Linode → Regions → US-Central (Dallas), , Third Party: Linode → Regions → US-West (Fremont), Autocomplete Search API (v1), Raster Map Tile API, Satellite Map Styles (Raster Tile and Static Map APIs), Autocomplete Search API (v2), Bulk Geocoding API, Place Lookup API (v1), Maps → Geocoding & Search → Map Styles →

Aug 26, 00:15:00 GMT+0
Identified - We have investigated and identified an on-going networking and performance issue impacting our services in North America. We are not observing any errors or request failures, but we are seeing degraded performance on approximately 5% of requests across our services. We have reported the issue to the responsible service provider, and we will share further updates as they become available..

Aug 26, 12:01:38 GMT+0
Identified - Our infrastructure provider is continuing to work on this issue. We have not yet been provided a resolution timeline, but we are actively investigating alternative solutions to prevent future reoccurrence of this issue. Thank you for your patience. If you have any questions or concerns, please do not hesitate to reach out to support..

Aug 26, 15:29:03 GMT+0
Monitoring - After continue discussion with our cloud provider, we have opted to fail-over most North American traffic to our EU-based datacenters while we continue to wait for an upstream resolution. We expect North American performance to return to close to baseline with some additional latency. The worst case performance should be significantly lower. We will continue monitoring and discussing with our cloud provider..

Aug 27, 00:16:26 GMT+0
Monitoring - Our cloud provider has implemented a fix, and we have moved traffic back to normal regional operations. We do not expect further impact of this issue, but we will continue to monitor to ensure no further impact..

Aug 27, 07:41:01 GMT+0
Resolved - Performance remains as expected. This incident has been resolved. We will continue working with our cloud provider to ensure this does not reoccur..

Duration: 1 day, 18 hours and 42 minutes

Affected Components: Stamen Map Styles, Vector Map Tile API, , Static Maps API, Reverse Geocoding API, , Forward Geocoding API, Standard Map Styles, , Structured Geocoding API, Place Details API (v2), Autocomplete Search API (v1), Raster Map Tile API, Satellite Map Styles (Raster Tile and Static Map APIs), Autocomplete Search API (v2), Bulk Geocoding API, Place Lookup API (v1), Maps → Geocoding & Search → Map Styles →

Aug 14, 11:30:00 GMT+0
Identified - We have investigated and identified an on-going networking and performance issue impacting our services in North America. We are not observing errors or request failures, but are seeing degraded performance on approximately 5% of requests across our services. We have reported the issue to the responsible service provider, and we will share further updates as they become available..

Aug 15, 15:44:42 GMT+0
Identified - Our cloud provider has identified the cause of the latency, escalated it to the appropriate engineers, and they are working on a solution. We will continue to provide updates as we have them..

Aug 15, 16:32:03 GMT+0
Monitoring - Our cloud provider has migrated the relevant infrastructure, and we are seeing full recovery of performance. We will continue monitoring, but we do not expect any further impacts of this event..

Aug 16, 06:12:09 GMT+0
Resolved - After monitoring, performance remains within expectations. This incident has been resolved..

Duration: 23 hours and 56 minutes

Affected Components: Stamen Map Styles, Third Party: Linode → Regions → US-East (Newark), Client Dashboard, Marketing Website

Jul 27, 17:34:36 GMT+0
Identified - Our upstream vendor is continuing to implement a fix after identifying the cause as a power outage and subsequent cooling failure. We continue to monitor and mitigate to the extent possible within our systems. Due to unexpected points of failure, some Stamen Map Tiles and other tiles are affected. Our primary systems remain online and unaffected..

Jul 27, 21:30:25 GMT+0
Identified - Our vendor is continuing working on fixes. Tiles, marketing site, and all other systems other than our client dashboard have been restored..

Jul 27, 10:05:00 GMT+0
Identified - Our client dashboard is currently unavailable due to an upstream outage with our cloud provider. We will provide updates as we receive them..

Jul 27, 12:02:30 GMT+0
Identified - The dashboard remains unavailable due to the upstream issue. We continue to monitor with our upstream provider, and we will provide updates as they are available..

Jul 28, 08:48:09 GMT+0
Monitoring - As of 08:30 UTC, all services have been restored. Our vendor is continuing to implement fixes on their side, but we opted for a full recovery procedure to restore operations. Due to the complexity of the failure, database recovery procedures were required, and approximately 24 hours of usage data were lost. We will continue to monitor the situation, but we do not expect any further impacts for this incident. The team is working on a strategy to ensure this level of impact cannot reoccur in the future..

Jul 28, 10:01:06 GMT+0
Resolved - All services remain operational. This incident has been resolved..

Duration: 7 hours and 29 minutes

Affected Components: Client Dashboard, Marketing Website, Raster Map Tile API, Vector Map Tile API, Static Maps API, Satellite Map Styles (Raster Tile and Static Map APIs), Reverse Geocoding API, Forward Geocoding API, Stamen Map Styles, Standard Map Styles, , Structured Geocoding API, Autocomplete Search API (v1), Autocomplete Search API (v2), Bulk Geocoding API, Place Details API (v2), Place Lookup API (v1), Map Styles →

Apr 16, 20:33:11 GMT+0
Investigating - We've received reports of elevated errors across multiple services. We are currently investigating this incident..

Apr 16, 20:46:37 GMT+0
Identified - We have identified an issue affecting multiple regions of our primary infrastructure provider. They are actively investigating the service issue, and we are actively working on mitigation approaches. This issue is primarily impacting our Geocoding & Search products, but may also impact other services sporadically..

Apr 17, 04:02:13 GMT+0
Resolved - This incident has been resolved..

Apr 16, 21:39:32 GMT+0
Identified - We are continuing to work on a fix for this incident..

Apr 16, 22:14:32 GMT+0
Identified - While we are waiting for an upstream resolution ETA, we are continuing to take every action we can to mitigate the impact of the incident..

Apr 16, 22:49:28 GMT+0
Identified - The upstream incident has been corrected, and our systems are in the process of returning to normal operation. We expect the peak impact of this incident to be resolved within 15 minutes..

Apr 16, 23:01:14 GMT+0
Monitoring - The infrastructure and services have recovered, and we are seeing nominal error rates. There may be a brief period of continued performance impact, which will resolve over the next 15-30 minutes..

Apr 17, 00:36:53 GMT+0
Monitoring - Our upstream infrastructure provider has advised that they are still in monitoring mode, but we have not seen any significant issues in the past 90 minutes..