The Google Cloud Service Health (CSH) Dashboard provides status information ofthe Google Cloud services organized by region and global locale.
Major incident
The impact of a major incident extends to two or more of the following scenarios:
- Multiple services
- Multiple regions
- Multiple hours
- Multiple customers
In the rare instance a major incident occurs, we act with urgency to resolve any issues.
During a major incident, the status of the issue is communicated through theGoogle Workspace Status Dashboardand theGoogle Cloud Service Health Dashboard.A major incident is marked as cancelService outage on the status dashboards. After the issue is resolved, wepublish a public incident report that includes the details of the factors thatcontributed to the incident and the steps we plan to take to prevent suchincidents from reoccurring.
In the case of smaller-scoped incidents, a nonpublic report might be madeavailable to customers.
Lifecycle of an incident
When a product degradation is detected, the Cloud Customer Care team andproduct engineering team work together to resolve the incident and provide youwith updates.
The following diagram shows the responsibilities of the product engineering andsupport teams:
You can read more about each of these responsibilities in the followingsections.
Detection
Google Cloud uses internal and black box monitoring to detect incidents. For moreinformation, seeChapter 6 of the Site Reliability Engineering book.
If you have Premium, Enhanced, or Standard Support, you can reportan incident by creating a support case in theGoogle Cloud console. Otherwise, you canuse this form.
Initial response
When an incident is detected, the Customer Care team leadscommunication with you. Initial notification of an incident is often sparse,frequently only mentioning the product in question. This is because weprioritize fast notification over detail. Detail can be provided in subsequentupdates.
To provide you as much information as possible without overwhelming youwith issues that do not affect you, different communication channels are useddepending on the scope and severity of an issue:
The Google CSH Dashboard isthe first place to check when you discover an issue is affecting you. Thedashboard shows incidents that affect many customers, so if you see an incidentlisted it is likely related to your problem. To indicate severity, the dashboardmarks incidents as either a disruption or outage. More minor, but stillwidespread issues are posted as temporary notices.
When a relevant Google Cloud product or service reports an issue in thedashboard, you might also see an outage notice in theGoogle Cloud console. If an outage notice appears in the Google Cloud console,you can click the notice to learn more about the status of the issue.
Some Google Cloud products have Google Groups that you can subscribeto in order to receive announcements and notifications about new incidentson the dashboard.
The known issues displayed in the Google Cloud Support Centerand in the Google Cloud console Supportpage are the mostcomprehensive view of issues, and includes issues that affect fewer people thanare shown on the dashboard. If you suspect a Google Cloud issue but do not see anythingon the dashboard, then you should check here.
Support cases are appropriate for issues that do not qualify as incidents orwhere a one-to-one human touch is needed. The known issues page lets youcreate a case from a posted incident so that you get regular updates and cantalk to support staff.
Investigate
Product engineering teams are responsible for investigating the root cause ofincidents. Incident management is often done by Site Reliability Engineers butmight be done by software engineers or others, depending on the situation andproduct. For more information, seeChapter 12 of the Site Reliability Engineering Book.
Mitigation/Fix
An issue is considered fixed only when changes have been made that Google isconfident will end the impact indefinitely. For example, the fix could be rollingback a change that triggered an incident.
While an incident is in progress, Customer Care and the product teamattempt to mitigate the issue. Mitigation is when the impact or scope of anissue can be reduced, for example, by temporarily providing additional resourcesto a service suffering overload.
If no mitigation has been found, when possible, the Customer Care teamfinds and communicates workarounds. Workarounds are steps that you can take to solvethe underlying need despite the incident. A workaround might be to use differentsettings for an API call to avoid a problematic code path.
Follow up
While an incident is ongoing, the Customer Care team provides regularupdates. Updates typically provide:
More information about the incident, such as error messages, zones orregions affected, which features are affected, or percentages of impact.
Progress towards mitigation, including any workarounds.
Timelines for communication, tailored to the incident.
Changes in status, such as when an incident is fixed.
Postmortem
All incidents have a postmortem internally to fully understand the incident andidentify reliability improvements that Google can make. These improvements arethentracked and implemented. For more information on postmortems at Google, seeChapter 15 of the Site Reliability Engineering Book.
Incident report
When incidents have very wide and serious impact, Google provides incidentreports that outline the symptoms, impact, root cause, remediation, and futureprevention of incidents. As with postmortems, we payparticular attention to the steps that we take to learn from the issueand improve reliability. Google's goal in writing and releasing postmortems isto be transparent and demonstrate our commitment to building stable servicesfor our customers.
FAQ
What type of status information can I find on the Google CSH Dashboard?
The Google CSH Dashboard provides status information on services thatare part of Google Cloud. Status can include service disruptions,outages, or informational messages about a temporary issue.
Where can I find information about past service disruptions and outages?
The Google CSH Dashboard keeps a record of disruptions and outages for theGoogle Cloud services for up to five years. TheOverview tab of thedashboard shows the current status of the services by locale. To view informationabout service disruptions and outages in the last year, clickView history on the dashboard.To view a service's outage history for the last five years, click See morefor that service.
How can I view regionalized status information for Google Cloud services?
The Google CSH Dashboard displays the status of all Google Cloud servicesorganized by region and global locale. To view service status for a multi-region,click on the region-specific tab.
Can I build integrations to consume the data displayed on the Google CSH Dashboard programmatically?
Yes, you can consume the data displayed on the Google CSH Dashboard in thefollowing ways:
- Through an RSS feed
Through a JSON History file
You can download the schema for JSON file here.
The RSS feed and JSON History file provide incident status information which canbe consumed through integrations.
What if I have pre-built integrations based on the Google Cloud Status Dashboard prior to the introduction of regionalized status reporting and name change to Google Cloud Service Health Dashboard?
In both the RSS feed and the JSON file, the regional status information isadditive to the information that was already being published prior to theintroduction of regionalized status reporting and change in the name ofGoogle Cloud Status Dashboard. Therefore, we expect your existingintegrations to continue working. However, if you want to consume the regional statusinformation through your integrations, then you need to modify them.
Here's a detailed description of how regional information is presented in bothRSS feed and JSON file:
RSS feed
The regional status information is a new addition to the feed information thatwas provided prior to the introduction of regionalized status. Any locations that arereported as affected are appended to the RSS message.
JSON file
Prior to the regional status update, Google Cloud published a stream ofincidents where each incident contained a list of affected products and a listof status updates for each, if any. These status updates contained anunstructured string field that did or did not contain the locationinformation.
Now, Google Cloud publishes a stream of incidents just as it did before.However, for every incident, each status update contains the following newfields:
updates.affected_locations: contains a structured list of affectedlocations at the time the update was posted. Every update record and themost_recent_updaterecord contain this field.currently_affected_locations: contains the most recent information on thelocations that are actively impacted by the incident. Unlikeupdates.affected_locations, this list becomes empty after the incident isresolved (that is, whenendis set to a non-empty value).previously_affected_locations: contains a list of locations that werepreviously impacted during an incident, but aren't currently. As theincident progresses, some locations might have an outage resolution. Theselocations will still exist in thepreviously_affected_locations field.Once the incident is resolved (that is, whenendis set to a non-empty value),this field contains a list of all locations that were impacted during thisincident.
What if I am experiencing an issue, but it is not listed on the dashboard?
The issue may be isolated to your projects or instances, or it may be impactinga limited number of customers. You can contactCustomer Care about any issuesyou are experiencing that are not listed on the dashboard.
If you are using Google Cloud console, you can click the Send feedback tool inthe upper right corner to report problems.
Who updates the dashboard?
The global Customer Care team monitors the status of servicesusing many different types of signals and updates the dashboard in the event ofa widespread issue. If needed, they will post a detailed incident analysisreport after an incident has been resolved.
What is the difference between an incident and an outage?
Although these terms are often used interchangeably, Google CSH Dashboard andour external communications uses incident to refer to any period of degradedservice and outage to refer only to the most serious, where a product isnonfunctioning to a large extent.
