The SharePoint Health Score was introduced in SharePoint 2010 and plays an even more important role in SharePoint 2013. The Health Score determines the health of a SharePoint server/web application on a scale from 0 to 10, where 0 is the most healthy state. SharePoint automatically starts throttling request once the Health Score is to high. The Health Score is can be calculated using many parameters, such as memory usage, concurrent requests etc. In this post I will give you some details on how the Health Score works, how you can troubleshoot it and how you can use it and how you can configure it.
Note: this article contains some registry changes that never ever should be done on a production server, unless told by Microsoft support…or whatever…
What is the Health Score?
Let’s start with some background facts. Stop whining, I know plenty of peeps have already blogged about it but I think it is good to have all the facts in this same blog post.
The Health Score is as an integer between 0 and 10, where 0 represents a really good health of the server, or rather Web Application, and 10 is where you do not want to be. SharePoint continuously calculates the Health Score and for each and every request the Health Score is sent as an HTTP Response Header, called
X-SharePointHealthScore
.
This Health Score is used by (or at least should be used by) client applications to determine the health of the web application and then use it to make a decision on whether it should stop hammering the web application or not. If the Health Score reaches up to 10, SharePoint itself will start to throttle requests (which we’ll talk about in just a bit).
Worth noticing here is even though this article is written for SharePoint 2013, most of it is valid on SharePoint 2010 as well.
How the SharePoint Health Score is calculated
So, how does SharePoint calculate this Health Score? When a SharePoint Web Application spins up, internally a new thread is created (you can see it in debuggers with the name SPPerformanceInspector). This thread regularly reads performance counters and calculates the Health Score. Each time it calculates the Health Score it will log this to the Trace Logs (Category=Http Throttling, Level=Verbose):
“The current health score for web application is X”
Performance Counters
The Health Score is calculated from a set of Performance Counters. By default SharePoint 2013 (and SharePoint 2010) uses two performance counters for this:
- Memory/Available MBytes
- ASP.NET/Requests Current
You can easily retrieve these performance counters by using the following cmdlet:
Get-SPWebApplicationHttpThrottlingMonitor http://app.contoso.com
You can also use the [
Set-SPWebApplicationHttpThrottlingMonitor
](http://technet.microsoft.com/en-US/library/ff607944(v=office.15).aspx) cmdlet to modify the current monitors.
Buckets, what is that!?
When running the cmdlet above, to retrieve the performance counters used to calculate the Health Score, you should have noticed that each associated performance counter has a property called
AssociatedHealthScoreCalculator
.
This is an array of values, or a set of Buckets, that SharePoint uses to calculate the Health Score. There are 10 buckets in each of those (you can have fewer). Each bucket represents a Health Score value and corresponds to a lower limit of the performance counter. For instance if you have 24 current ASP.NET requests – you will have a Health Score of 4 and if you have less than 200 MB of RAM available you will have a Health Score of 6. The Health Score of the Web Application is equal to the largest Health Score of all measured performance counters.
Adding your own ,monitor
To add your own performance counters to monitor you need to use the API. Here’s a sample on how to add the CPU usage as a monitored object. You should not add this one into any production environment, but it can be used in testing scenarios when you would like to see what happens when the Health Score is high.
$throttle = (Get-SPWebApplication http://app.contoso.com).HttpThrottleSettings
$throttle.AddPerformanceMonitor("Processor", "% Processor Time", "_Total", 10, $true)
$throttle.Update()
This will create a new monitor for the Web Application that checks the processor time on the machine. It will only have one bucket. So if the processor time is larger than 10 it will yield a Health Score of 10.
Refresh Interval
The Health Scores are by default updated every 5 seconds. You can get (or set) this setting by looking at the
HttpThrottlingSettings
of a Web Application like this:
$throttle = (Get-SPWebApplication http://app.contoso.com).HttpThrottleSettings
$throttle.RefreshInterval
To avoid having spikes in performance counters creating high Health Scores, the actual value is calculated from an average of a number of samples, by default 12 samples. This can be configured using the
NumberOfSamples
property on the
HttpThrottleSettings
.
How SharePoint uses the Health Score
Now you know a little bit more on how the SharePoint Health Score is calculated. So what is it used for then? SharePoint internally uses it for mainly two reasons; HTTP throttling and for the Request Management Service.
HTTP Throttling
Whenever the SharePoint Health Score reaches the value of 10, SharePoint 2013 starts its throttling mechanism to protect the servers. As soon as the Health Score is below 10, the servers stops the throttling. The actual throttling is divided into two steps – the first and second stage. After receiving a Health Score of 10 and SharePoint enters throttling it will start with the first stage and requests will start to be throttled. After one minute of throttling the throttling enters the second stage and starts throttling even more. What’s being throttled in the different stages depends on a set of throttling classifiers. Out of the box (in SharePoint 2013) there is only one classifier used – which checks if the current request is a search crawler. This classifier starts throttling request directly in the first stage – all other requests are handled as usual. When the throttling enter the second stage all requests will be throttled (default, if you have any custom classifiers some requests might be allowed).
Throttled requests are met with the Error message above: “The Server is busy now. Try again later”, and the HTTP Response: “503 Service Unavailable”.
When entering the second stage of throttling, you will see the following lines in the ULS logs. The logs also includes the “Excessive” counters.
To the Server is Busy response SharePoint also adds two Response Headers; one called “Retry-After” which contains a numeric value which should be taken as an indication (by clients) when a retry operation should be done (it’s always has the same value as the Health Score refresh interval) , and a second one “SharePointError” which always has the value of “2”.
By the way, if you notice the nice little error icon in the Busy error. That image is not throttled, why? Well, I can tell you…there’s actually an exclusion for that specific image with regards to resource throttling. Requests to that image will never be throttled.
HTTP Throttling of POST Requests?
There’s this myth out there that POST and PUT requests are never throttled. Well, that is to good to be true! My tests and reverse engineering skills shows that all requests are throttled in SharePoint 2013 (I would be glad to be proven wrong :). Fortunately Office 2010 and later has the “Upload Center” which will retry any save operations and eventually succeed once throttling has stopped.
But, using one of the built-in classifiers we can actually make sure that our precious documents are saved regardless of HTTP Throttling, and this is how this is achieved.
$classifier = New-Object Microsoft.SharePoint.Utilities.SPHttpUserAgentAndMethodClassifier
-Args "","POST","HttpMethodMatch", "Never"
$throttle = (Get-SPWebApplication http://app.contoso.com).HttpThrottleSettings
$throttle.AddThrottleClassifier($classifier)
$throttle.Update()
First of all we create a new classifier of the type
SPHttpUserAgentAndMethodClassifier
, when creating that object we pass in “POST” as HTTP method, make sure it’s using the HttpMethodMatch method to match the method :), and finally we tell it to never throttle these requests. We add this classifier object to the throttle settings of the web application and badabing, POST operations are not longer throttled.
Turning off HTTP Throttling
Ok, so you’re fed up with this throttling and want to turn it off. It’s an easy operation and you can do it from Central Administration or from PowerShell. In PowerShell just update the
PerformThrottle
property of the
HttpThrottleSettings
of the web application. And in Central Administration you select the Web Application for which you want to turn off throttling, then select General Settings > Resource Throttling, then at the bottom of the dialog you will find HTTP Request Monitoring and Throttling.
HTTP Throttling and automagic garbage collection
The HTTP Throttling is actually a quite smart implementation, and internally tries to fix your machine if throttling occurs. One thing that is important here is if you start fiddling with the different performance counters used for calculating the Health Score, make sure that you do not remove the monitor that monitors memory usage. That plays a special role in the throttling scenario. If that specific monitor is considered an “Excessive” monitor, which means that it has an Health Score of 10, it will actually force a Garbage Collection.
Request Management
SharePoint 2013 introduced the new Request Management Service, which is used to “redirect” requests to specific servers based on rules and/or health. When using health based rules, it is the SharePoint Health Score that is used to determine the health of a server. Whenever a machine receives a request it will update the Health Score in the Request Management service, which makes the Health Based Request Management a very efficient way to “load balance” the requests for optimal performance.
For more information about Request Management read Spencer Harbars series about it.
How to use the SharePoint Health Score
Whenever you’re building a client that interacts with SharePoint 2013, for instance a SharePoint App, a Windows 8 app or a Windows Phone app, then you should strongly consider always take the Health Score in consideration. You will receive the
X-SharePointHealthScore
response header for each and every CSOM, REST or web service call to SharePoint. If the value is 10 (and SharePoint starts throttling) then you will most likely not be able to get data. If your application is constantly polling SharePoint for information, perhaps you should increase the delays between your calls when the Health Score raises above a predetermined number.
Andrew Connell has a small code sample on how to leverage this in a Silverlight app.
The SPPerformanceInspector
You could also use the Health Score in traditional SharePoint Farm solutions (timer jobs, or other long running or resource consuming operations) using the
SPPerformanceInspector
object. Here’s a quick sample how you can show the data in a Web Part:
SPPerformanceInspector pi =
SPPerformanceInspector.GetInspector(SPContext.Current.Site.WebApplication);
this.Controls.Add(new LiteralControl(
String.Format("Current Health Score: {0}<br/>", pi.HealthScore)));
this.Controls.Add(new LiteralControl(
String.Format("Is throttling: {0}", pi.IsInThrottling())));
This Web Part retrieves the performance inspector object and then prints out the current Health Score for the current Web Application on the current machine. Also this Web Part will print out if throttling is currently going on (which you never should see anyways, since you’re throttled!).
How to debug the SharePoint Health Score
To wrap this little post up I will give you two great debugging tips – they both involve editing the registry (you’re warned!). First of all if you’re building a remote application that communicates with SharePoint and you want to test how it behaves at different Health Scores. Then there’s a neat trick to give a server a constant Health Score value. You do this by adding a new DWORD key called
ServerHealthScore
to the
HKLM\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\15.0\WSS
hive and then do an IISRESET. You can give this key any value from 0 to 10 and the Response Header will always show that value. It will not invoke throttling though.
It will of course log a warning in the ULS telling you how bad this really is:
The second tip is a cool debugging feature, and its also a small regkey hack. In the same hive as above create a new DWORD key called
DebugThrottle
and give it the value of 1.
Once again you need to do an IISRESET. After that you can browse to any page and then append
debugthrottle
to the address, for instance: http://site.contoso.com/debugthrottle. This will give you a sweet looking dashboard with all the details you need about the health of the server/web application. You can see the monitors and their Health Score history, the throttling stage, your classifiers etc.
Summary
There you have it! All you need to know about the SharePoint Health Score and HTTP Throttling. As you can see it’s a really nice feature if used correctly – by you, by third party vendors (I’m pretty sure 99% of the vendors do not care about this though) and by SharePoint itself. Just one final word of warning – if you’re fiddling with monitors and classifiers make sure that you thoroughly test them, it’s really easy to get things funked up here…