I’ve recently been experiencing many issues with the Windows Search service on Server 2012R2 – The search index service would crash and cause entire RDP sessions to hang whenever typing something in the start menu. Of course on RDS servers this was a major issue and my clients got very upset due to not being able to work for multiple hours without calling us for assistance.
This issue was actually rather difficult to troubleshoot as all of the servers experiencing this had all Windows Updates installed and no similar software. We even found the issue on a brand new fresh install without any external software once.
After several days of constant troubleshooting we’ve found the following symptoms to be true in all cases;
- The Search Service itself does not crash – a subprocess of the search service(FilterHost) crashes.
- Restarting the Search service resolves the issue temporarily.
- Deleting and recreating the index does not resolve the issue.
- The Search Index grows to extreme sizes(50GB+).
- Offline Dedrag of the Search Databases often helps in shrinking the size, but does not resolve the issue.
Of course we’ve tried everything to resolve this – We’ve performed clean boots on servers, used procmon to find if another process causes the crash, enabled crash dumps and increased the time-out value on the search service to check if this was not a performance issue. Unfortunately none of these items helped in the slightest. The crashes kept on coming and our client actually considered removing the Windows Servers and moving to a different provider.
Afterwards I’ve been reading the details and I found we may have a classic scenario of the built-In search feature causing NTDLL thread pool exhaustion. Windows Search uses the NTDLL thread pool to achieve natural language process such as word breaking. It is possible that at the time of the crash there were many queued requests for Search. Maybe the server got too many requests and search couldn’t handle it. The Windows search has a limit of 512 threads to cater to the search operations. Seeing our clients have large operations with many Outlook and full-text-indexes for files this could be the case.
to resolve we restricted the search to use only a single thread per-query, which should resolve the problem if its indeed reaching the threads threshold due to many search queries.
This can be done by configuring this Registry value:
Name : CoreCount
Type : DWORD
Value : 1
Location : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search
After configuring this value and rebooting we’ve not seen these issues at all – And deployed the solution to our entire RDS farm, Of course some credits should go to my co-worker Maarten van der Horst as he was relentless and did not let this issue go. 🙂