He’s dead, Jim: SharePoint 2013 Search Troubleshooting

Recently we had some strange search issues in our very own intranet and its time to blog about some SharePoint 2013 Search Troubleshooting.

At first I noticed that our Search results were quite old, usually that is not the case because we enabled continuous crawling and I was surprised that it worked that well. How I noticed? I have tiny app that utilized search and supports our core business process: billing. But it suddenly stopped finding new items.

Issue 1: Broken web.config on our search server

As mentioned above I noticed that new items in our SharePoint Environment did not show up - you can quickly confirm that when you go to your search center and search for *****. Then check the Last Modified Refiner and as you can see below: we had a problem.

Last Modified refiner - with zero results for today

But once you identified the problem you can tackle it. In this case I told our internal IT that there is something wrong, we checked the crawl log: ~560.000 errors.

That this was not monitored is a different story - but the error I saw that was quite familiar: “Search error: crawler could not connect to the repository”. That means either the SharePoint is not running, there is a firewall in between - or as in our case it returned HTTP 500 errors for every request. I could easily confirm that by opening a browser on the SharePoint crawler itself and open the target site. I looked at our IT admin: “Did you change something?” -  he: “We deployed a feature and that did not work out that well…” - me: “Did you remove it everywhere except the search crawler?” - he nodded. The in this case broken web.config was quickly restored - started a full crawl to get rid of the many errors and we should be fine, right? Right?!!

Issue 2: Cannot plan query for index system

Because full crawls tend to take quite a while - even with like 100 dp/s - I checked the search results the next morning:

That did not work - every search query resulted in that error. The important part is the correlation ID

Ouch - what happened? I right away called our admin: “What did you do?” - he said  “I just changed the password of the farm account via central admin like I did several times” - mhh, that should work, nothing to blame here. So I asked him to send me the uls logs and some screens of the SharePoint Search Service Application:

Full crawl seems to work - a bit slower than it should tho

Crawl logs do not look too suspiciously.

Event log #1

Huston - we have a problem!

Event log - useful as ever to identify that you have a problem but sometimes it does not help you identify what problem you have… so I was keen to see the ULS entry for the correlation ID I saw during the search - so I got it:

SearchServiceApplication::Execute-Exception: Microsoft.SharePoint.SPException: Tried IMS endpoints for operation Execute: Cannot plan query for index system SP2f28f6df0f54. Index fragment ‘1’ has no available cells. Cell statuses: [Cell I.0.1 on node IndexComponent3: Cell status is set to ‘not available’ (cell out of sync or seeding)] at Microsoft.Office.Server.Search.Query.Ims.LoadBalancer.RoundRobinLoadBalancerContext.NextEndpoint(String operationName, String failMessage) at Microsoft.Office.Server.Search.Administration.SearchServiceApplication._ImsQueryInternalType.DoSpLoadBalancedImsOp[T](ImsBackedOperation`1 imsCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) at Microsoft.Office.Server.Search.Administration.SearchServiceApplication…. fe0fc09c-8c99-b017-25c9-d3feac78a369

That sounds not that healthy, especially the Cell status is set to ‘not available’ (cell out of sync or seeding). The index is made out of cells - if they are not there, Search is very unhappy. I can only conclude that changing the password during a full crawl is a bad idea.

So we first tried to restart the SharePoint Search Service - no result.

We restarted the entire box - no result.

So we heavy-heartedly did an index reset and the search results immediately came back - empty but working again. But trust me, resetting the index during a workday, with many search-driven applications around is not the greatest experience you can have - and you shouldn’t. There is room for improvement in our intranet, do you agree?