Tuesday, June 01, 2010

Fixing a Mac OS X Spotlight Index, that Doesn't

Mail.app "Entire Message" Greyed OutAfter a spate of Mail.app problems probably related to having too damn many mail accounts and messages, I had to do the "Mail.app Reimport Samba" taking hours to let Mail.app reimport the entire mail store. After that got resolved, I found out that my Mail.app "Entire Message" selection in Search was greyed out. This turned out to be a symptom of a problem with the Spotlight index, since this search function of Mail.app is dependent upon the Spotlight index of the hard drive where your mail is stored. Generally speaking, the Solution was to delete the Spotlight index, and then re-index the drive. However, it was not a simple process (is anything?!), and as such was very "un-Apple", so I thought I would take the time to document it.

How to Make Spotlight Re-index Your Drive, Even When She Says No!

A.k.a., How to Fix Greyed Out "Entire Message" in Mail.app Search. As I was learning to do this procedure in my OS X 10.6.3 system, looking at the Apple forums and elsewhere, I observed that Spotlight was taking "forever" to re-index, while it should re-index a 500GB drive in an hour or so. When I clicked the "pulsating" Spotlight icon in the upper right of the screen (it pulsates or winks while re-indexing), I saw a message that Spotlight was calculating the time required to index the drive with a barber pole progress bar, but it was just stuck in this state for days. This prompted me to try various things to fix it, and no one method from any one forum ever worked for me, in practice.

So, let me try to explain what did work.

Remove the Extraneous

Disconnect any external drives and shut down any unneeded programs.

Remove Unneeded Spotlight Importers

Spotlight importers which allow programs to get their output files indexed are sometimes the cause of Spotlight crashes. You can find them in /Library/Spotlight and ~/Library/Spotlight (where ~ is your home directory, of course). If you have third-party importers, especially in ~/Library/Spotlight, you can move them to another folder for safekeeping, then move them back one by one to see if the "indexing forever" problem recurs. I deleted ones that were associated with programs I never use, but kept the ones I do use.

You can search the entire drive for the .mdimporter files, using this command. Hat tip to "Hal Itosis" (lol) on the MacFixit Forum.

# find -x / -iname \*.mdimporter -exec ls -lndotT {} +

Repair Disk Permissions

I use the excellent Cocktail for this but you can do it in 3rd Party OnyX or Apple's included Disk Utility as well. In Cocktail, use the Disks menu, Permissions tab. Choose to Reset Permissions and ACLs for Home Directories for All Users, then click Repair. This takes about 30 mins to execute.

Prepare the Index

Using Terminal, use launchctl to unload the indexer by controlling the appropriate launchd command, use mdutil to turn off indexing for the root folder (the /) of the local hard drive, use rm to delete the index itself (.Spotlight-V100 in the root of the drive), and finally trash Spotlight's plist.

$ sudo bash

Password: *******

# launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

# mdutil -i off -E /

# cd /

# rm -rf .Spotlight-V100

# rm -rf ~/Library/Preferences/com.apple.spotlight.plist

# exit

$

To explain, note the sudo bash and exit lines, bookending the procedure. This gets you a root prompt (the #) after you enter the admin password and then exits the root prompt at the end (returning to the $ prompt). You can also use sudo before every command, to be extra safe.

Clear All Caches

Noting that this part will force a reboot, again in Cocktail Files, Caches, choose Options and select all caches you can. Executing this resets the system to a state close to what you get when you install anew or do a major upgrade. Note that the system will take longer to restart because it is rebuilding caches, and applications will be sluggish the first time they restart, because they are rebuilding font caches.

Click Clear to execute the cleaning procedure and wait while it finishes. Choose to let the system restart, but when you hear the startup chime, hold down Shift so the system restarts in Safe Mode.

Safe Mode Index Rebuild

The system will take a while to get into Safe Mode. After you press and hold Shift while starting up, once you see the progress bar during startup (which is not present in normal mode), you can let go of Shift and go get coffee while it starts up. The login prompt will have a red Safe Mode in it, to alert you to the difference.

Again in Terminal, issue some commands to rebuild.

$ sudo bash

Password: *******

# launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

# mdutil -i on -E /

/:

Indexing enabled.

# exit

$

After a few minutes you should see the Spotlight icon start to pulsate, but this time, the "Calculating Indexing Time" should change to some value, and a blue progress bar, which means the indexing is actually working. You can restart into regular mode, and the indexing just picked up where it left off, for me. However, do not connect any additional drives until it is done indexing the main drive.

Observing and Confirming

There are a few things you can do to observe and confirm Spotlight's activities, in Terminal. This assumes you are at a root prompt.

Check the size of the Spotlight index using du (directory usage):

# du -hsc /.Spot*

1.1G /.Spotlight-V100

1.1G total

Note that this should be a fairly large file. If it is only a megabyte or so, something is broken or your Spotlight index is off.

Check Spotlight-related processes with ps (process lister):

# ps axcru |sed '1p;/ md/!d'

USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND

root 352 0.0 2.8 3696728 117196 ?? Ss 8:44AM 24:25.73 mds

_spotlight 6677 0.0 0.2 2532748 10292 ?? SNs 3:19AM 0:00.33 mdworker

rcogley 6664 0.0 1.2 2638620 48240 ?? SNs 3:19AM 0:02.95 mdworker

Check index status with mdutil.

# mdutil -as

/:

Indexing enabled.

/Volumes/COGLEY-WD:

Indexing enabled.

/Volumes/COGLEY-WD/Backups.backupdb:

Indexing enabled.

/Volumes/COGLEY-WD2:

Indexing enabled.

/Volumes/COGLEY:

Indexing enabled.

List open files in /System/Library related to Spotlight, using lsof (list open files).

# lsof -c md |grep -v /System/Library |grep -v Spotlight

Get general system information with df and diskutil:

# df -lh

# diskutil info /

Show commands you have entered in the bash shell:

# history

Note that if you use an alternative shell like fish, some of the above commands will not work. From fish, just do sudo bash to get a bash-based root prompt. Be sure also to check the Console GUI application in /Applications/Utilities. This will show crash information for the mdworker program, which may give you hints to what is going on.

I hope this article helps someone, especially since this problem seems to be pretty prevalent, these days. Enjoy!

14 comments:

doug said...

Can I ask for your help? I have followed your procedure and it gets the internal drive to re-index, but I have one external drive out of many that will not re-index. Every-time I issue the mdutil -i on -E "/Volumes/Navy Films 8" I get a response of "Index disabled." Any ideas how to get this fixed?

Rick Cogley said...

Hi Doug - I wonder if the non-terminal way would work? System Preferences, Spotlight, Privacy. It is kludgy but you can drag the drive there, save, then hit the minus to remove it. It should kickstart the indexing.

Marshall said...

It seems the problem is being booted in safe mode.

Whenever I would run

sudo mdutil -i on -E /

in safe boot it would simply return "Index disabled".

As soon as I booted into normal mode and attempted to reinitialize the database, it began working.

Rick Cogley said...

Marshall - thanks for your investigation and followup. I will keep safe mode in mind for next time. --Rick

Will said...

The procedure given worked for me (snow leopard 10.6.6), but, like Marshall, I needed to re-enable indexing (mdutil -i on -E /) when NOT in safe mode (i.e., after logging out from safe mode and restarting).

Thanks!

Rick Cogley said...

Will, thanks for stopping by and for the additional corroboration.

Regards,
Rick

Eric Feurich BLOG said...

Hi There,

Thanks for posting this blog it helped me on the way. But I still have a problem with Spotligth. I have done everything in te article. The rebuild started but the .Spotligth folder stays stuck at 271 Mb but the mds process runs on 100%.
Any ideas on this?

Rick Cogley said...

Hi Eric- I had that once. It was getting stuck on a particular corrupt file but the details escape me.

Steve Hyde said...

Thanks for the post. The procedure worked for me, (10.6.8 2.8GHz iMac) though like some others I was unable to restart indexing in Safe Mode. However, I re-issued the commands after booting normally and all was well.

There were a few horrid moments upon Spotlight starting when it looked like it was going back into the Indexing forever mode, but after a while a realistic time for Indexing my hard drive came up and Spotlight carried on.

Much more importantly, my Mac which had been incredibly sluggish was back to its usual speed.

Brilliant!

Steve

rajesh said...

I have an iPad 3 (I love it very much) and I have an old Linux desktop I repaired last year. I need a MacBook Air as a primary computer because god only knows if on my Linux Desktop will crap out. Plus Macs are excellent for computer programming (what I want to do). I am 13 if that helps, so I can't get a job. Any suggestions on how to make money and save up?
@ imac repairs

Rick Cogley said...

Rajesh, I'm not sure what this has to do with OS X Spotlight.

Patrick Svensson said...

Thanks for the help!
I was unable to search in my documents folder and in mail.app
Then I ran the terminal scripts to "prepare" and rebuild, after eachother and without any restart in safemode or anything else.
Now it works just perfect again! :)

Carsten Dreesbach said...

These steps are good, but in addition I had to use Onyx in order to delete the system and user caches (I used Onyx's default settings for which caches to delete). Only after doing that did Spotlight indexing finally start up again.

YMMV: I have seen gazillions of different suggestions in various forum posts and everybody's experience seems to vary a bit. Looks like Spotlight has lots of different failure modes, none of which seem to be very well understood.

Anonymous said...

Thank you so much for this incredible tutorial that finally got the index bar to move again !

I've read dozens of post on Apple boards and elsewhere but this one is the only thing that fixed my system !

THANK YOU VERY MUCH !!!