Slash Dot Dash: Rails searching with Sphinx

Rails searching with Sphinx

ben@slashdotdash.net (Ben) — Mon, 06 Aug 2007 20:30:00 +0100

Over the weekend I was implementing search for trawlr.com using Sphinx, the nginx of the search world (fast and Russian) according to Evan Weaver. Previously I was using Ferret, but I had to remove the search feature almost immediately due to the ferret indexes constantly corrupting and causing me a major headache. I decided to drop ferret in favour of Sphinx which I’ve lots of good things about recently.

Installation on my MacBook Pro required a slight adjustment of the mysql directories with mysql5 from MacPorts.

$ wget http://www.sphinxsearch.com/downloads/sphinx-0.9.7.tar.gz
$ tar xvzf sphinx-0.9.7.tar.gz
$ cd sphinx-0.9.7
$ ./configure --with-mysql-includes=/opt/local/include/mysql5/mysql/ --with-mysql-libs=/opt/local/lib/mysql5/mysql/
$ make
$ sudo make install

Initally, I chose to use Evan’s UltraSphinx plugin and it was very helpful to start with by auto-generating the sphinx.conf. After indexing the entire content of trawlr.com – almost 1.5 million blog posts – in just a few minutes I was suitably impressed. The search speed was also lightening fast. Unfortunately I had problems with my Rails app with the UltraSphinx plugin installed – very strange errors started occurring.

Having already looked at the alternative Sphinx plugins I decided to try acts_as_sphinx. After some small tweaks to the sphinx.conf file (and a re-index) the search was working and more importantly so was my Rails app. An alternative option is Sphincter which I did experiment with but struggled with the limited documentation – mostly concerning the configuration file required but YMMV.

$ rake sphinx:index
$ rake sphinx:start

Indexing on my MacBook Pro…

$ time rake sphinx:index

using config file 'sphinx.conf'...
indexing index 'items'...
collected 1455733 docs, 1255.2 MB
sorted 182.4 Mhits, 100.0% done
total 1455733 docs, 1255246639 bytes
total 438.695 sec, 2861316.50 bytes/sec, 3318.32 docs/sec

real    7m25.307s
user    4m28.963s
sys     0m17.578s

Searching with acts_as_sphinx via the console (ruby script/console) for the term ‘Google’, sorted by published date.

>> search = Item.find_with_sphinx 'Google', :sphinx => {:sort_mode => [:attr_desc, 'pub_date'], :page => 1}, :order => 'items.pub_date DESC'; 0
=> 0
>> search.total
=> 1000
>> search.total_found
=> 73717
>> search.time       
=> "0.000"

That’s with an index of 943Mb and almost 1.5 million items. Note that the search results are limited to 1,000 items due to the settings in my spinx.conf file.

Within the Rails controller, search is done via:

@items = Item.find_with_sphinx(params[:query], 
      :sphinx => {:sort_mode => [:attr_desc, 'pub_date'], :limit => 50, :page => (params[:page] || 1)}, 
      :order => 'items.pub_date DESC')

Updating the Sphinx index

There’s another rake task for updating the Sphinx index which can be called via a cron job, rather than ‘live’ updates. The rotate command allows the index to be rebuilt whilst the Sphinx daemon is running, forcing a restart once completed.

$ rake sphinx:rotate

Update

It looks like the UltraSphinx plugin requires edge Rails (thanks Evan)!
I’ve deployed the search updates to the live trawlr.com site (currently search is only visible from the reader view for logged in users).
Live search is fantastically quick
Created some new rake tasks (to go in lib/tasks/sphinx.rake) that allow you to have a sphinx.conf file per Rails environment (config/sphinx.development.conf and config/sphinx.production.conf). The available tasks are: s:index, s:rotate, s:start, s:stop, s:status and s:restart. The rake tasks assume that the Sphinx pid file is in the log directory (pid_file = log/searchd.pid).

"Rails searching with Sphinx" by Jan

Fri, 28 Mar 2008 10:29:28 +0000

Found it. The above error happened, because I set max_matches = 120. I found out, when I installed Dmytros client API and ran a query.

Sphinx::SphinxInternalError: searchd error: per-query max_matches=1000 out of bounds (per-server max_matches=120) from /Users/jan/Documents/test/Sphinx/vendor/plugins/sphinx-0.3.1/lib/client.rb:622:in `GetResponse’ from /Users/jan/Documents/test/Sphinx/vendor/plugins/sphinx-0.3.1/lib/client.rb:362:in `Query’ from (irb):2

Setting @maxmatches = 120 in acts_as_sphinx’ sphinx.rb fixed the problem.

Passing on this helpful error message instead of “112” would be great.

"Rails searching with Sphinx" by Jan

Fri, 28 Mar 2008 10:10:07 +0000

Another question on the per-environment sphinx configs. How did you modify acts_as_sphinx to be using different servers for different environments?

"Rails searching with Sphinx" by Jan

Fri, 28 Mar 2008 10:09:07 +0000

I cannot get it running on rails 1.2.3 with sphinx 0.9.7. :-/

@articles = Article.find_with_sphinx(‘Jan’, :sphinx => {:page => @page})

Sphinx::SphinxInternalError in ArticleController#fulltext_search searchd error: 112

/Users/jan/Documents/Platinnetz/vendor/plugins/acts_as_sphinx/lib/sphinx.rb:437:in `get_response’ /Users/jan/Documents/Platinnetz/vendor/plugins/acts_as_sphinx/lib/sphinx.rb:232:in `query’ /Users/jan/Documents/Platinnetz/vendor/plugins/acts_as_sphinx/lib/acts_as_sphinx.rb:83:in `ask_sphinx’ /Users/jan/Documents/Platinnetz/vendor/plugins/acts_as_sphinx/lib/acts_as_sphinx.rb:112:in `find_with_sphinx’ /Users/jan/Documents/Platinnetz/app/controllers/article_controller.rb:353:in `fulltext_search’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:76:in `process’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in `synchronize’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in `process’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:159:in `process_client’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `each’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `process_client’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `initialize’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `new’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `initialize’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `new’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `run’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:282:in `run’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in `each’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in `run’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:128:in `run’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/lib/mongrel/command.rb:212:in `run’ /Library/Ruby/Gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:281

"Rails searching with Sphinx" by seog

Tue, 22 Jan 2008 18:05:28 +0000

Thanks for the tutorial! I am looking for a good search engine for rails and it seems more people are mentioning Sphinx over Ferret due to stability issues in deployment.

"Rails searching with Sphinx" by urlsbox

Mon, 03 Dec 2007 12:30:45 +0000

Very useful information. Thanks !

"Rails searching with Sphinx" by nguma

Wed, 28 Nov 2007 19:36:56 +0000

did you manage to use the filters? can’t seem to find the correct syntax.

tried :sphinx => {:filter => [‘attribute_name’,[1,2,3]]} without any success

"Rails searching with Sphinx" by Eric Hodel

Fri, 31 Aug 2007 17:39:47 +0100

Sphincter runs on 1.2.3 and has per-environment sphinx.conf files built-in.

"Rails searching with Sphinx" by Ben

Fri, 31 Aug 2007 14:57:34 +0100

acts_as_sphinx doesn’t require Edge; I’m using it with Rails 1.2.3.

"Rails searching with Sphinx" by Joey Marchy

Fri, 31 Aug 2007 14:45:10 +0100

Thanks for the Article Ben. I too am having issues with ferret and indexes so I am looking to implement Sphinx as my search also. I saw the UltraSphinx plugin requires running rails edge. Do you know if the acts_as_spinx will run without edge?

I am currently running rails 1.2.3

"Rails searching with Sphinx" by Eric Hodel

Tue, 21 Aug 2007 02:35:38 +0100

For Sphincter there is no configuration file. I built it that way on purpose.

If you need to tweak something you can, but most of the time you won’t.

"Rails searching with Sphinx" by evan

Tue, 07 Aug 2007 00:00:07 +0100

Yeah, sorry about that. I added a note to the documentation.

"Rails searching with Sphinx" by Ben

Mon, 06 Aug 2007 21:02:53 +0100

@evan – Ah, that’s probably why it all went tits-up – I’m running Rails 1.2.3 rather than Edge.

"Rails searching with Sphinx" by evan

Mon, 06 Aug 2007 20:56:59 +0100

What errors did you get? It might not work well if you’re not on edge Rails.