Slash Dot Dash

Evan Weaver has just announced delta indexing support for his UltraSphinx Rails plugin.

Delta indexing speeds up your updates by not reindexing the entire dataset every time.

First, in your .base file, set the indexer option delta to your maximum interval between full reindexes. A day or a week is good, depending. Add a little bit to account for the time it takes the actual index to run:

delta = <%= 1.day + 30.minutes %>

Now, configure your models for delta indexing in the is_indexed call:

is_indexed :fields => ['created_at', 'title', 'body'], :delta => true

Now you can run rake ultrasphinx:index:delta frequently, and only records that were changed within 1 day will be reindexed. You will need to run rake ultrasphinx:index:main once a day to move the delta contents into the main index.

From the UltraSphinx documentation.

I’ve previously written about how fast Sphinx search is in Rails searching with Sphinx using the acts_as_sphinx plugin (and also contributed a chapter to the Advanced Rails Recipes book using the UltraSphinx plugin). I’d now recommend using the UltraSphinx plugin if you want to take advantage of Sphinx for your Rails application.

In my opinion, this completes the ‘missing’ piece of functionality and should make Sphinx your default full-text search engine (if you’re using MySQL).

The latest Beta update to the new Pragmatic Programmers book Advanced Rails Recipes: 72 New Ways to Build Stunning Rails Apps includes two chapters authored by myself.

They are both complete rewrites and updates of two of my popular blog posts Rails searching with Sphinx and iPhone on Rails – Creating an iPhone optimised version of your Rails site using iUI and Rails 2.

The new recipes are:

• Full-Text Search with Sphinx – includes a Rails site that uses Sphinx to index the Rails API.

• Support An iPhone Interface – with a TODO list app.

Along with these two (worth the purchase price on their own!), there are another 79 great recipes with some of the latest Rails best practices in small, easy to implement chunks.

Just a small tip, if you wish to ensure a snippet of Ruby code doesn’t run for too long you can use the timeout function. You might want to do this when making a request to a remote server with net/http for example.

timeout.rb

A way of performing a potentially long-running operation in a thread, and terminating it‘s execution if it hasn‘t finished within fixed amount of time.

Here’s a quick example using the excellent rFeedParser (Universal Feed Parser in Ruby) to fetch an RSS feed.

require 'timeout'
require 'zlib'
require 'rubygems'
require 'rfeedparser'

fp = nil
begin
  # Don't take longer than 20 seconds to retrieve & parse an RSS feed
  Timeout::timeout(20) do
    fp = FeedParser.parse("http://feeds.feedburner.com/slashdotdash")
  end
rescue Timeout::Error
  # Too slow!!
end

This Thursday, 24th Jan, is the latest Pizza on Rails event where you can enjoy free pizza & beer, meet-up with fellow Rails developers and find out what Cominded has been working on when we announce our exciting new Rails site.

Join the Pizza On Rails mailing list for further details and to RSVP to indicate your attendance. It looks like it’ll be another great night with almost 100 people already signed up.

Got a copy of Agile Web Development With Rails

Wed 9th Nov, 2005
Order #: 202-0160452-2867026 (amazon)
Items: 1 x Agile Web Development with Rails

Purchased TextMate (and a Mac to run it on)!

Announced first Rails site

Sat, 29th Apr 2006
trawlr.com online rss agregator

Started working professionally with Rails

Mon, 5th November 2007
Cominded

After upgrading trawlr.com to Rails 2 I thought I’d make use of some of the new features and attempt to create an iPhone version of the site. With Rails 2 you can create a mime type specifically for the iPhone and then use that format in a respond_to block (along with views such as index.iphone.erb).

Before you start – iPhoney

iPhoney is an indispensable Mac-only tool for aiding the development of an iPhone specific site.

Looking for a way to see how your web creations will look on iPhone? Look no further. iPhoney gives you a pixel-accurate web browsing environment—powered by Safari—that you can use when developing web sites for iPhone. It’s the perfect 320 by 480-pixel canvas for your iPhone development. And it’s free. iPhoney is not an iPhone simulator but instead is designed for web developers who want to create 320 by 480 (or 480 by 320) websites for use with iPhone. It gives you a canvas on which to test the visual quality of your designs.

Ensure iPhoney’s user agent is set to iPhone User Agent in the menu.

iPhone mime type

Create an iPhone mime type alias using Rails 2 initializers.

config/initializers/mime_types

Mime::Type.register_alias "text/html", :iphone

Detecting iPhone user agents

Apple recommends that rather than redirecting iPhone users to an iPhone-optimised version of your site you should instead show the original site with a link to the alternative.

This can be achieved via user agent sniffing; looking for Mobile Safari (as Apple suggests), rather that iPhone or iPod touch, to allow for future device support.

Adding a helper method to application_helper.rb allows a notification message to be shown for only iPhone users (try accessing www.trawlr.com from an iPhone).

app/helpers/application_helper.rb

# Request from an iPhone or iPod touch? (Mobile Safari user agent)
def iphone_user_agent?
  request.env["HTTP_USER_AGENT"] && request.env["HTTP_USER_AGENT"][/(Mobile\/.+Safari)/]
end

In your view, show a message for iPhone user agents directing them to the iPhone version.

<% if iphone_user_agent? # Show message for iPhone users -%>
<div class="message">
    <p>Using an iPhone? <a href="http://iphone.trawlr.com/">Use the optimised version</a>.</p>
</div>
<% end -%>

iPhone subdomain

Instead of forcing users straight to our iPhone version, we offer them the option by using a separate subdomain (iphone.trawlr.com) with a link back to the regular site if they wish. When developing locally I modified my /etc/hosts file as follows so that I could use http://iphone.localhost.com:3000/.

/etc/hosts

127.0.0.1 iphone.localhost.com

You may need to flush the DNS cache after making the changes.

sudo dscacheutil -flushcache

Adjust format for iPhone

I chose to require login for all requests to the iPhone version of the site.

class ApplicationController < ActionController::Base
    before_filter :adjust_format_for_iphone
    before_filter :iphone_login_required

private

  # Set iPhone format if request to iphone.trawlr.com
  def adjust_format_for_iphone    
    request.format = :iphone if iphone_request?
  end

  # Force all iPhone users to login
  def iphone_login_required
    if iphone_request?
      redirect_to login_path unless logged_in?
    end
  end

  # Return true for requests to iphone.trawlr.com
  def iphone_request?
    return (request.subdomains.first == "iphone" || params[:format] == "iphone")
  end
end

Note that sessions_controller.rb (handles login) requires skip_before_filter :iphone_login_required.

Using iUI and creating iPhone views

The iUI framework, based on Joe Hewitt’s iPhone navigation work, hugely simplifies iPhone web development. All you need to do is include the iUI JavaScript and CSS files along with included images and create your views in a particular structure to have native iPhone behaviour such as sliding menus and AJAX page loading.

Rails 2 makes it trivial to create different views depending upon the format, including layouts. Our iPhone layout includes a few specifics for iUI and a viewport meta tag for the device.

app/views/layouts/application.iphone.erb

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8" />
    <meta id="viewport" name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>
    <title><%= @page_title -%></title>
  <%= stylesheet_link_tag 'iui' %>
  <%= javascript_include_tag 'iui' %>
</head>
<body>
    <div class="toolbar">
        <h1 id="pageTitle"></h1>
        <a id="backButton" class="button" href="#"></a>
    </div>

    <%= yield %>
</body>
</html>

When creating your iPhone views you should follow the iUI style guide, an example page is given below. Standard hyperlinks are loaded using AJAX and slide into view, navigating back is handled by iUI. Links may be prefixed with target="_self" to replace the entire page or target="_replace" to replace the element with the response (using AJAX).

index.iphone.erb

<ul title="Home" selected="true">
    <li><%= link_to 'Example action', example_path %></li>
    <li><%= link_to 'Logout', logout_path, :method => :delete, :target => '_self' %></li>
</ul>

show.iphone.erb

<div class="panel" title="Example" selected="true">
    <h2>Example Content</h2>
    <p>Here's some content</p>
</div>

It’s important to remember that iUI will load content using AJAX, thus you only need to render a layout (such as application.iphone.erb) for the first request or page of your iPhone site. All following requests should use render :layout => false (unless loaded into a new page with target="_replace"). If you experience any wierd rendering issues it’ll most likely be due to this irregularity.

respond_to do |format|
    format.iphone do  # action.iphone.erb
      render :layout => false
    end
end

Show, don’t tell

Why not try iphone.trawlr.com for yourself? It’ll work for your iPhone or any web browser. You’ll need to register via the normal site if you don’t already have an account (quick, no email registration required)!

References

The following resources on the new Rails 2 iPhone format ability and iUI library were extremely helpful; the documentation from Apple not so much!

• http://developer.apple.com/iphone/
• http://www.railspikes.com/2007/11/8/iphone-subdomains-with-rails
• http://blog.nicksieger.com/articles/2007/09/18/railsconf-europe-david-heinemeier-hansson
• http://www.joehewitt.com/blog/introducing_iui.php
• http://code.google.com/p/iui/

Yes, I’m still loving the iPhone!

Over the weekend I decided to try out Rails 2 by upgrading an existing site (http://www.trawlr.com/) from Rails 1.2.6 to Rails 2 RC2.

Check for deprecations before you upgrade

You may want to check your existing application for deprecated code before upgrading using the following rake task.

lib/tasks/rails.rake

desc "Checks your app and gently warns you if you are using deprecated code." 
task :deprecated => :environment do
  deprecated = {
    '@params'    => 'Use params[] instead',
    '@session'   => 'Use session[] instead',
    '@flash'     => 'Use flash[] instead',
    '@request'   => 'Use request[] instead',
    '@env' => 'Use env[] instead',
    'find_all'   => 'Use find(:all) instead',
    'find_first' => 'Use find(:first) instead',
    'render_partial' => 'Use render :partial instead',
    'component'  => 'Use of components are frowned upon',
    'paginate'   => 'The default paginator is slow. Writing your own may be faster',
    'start_form_tag'   => 'Use form_for instead',
    'end_form_tag'   => 'Use form_for instead',
    ':post => true'   => 'Use :method => :post instead'
  }

  deprecated.each do |key, warning|
    puts '--> ' + key
    output = `cd '#{File.expand_path('app', RAILS_ROOT)}' && grep -n --exclude=*.svn* -r '#{key}' *`
    unless output =~ /^$/
      puts "  !! " + warning + " !!" 
      puts '  ' + '.' * (warning.length + 6)
      puts output
    else
      puts "  Clean! Cheers for you!" 
    end
    puts
  end
end

Use rake to execute the task.

rake deprecated

With any luck you won’t get many warnings; it should give you a rough estimate on how long your upgrade may take.

Getting Rails 2

First you need to get the RC2 (or newer) tagged Rails source using rake. This downloads the Rails framework to your appliction’s vender/rails directory.

rake rails:freeze:edge TAG=rel_2-0-0_RC2

Upgrade issues

After upgrading, it’s worth running your test suite looking for any problems. The following are issues I ran into to get trawlr.com working.

Singular resources now map to plural controllers – Override by using the :controller option in routes.rb

config/routes.rb

# Singleton reader resource
map.resource :reader, :controller => 'reader'

with_scope is now protected – Use .send(:with_scope) to call method

This caused an issue due to using the magic join model pattern, although the fix is relatively simple.

MyModel.send(:with_scope, args)

asset_packager plugin broken – Apply fix

The fabulous asset_packager plugin required a quick change to get working again (single-line change).

Index: vendor/plugins/asset_packager/lib/synthesis/asset_package_helper.rb
===================================================================
--- vendor/plugins/asset_packager/lib/synthesis/asset_package_helper.rb    (revision 86)
+++ vendor/plugins/asset_packager/lib/synthesis/asset_package_helper.rb    (working copy)
@@ -37,7 +37,7 @@
     private
       # rewrite compute_public_path to allow us to not include the query string timestamp
       # used by ActionView::Helpers::AssetTagHelper
-      def compute_public_path(source, dir, ext, add_asset_id=true)
+      def compute_public_path(source, dir, ext = nil, add_asset_id=true)
         source = source.dup
         source << ".#{ext}" if File.extname(source).blank?
         unless source =~ %r{^[-a-z]+://}

I still prefer this to the new Rails 2 asset merging as it also minifies JS and CSS files (including stripping comments) and also allows you to specify multiple asset groups (for example one grouping for the main site, another for an iPhone version). A word of caution if you use asset_packager you cannot take advantage of the new Rails 2 asset servers (see below).

# DOES NOT work with asset_packager
config.action_controller.asset_host = "http://asset%d.site.com" 

Nested route helpers changed – Must specifiy parent resource

If you have any nested routes it’s likely you will have to alter the named route helpers, for example child_path is now parent_child_path.

start_form_tag and end_form_tag have been deprecated – Quick fix is to replace with form_tag and </form> respectively.

<%= form_tag articles_path %> 
  <%= text_field :article, :title %> 
  <%= submit_tag "Save" %> 
</form>

render_without_layout has been deprecated – Use :layout => false instead.

render :layout => false

restful_authentication plugin is broken – Replace redirect_to_url (deprecated) in lib/authenticated_system.rb with to redirect_to.

Index: authenticated_system.rb
===================================================================
--- authenticated_system.rb    (revision 86)
+++ authenticated_system.rb    (working copy)
@@ -94,7 +94,7 @@
     # Redirect to the URI stored by the most recent store_location call or
     # to the passed default.
     def redirect_back_or_default(default)
-      session[:return_to] ? redirect_to_url(session[:return_to]) : redirect_to(default)
+      session[:return_to] ? redirect_to(session[:return_to]) : redirect_to(default)
       session[:return_to] = nil
     end

Optionally, rename your views

Rails 2 includes some changes to the way views are named. Previously you may have had show.rhtml and show.rjs which now become show.html.erb and show.js.rjs to indicate the mime type and template engines used. You don’t have to change your old views, but the following rake task should make it a quick change if you decide to. For trawlr.com I chose to use the new format for any new view templates but left the existing ones as they were (I’ll probably rename as changes are made on an individual basis).

lib/tasks/rails.rake

namespace 'views' do
  desc 'Renames all .rhtml views to .html.erb, .rjs to .js.rjs, .rxml to .xml.builder, and .haml to .html.haml'
  task 'rename' do
    Dir.glob('app/views/**/[^_]*.rhtml').each do |file|
      puts `svn mv #{file} #{file.gsub(/\.rhtml$/, '.html.erb')}`
    end

    Dir.glob('app/views/**/[^_]*.rxml').each do |file|
      puts `svn mv #{file} #{file.gsub(/\.rxml$/, '.xml.builder')}`
    end

    Dir.glob('app/views/**/[^_]*.rjs').each do |file|
      puts `svn mv #{file} #{file.gsub(/\.rjs$/, '.js.rjs')}`
    end
    Dir.glob('app/views/**/[^_]*.haml').each do |file|
      puts `svn mv #{file} #{file.gsub(/\.haml$/, '.html.haml')}`
    end
  end
end

Resources

For more information on Rails 2 refer to the preview release blog post. For $9 you can pick up a copy of Ryan Daigle’s worthwhile Rails 2 PDF book available via peepcode.

RailsConf Europe 2007 in Berlin finished just over a week ago and for those of us who didn’t make it along the presentation slides and videos are now appearing online. David Heinemeier Hansson’s keynote was recorded and published thanks to the guys at Rails on Wave.

RailsConf Europe 2007 Presentations

DHH Keynote Video (full-screen)

Further details from DHH’s talk have been blogged in detail. Personally, I’m really looking forward to the release of Rails 2.0; evolution rather than revolution.

Videos and slides for every session from the Ruby Hoedown 2007 (August 10th – 11th) are now available to watch on the confreaks.com site. Haven’t watched any of these yet but there are a few gems, such as Marcel Molina Jr.’s keynote on beautiful code.

Charity Workshop: Ruby and Rails Testing Techniques Marcel Molina, Jr., Bruce Tate, Chad Fowler

Exploring Merb Ezra Zygmuntowicz

Next-Gen VoIP Development with Ruby and Adhearsion Jay Phillips

Keynote Address: The Journey Bruce Tate

Building Games with Ruby Andrea O.K. Wright

Lightning Talks Various Authors

Does Ruby Have a Chasm to Cross? Ken Auer

Using C to Tune Your Ruby (or Rails) Application Jared Richardson

Keynote Address: What makes code beautiful? Marcel Molina, Jr.

Over the weekend I was implementing search for trawlr.com using Sphinx, the nginx of the search world (fast and Russian) according to Evan Weaver. Previously I was using Ferret, but I had to remove the search feature almost immediately due to the ferret indexes constantly corrupting and causing me a major headache. I decided to drop ferret in favour of Sphinx which I’ve lots of good things about recently.

Installation on my MacBook Pro required a slight adjustment of the mysql directories with mysql5 from MacPorts.

$ wget http://www.sphinxsearch.com/downloads/sphinx-0.9.7.tar.gz
$ tar xvzf sphinx-0.9.7.tar.gz
$ cd sphinx-0.9.7
$ ./configure --with-mysql-includes=/opt/local/include/mysql5/mysql/ --with-mysql-libs=/opt/local/lib/mysql5/mysql/
$ make
$ sudo make install

Initally, I chose to use Evan’s UltraSphinx plugin and it was very helpful to start with by auto-generating the sphinx.conf. After indexing the entire content of trawlr.com – almost 1.5 million blog posts – in just a few minutes I was suitably impressed. The search speed was also lightening fast. Unfortunately I had problems with my Rails app with the UltraSphinx plugin installed – very strange errors started occurring.

Having already looked at the alternative Sphinx plugins I decided to try acts_as_sphinx. After some small tweaks to the sphinx.conf file (and a re-index) the search was working and more importantly so was my Rails app. An alternative option is Sphincter which I did experiment with but struggled with the limited documentation – mostly concerning the configuration file required but YMMV.

$ rake sphinx:index
$ rake sphinx:start

Indexing on my MacBook Pro…

$ time rake sphinx:index

using config file 'sphinx.conf'...
indexing index 'items'...
collected 1455733 docs, 1255.2 MB
sorted 182.4 Mhits, 100.0% done
total 1455733 docs, 1255246639 bytes
total 438.695 sec, 2861316.50 bytes/sec, 3318.32 docs/sec

real    7m25.307s
user    4m28.963s
sys     0m17.578s

Searching with acts_as_sphinx via the console (ruby script/console) for the term ‘Google’, sorted by published date.

>> search = Item.find_with_sphinx 'Google', :sphinx => {:sort_mode => [:attr_desc, 'pub_date'], :page => 1}, :order => 'items.pub_date DESC'; 0
=> 0
>> search.total
=> 1000
>> search.total_found
=> 73717
>> search.time       
=> "0.000" 

That’s with an index of 943Mb and almost 1.5 million items. Note that the search results are limited to 1,000 items due to the settings in my spinx.conf file.

Within the Rails controller, search is done via:

@items = Item.find_with_sphinx(params[:query], 
      :sphinx => {:sort_mode => [:attr_desc, 'pub_date'], :limit => 50, :page => (params[:page] || 1)}, 
      :order => 'items.pub_date DESC')

Updating the Sphinx index

There’s another rake task for updating the Sphinx index which can be called via a cron job, rather than ‘live’ updates. The rotate command allows the index to be rebuilt whilst the Sphinx daemon is running, forcing a restart once completed.

$ rake sphinx:rotate

Update

• It looks like the UltraSphinx plugin requires edge Rails (thanks Evan)!
• I’ve deployed the search updates to the live trawlr.com site (currently search is only visible from the reader view for logged in users).
• Live search is fantastically quick
• Created some new rake tasks (to go in lib/tasks/sphinx.rake) that allow you to have a sphinx.conf file per Rails environment (config/sphinx.development.conf and config/sphinx.production.conf). The available tasks are: s:index, s:rotate, s:start, s:stop, s:status and s:restart. The rake tasks assume that the Sphinx pid file is in the log directory (pid_file = log/searchd.pid).