Thursday, April 19, 2012

Ruby Metaprogramming with class_eval

One of the really cool and exciting aspects of using a dynamic programming language like Ruby is that you can generate code dynamically at runtime. Yup, you heard me, at runtime. Try doing something like this in Java. This is one of the features that allows all the magic that happens behind the scenes in Ruby on Rails possible. For example, this is how Rails' attr_accessor automatically creates getters and setters for you. If you don't remember, you can call attr_accessor in an ActiveRecord class to define a "virtual" attribute, or in other words an attribute that isn't persisted in the database table. I want to explain this idea in more detail by creating a new method called attr_accessor_with_history, which is similar to attr_accessor, but it will also store the attribute's history. Say we have the following class:
class Test
   attr_accessor_with_history :sample
end
Then this is how the attr_accessor_with_history method should work in the Ruby console:
t = Test.new
t.sample = "test"  # => "test"
t.sample = 1       # => 1
t.sample = :abc    # => :abc
t.sample_history   # => [nil, "test", 1, :abc]
So how can we do this? Let's first create a new file called class.rb and put the following inside:
class Class
   def attr_accessor_with_history(attr_name)

   end
end
By defining our method in the class Class, all Ruby classes will be able to use it. This is because in Ruby, all classes are an object of the class Class. You can find more information in the API here. The second thing we want to know is that Ruby provides a method called class_eval, which takes a string and evaluates it in the context of the current class. So in our example, the class Test is calling our method, so when class_eval is called, it will be evaluated as if it were called in the class Test.

Let's take care of the easy part first, which is creating the attribute's getter methods. This is pretty straight forward because we don't really need to do anything special here.
class Class
   def attr_accessor_with_history(attr_name)
      attr_name = attr_name.to_s         # make sure it's a string
      attr_reader attr_name              # create the attribute's getter
      attr_reader attr_name + "_history" # create bar_history getter

      # write our setter code below
      class_eval %Q(
                  def #{attr_name}=(attr_name)
                  end
                 )
   end
end
The %Q is Ruby syntax for an interpolated string, which allows us to create a string, and it will also interpret Ruby values such as anything in #{}. So in thinking about how to define our setter, we need to store the previous history of the variable. How can we do this? Well we can store stuff in an Array. If we push things in to the Array like a Stack, we can also figure out the order in which things were inserted. The one thing we need to be careful about is the initial case where we should initialize the array, and also insert the first value as nil. This is what we get:
class Class
   def attr_accessor_with_history(attr_name)
      attr_name = attr_name.to_s         # make sure it's a string
      attr_reader attr_name              # create the attribute's getter
      attr_reader attr_name + "_history" # create bar_history getter

      # write our setter code below
      class_eval %Q(
                  def #{attr_name}=(attr_name)
                     @#{attr_name} = attr_name

                     unless @#{attr_name + "_history"}
                        @#{attr_name + "_history"} = []
                        @#{attr_name + "_history"} << nil
                     end

                     @#{attr_name + "_history"} << attr_name

                   end
                 )
   end
end
First we set the value of the variable. Second, we check if the _history array has been initialized yet, and if it hasn't, then go ahead and initialize it. Lastly, we insert the new value into the _history array.

That was just a quick example of some of the cool things you can do with metaprogramming in Ruby. I want to end by showing a real world example of how I used metaprogramming on a project I'm currently working on. If you haven't read my previous blog posts, I'm working on a Rails site using MongoDB (with the Mongoid gem) that deals with computer games. I have a class called Event, and I need to define some scopes based on the game type so that I can do queries such as Event.all.starcraft2, which would return all Events that have a game attribute of 'starcraft2'. I could easily hard code these scopes, but it will be annoying to always have to add a new scope every time we add support for a new game. Instead I can dynamically create these scopes using Ruby metaprogramming.
  class_eval %Q(
              #{Constants::GAMES}.each_pair do |key, value|
                scope key, where(game: value)
              end
             )
First I need to explain the Constants::GAMES variable. I created a dynamic_fields.yml file that stores all the games that my application supports. Please refer to my previous post here for more detailed information. So when my application loads, it reads in that YML file into the Constants::GAMES variable, which is a hash. The key is the game name as a symbol, and the value is the game name as a string. For each game, it will create a scope with the same name (as the game), and it does all of this automatically. I could have wrote a line of code for each game, but this is much cleaner to write and easier to maintain in the future.

Friday, April 6, 2012

Installing and compiling Ruby from the source code

There are several different ways to install Ruby with the most popular being RVMand rbenv. Both have their advantages and disadvantages, but rbenv is a newer and more "lightweight" Ruby installer. I'll let you Google the differences between RVM and rbenv if you are more curious. I decided to install Ruby by downloading the source code and then compiling it myself. I'll be installing Ruby 1.9.3-p125 (the latest version as of writing this) on Debian Linux 6 (Squeeze).

1. First we need to install some development tools and libraries that Ruby needs to compile. Run the following commands from your terminal:
   apt-get install build-essential vim git-core curl

   apt-get install bison openssl libreadline6 libreadline6-dev zlib1g zlib1g-dev libssl-dev libyaml-dev libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev

   apt-get install libcurl4-openssl-dev libopenssl-ruby apache2-prefork-dev libapr1-dev libaprutil1-dev

   apt-get install libx11-dev libffi-dev tcl-dev tk-dev
Note you may need to run these commands as root or use sudo.

2. Download the latest version of Ruby here: http://www.ruby-lang.org/en/downloads/ under the Compiling Ruby — Source code section.

3. Extract the tar.gz file e.g. ruby-1.9.3-p125.tar.gz by typing:
tar -xvf ruby-1.9.3-p125.tar.gz
4. Open the README and follow the instructions on how to compile and install Ruby.
./configure
make
make test
make install
The install process will probably take a while, but once it's done, we can test to see if ruby installed successfully by typing "ruby -v" in the terminal. You should see something like, "ruby 1.9.3p125 (2012-02-16 revision 34643) [i686-linux]" if ruby installed correctly. Next we can install the Bundler and Rails gems to make sure Ruby is working correctly.
gem install bundler
gem install rails
Lastly, create a new Rails project to test to make sure everything works!

Wednesday, March 7, 2012

Dynamic attributes and MongoDB/Mongoid

I recently started learning more about MongoDB, and how to use it in Rails 3.1 with the Mongoid gem. MongoDB is not your typical relational database; It is considered a NoSQL database because it stores your data in one huge JSON document instead of a bunch of relational tables. Actually, instead of storing the data in JSON, it stores it in BSON (Binary JSON) for performance reasons. The really cool thing about MongoDB is that it is schema-less, which allows you to store non-homogeneous objects in your database.

Why is this cool you might ask? Well let me explain my problem; I want to create a web service that allows CRUD operations on objects called Events. Events have a name, start_time, and most importantly a game. Depending on the game attribute, each Event will have different game specific attributes. For example, I have an Event with a game value of 'starcraft2'. Only Starcraft 2 Events have a region and a league attribute, while League of Legends Events have a min_elo attribute. How can we store these Event objects that have different attributes in our one document? We could do this in a standard MySQL database, but our table would be very wide, meaning we would have to have a column for every game specific attribute. This isn't very ideal because for each row, there would be a bunch of columns that wouldn't be used depending on the game.

The great thing about Mongoid is that it supports dynamic attributes. I quote this from their website, "By default Mongoid supports dynamic fields - that is it will allow attributes to get set and persisted on the document even if a field was not defined for them." This sounds like exactly what we need. We can just define the common Event attributes in the model itself, and then specify the game specific attributes at run time. So how do we go about doing this? Well, let's start with the Event model:
class Event
   include Mongoid::Document

   field :name, type: String
   field :start_time, type: DateTime
   field :game, type: String
end
This is pretty straight forward. Now the way I chose to define game specific attributes is by defining them in a YML file. I created a file called dynamic_fields.yml and put it in the Rails.root/config folder. Here is how I defined it:
starcraft2:
   attributes:
      regions:
         name: regions
      leagues:
         name: leagues
You can define your dynamic attributes however you want, but this is just how I decided to do mine. So now that we have our Event model, and our Starcraft 2 attributes, how do we set these when an Event gets created, and how do we make sure these attributes only get set if the game is Starcraft 2? First we need to understand what mass assignment is, and how it could be a security vulnerability. Here is an excellent RailsCasts on that topic, but the gist of it is that when the server receives a POST request to create a new Event, it will take the parameters it receives and mass assign themto the matching attributes of the Event model. For example, if we had a User model, and a is_admin attribute, the user shouldn't be able to set the value of the is_admin attribute. If they could, then everyone could make themselves an admin when they create a new account. So how do we prevent this? In Rails there is a very useful method called attr_accessible that we can set on attributes in our model. Only the attributes you set using attr_accessible will be available for mass assignment. So let's do that for our Event model for all the common fields:
class Event
   include Mongoid::Document

   field :name, type: String
   field :start_time, type: DateTime
   field :game, type: String

   attr_accessible :name, :start_time, :game
end
This reason this is so important, especially in Mongoid is that since we enabled dynamic attributes, a user can create an Event with any additional attributes they want. They can add an attribute called "some_random_field" by including it in the POST parameters and it would be inserted in our database no questions asked. By setting attr_accessible on name, start_time, and game we are saying users can only set the values for these attributes and that's it. But there's a problem now, how do we set the values for our game specific attributes?

Good thing for us, we can dynamically set attr_accessible thanks to this great RailsCast here. I'd recommend just watching the RailsCast to learn how to do this, but I want to mention that there are a couple changes that need to be made. The RailsCasts was filmed using Rails 3, but now that Rails 3.1 is out, we need to make some minor changes.
class Event
   include Mongoid::Document

   field :name, type: String
   field :start_time, type: DateTime
   field :game, type: String

   attr_accessible :name, :start_time, :game

  # :accessible is a variable used to store the game specific dynamic fields so that they can be set through mass assignment via attr_accessible. this allows the attr_accessible to be set dynamically.
  attr_accessor :accessible

  private
  # Overrided so that extra variables can be added to the attr_accessible dynamically to allow mass assignment.
  def mass_assignment_authorizer(role = :default)
    super + (accessible || [])
  end
end
We can now use the accessible variable in the EventsController to dynamically allow mass assignment of game specific attributes. The accessible variable is actually an array, and anything we put in that array will be added to attr_accessible since we overrided the mass_assignment_authorizer method.

One last detour before we get to the final step. We first need to figure out how load the YML file that contains our dynamic attributes in to memory so that we know what attributes to add to attr_accessible. I want to load the dynamic attributes in to a global variable so that we can use it during the lifetime of the server. To do this, we can create a file called event_config.rb in our Rails.root/config/initializers folder. Whenever the rails server starts up, it will automatically load all the files in the initializers folder. We can load the YML in to memory by calling YAML.load() and passing it a file.
module EventConfig
  DYNAMIC_FIELDS = YAML.load(File.open("#{Rails.root}/config/dynamic_fields.yml"))
end
Great, now we can access the DYNAMIC_FIELDS variable anywhere in our code. We have all the tools we need, so let's just jump right in to it. The DYNAMIC_FIELDS is a hash of what is defined in the YML file. So we can iterate through it by accessing the 'starcraft2' key, and then that will return another hash. Inside the 'starcraft2' hash we can acess the 'attributes' key, which will give us another hash that contains the attributes we want. Then we can just iterate through the attributes hash's keys to get the attributes we want to set to be mass assignable.
def create
   # we don't want to pass in the params right away because 
   # we have to set the accessible variable first
   @event = Event.new

   # the keys of the attribute hash are the game specific attributes 
   # we want to be assignable, so set the accessible variable to them
   game = params[:event][:game]
   @event.accessible = EventConfig::DYNAMIC_FIELDS[game]['attributes'].keys

   # lastly, set all the other attributes for the event using the params
   @event.attributes = params[:event]

   respond_to do |format|
      if @event.save
        format.html { redirect_to @event, notice: 'Event was successfully created.' }
      else
        format.html { render action: "new" }
      end
    end
end
That's it! It should work now, but how can we test this really easily? We can test it by using a tool called curl, which allows you to generate and send HTTP requests from the console. The most common use case will be a user filling out a form to create an Event on our server. Curl has a -F parameter, which we can use to emulate a filled-in form and it will send a HTTP POST request to the server.
curl -F 'event[name]=curlTest' -F 'event[game]=starcraft2' -F 'event[regions]=North America' -F 'event[leagues]=Grandmaster' 'http://localhost:3000/events'
This should successfully create a new Event object in our database! We can also test to make sure that our attr_accessible is working correctly by not letting users set attributes that they shouldn't be setting. For example, we can try this:
curl -F 'event[name]=curlTest' -F 'event[game]=starcraft2' -F 'event[regions]=North America' -F 'event[leagues]=Grandmaster' -F 'event[some_random_field]=should not get assigned' 'http://localhost:3000/events'
In the rails server log, you should see this error message, "WARNING: Can't mass-assign protected attributes: some_random_field." And we can see that it did create the Event, but without the some_random_field attribute.

That's it, you are all set and ready to go! I hope that this post was helpful in understanding how dynamic attributes can be a cool and useful feature for your projects. Good luck, and until next time.

Wednesday, February 15, 2012

Rails 3.1 testing with RSpec, Paperclip, and Mongoid

Coming in to this, I did not know much about how RSpec testing worked. I watched a couple of RailsCasts (here and here) to obtain a quick primer on the subject, but I still felt like it was not enough. So I decided to write this blog post in hopes that it will help anyone else who has had the same troubles as me. After watching the RailsCasts, the first thing I did was set up my test environment by installing the RSpec, FactoryGirl, and Guard gems. I will be writing tests for a model called Event and here is the basic structure of it:
class Event
   include Mongoid::Document
   include Mongoid::Paperclip

   field :name, type: String
   has_mongoid_attached_file :banner_image,
                 :storage => :s3,               
                 :s3_credentials => "#{Rails.root}/config/amazon_s3.yml",                
                 :s3_permissions =>'public-read',                
                 :s3_protocol => 'http',               
                 :s3_headers => {'Expires' => 1.year.from_now.httpdate},                 
                 :url => ':s3_alias_url'

   validates_presence_of :name, allow_nil: false
   validates_length_of :name, minimum: 1, maximum: 50, allow_blank: false
   validates_presence_of :start_time, allow_nil: false
end
So if you installed RSpec before creating this Event model, it should invoke an RSpec generator to automatically create the file Rails.root/spec/models/event_spec.rb, but if you installed RSpec after the fact, then just create the file manually.
require 'spec_helper'

describe Event do

end
So what are the most basic kinds of things that we can test in the model to start off? The easiest things I have found to test first are the model validations. The RSpec syntax is meant to be very human readable as you will see. The first validation to test is the mandatory presence of the name field. First we want to create a mock Event object to test with; we can do that by using FactoryGirl. To set up FactoryGirl, create a factories.rb file in the Rails.root/spec folder. This is where we will define what mock Event objects will look like, like so:
FactoryGirl.define do
   factory :event do
      name 'TestEvent'
      start_time { Time.now }
   end
end
Basically all our mock objects will have the name 'TestEvent' and a start_time of right now. There are many more advanced options and features in FactoryGirl that are available (here), but for now let's keep things simple.

Now back in the event_spec.rb file, I'll create a test called "should require a name", which will test that our validates_presence_of :name is working correctly. First, we need to use FactoryGirl to create a mock Event object. We can do this by calling FactoryGirl.build(:event). Notice that I am using the build method (creates an object, but does not save it to the database), rather than the create method (creates an object, and does save it to the database). We actually want to test that an Event object without a name should not be valid. So to create a mock Event with no name, we can actually set the Event's name by passing it in like so: FactoryGirl.build(:event, :name => nil). Putting this all together we get:
describe Event do
   it "should require a name" do
      event = FactoryGirl.build(:event, :name => nil)
      event.should_not be_valid
   end
end
The rest of the validation tests are similar and straight forward. I want to move on to the controller RSpec tests next. If this file is not already created, you should create it here Rails.root/spec/controllers/events_controller_spec.rb. Initially I had a lot of trouble trying to figure out how to write tests for the controller, but I came across a book called Rails Test Prescriptions: Keeping Your Application Healthy that was quite helpful. Now I didn't buy or read this book, but I downloaded the free sample code from the book here, and it contains some great examples of how to write simple tests for your basic CRUD controller methods. If you're interested in looking at the code, it is located in huddle3_rspec2/spec/controllers/projects_controller_spec.rb.

Let's start with a simple test of making a GET request on the index method of the EventsController. Just like in the model tests, we want to create some mock Event objects by calling FactoryGirl.create_list(:event, 10), which will create 10 mock Event objects. Now when we perform a get :index in the test, it's going to run the code that we have in the controller. Looking at the index method, we can see that in order to get all the events, it calls @events = Event.all, which is great for regular use, but not so much for testing. During testing, we would like to have a controlled environment where we can control the data being returned. We can accomplish this by something called method stubbing. Method stubbing allows you to intercept method calls within a function (so the method in question will not actually be called), and allow you to return whatever data you want. Luckily, this is very easy to do in RSpec. Here is the code:
describe EventsController do
   describe "GET index" do
      it "should assign all events as @events" do
         events = FactoryGirl.create_list(:event, 10)
         Event.stub(:all).and_return(events)

         get :index

         assigns(:events).should eq(events)
      end
   end
end
Let's walk through this code. First we create 10 events that we are going to make the Event.all method return. Then we actually stub the :all method so that it returns the events we created. Now in order to invoke the controller code, we need to send a GET request to the index method, which is what get :index does. Finally, we need to check to make sure that the @events variable (in the index method in the controller) gets correctly set to what we expect it to be, which are the Events we created with FactoryGirl. The assigns() method in RSpec allows you to access the instance variables (variables with an @ in front of it) of the controller. This test should pass because we stub out the :all method and therefore the @events variable gets set to the 10 events that we created in the beginning.

Now that we've had a crash course in RSpec, we can write a test to test if file attachments are working correctly with Paperclip. Before we start, I need to mention that Paperclip comes with RSpec matchers, which allow you test Paperclip specific attributes such as should have_attached_file(:file). This is great, but when using Mongoid, we have to use the mongoid-paperclip gem, which does not support these matchers. I did some research on Google and Stackoverflow, but I could not find a clear cut answer. So what I decided to do was just test if the object had the Paperclip fields defined and were set. When you define a Paperclip attachment in a model, it will create several fields for you. For example, if my Paperclip attachment is called :banner_image, then it will create the following fields in your model automatically: :banner_image_file_name, :banner_image_content_type, :banner_image_file_size, and :banner_image_updated_at; I wanted to test to make sure that these fields were not nil. Now you can just put these checks directly in to your test, but I wanted to be able to reuse these file checks without copying and pasting code. We can do this with something called RSpec Matchers, which allow you to define conditions that get tested after the should method. I won't go in to too much detail about them here, but you can watch this Railscasts for more information. I created a file called banner_image_matcher.rb in Rails.root/spec/support/matchers and put this inside:
# An RSpec matcher used to test the banner_image file attachment of the Event model. It tests whether the Paperclip fields for file attachments exist or not.
RSpec::Matchers.define :have_banner_image_attached_file do |expected|
   match do |actual|
      actual.banner_image_file_name != nil &&
      actual.banner_image_content_type != nil &&
      actual.banner_image_file_size != nil &&
      actual.banner_image_updated_at != nil
   end

   failure_message_for_should do |actual|
      "expected that #{actual} would have a banner_image file attachment"
   end

   failure_message_for_should_not do |actual|
      "expected that #{actual} would not have a banner_image file attachment"
   end
end
More detailed information can be found here, but basically inside the match block, you want to test your condition, and then return true or false. You can also modify the error messages that get printed when a test fails.

Now we can finally write the test. Here is what I did:
it "should assign a newly created event as @event with an image attachment" do 
   event = FactoryGirl.create(:event, :banner_image => File.new(Rails.root + 'spec/support/images/bannerTest.png'))
   Event.stub(:new).and_return(event)

   post :create, :event => event

   assigns(:event).should eq(event)
   assigns(:event).should have_banner_image_attached_file
end
Basically I do the same thing as my previous tests, but I add an extra check for the banner image using the RSpec matcher I just created. Also, I want to mention that you can provide a file to test with by reading one in using File.new. I put a test image in the Rails.root/spec/support/images folder and reference it there. This all works great, but there's on thing we're missing. If you're using Amazon S3 to upload your files to, these tests will actually upload your image every time you run your test. This is something we probably don't want to do, so we can prevent that from happening by stubbing out the method that uploads the file. That method is called save_attached_files, and we can stub it out by using the following code: Event.any_instance.stub(:save_attached_files).and_return(true). So now our test should look like this:
it "should assign a newly created event as @event with an image attachment" do 
   # stub out this method so that this test does not actually upload the image to amazon s3
   Event.any_instance.stub(:save_attached_files).and_return(true)

   event = FactoryGirl.create(:event, :banner_image => File.new(Rails.root + 'spec/support/images/bannerTest.png'))
   Event.stub(:new).and_return(event)

   post :create, :event => event

   assigns(:event).should eq(event)
   assigns(:event).should have_banner_image_attached_file
end
We can run this test (or have Guard run it for us) and see that it passes.

I hope that this post was helpful in getting started with RSpec testing, and also how to test with Paperclip and Mongoid. Feel free to ask questions in the comments and I will do by best to answer all of them.