Dynamic attributes and MongoDB/Mongoid

I recently started learning more about MongoDB, and how to use it in Rails 3.1 with the Mongoid gem. MongoDB is not your typical relational database; It is considered a NoSQL database because it stores your data in one huge JSON document instead of a bunch of relational tables. Actually, instead of storing the data in JSON, it stores it in BSON (Binary JSON) for performance reasons. The really cool thing about MongoDB is that it is schema-less, which allows you to store non-homogeneous objects in your database.

Why is this cool you might ask? Well let me explain my problem; I want to create a web service that allows CRUD operations on objects called Events. Events have a name, start_time, and most importantly a game. Depending on the game attribute, each Event will have different game specific attributes. For example, I have an Event with a game value of 'starcraft2'. Only Starcraft 2 Events have a region and a league attribute, while League of Legends Events have a min_elo attribute. How can we store these Event objects that have different attributes in our one document? We could do this in a standard MySQL database, but our table would be very wide, meaning we would have to have a column for every game specific attribute. This isn't very ideal because for each row, there would be a bunch of columns that wouldn't be used depending on the game.

The great thing about Mongoid is that it supports dynamic attributes. I quote this from their website, "By default Mongoid supports dynamic fields - that is it will allow attributes to get set and persisted on the document even if a field was not defined for them." This sounds like exactly what we need. We can just define the common Event attributes in the model itself, and then specify the game specific attributes at run time. So how do we go about doing this? Well, let's start with the Event model:

class Event
   include Mongoid::Document

   field :name, type: String
   field :start_time, type: DateTime
   field :game, type: String
end

This is pretty straight forward. Now the way I chose to define game specific attributes is by defining them in a YML file. I created a file called dynamic_fields.yml and put it in the Rails.root/config folder. Here is how I defined it:

starcraft2:
   attributes:
      regions:
         name: regions
      leagues:
         name: leagues

You can define your dynamic attributes however you want, but this is just how I decided to do mine. So now that we have our Event model, and our Starcraft 2 attributes, how do we set these when an Event gets created, and how do we make sure these attributes only get set if the game is Starcraft 2? First we need to understand what mass assignment is, and how it could be a security vulnerability. Here is an excellent RailsCasts on that topic, but the gist of it is that when the server receives a POST request to create a new Event, it will take the parameters it receives and mass assign themto the matching attributes of the Event model. For example, if we had a User model, and a is_admin attribute, the user shouldn't be able to set the value of the is_admin attribute. If they could, then everyone could make themselves an admin when they create a new account. So how do we prevent this? In Rails there is a very useful method called attr_accessible that we can set on attributes in our model. Only the attributes you set using attr_accessible will be available for mass assignment. So let's do that for our Event model for all the common fields:

class Event
   include Mongoid::Document

   field :name, type: String
   field :start_time, type: DateTime
   field :game, type: String

   attr_accessible :name, :start_time, :game
end

This reason this is so important, especially in Mongoid is that since we enabled dynamic attributes, a user can create an Event with any additional attributes they want. They can add an attribute called "some_random_field" by including it in the POST parameters and it would be inserted in our database no questions asked. By setting attr_accessible on name, start_time, and game we are saying users can only set the values for these attributes and that's it. But there's a problem now, how do we set the values for our game specific attributes?

Good thing for us, we can dynamically set attr_accessible thanks to this great RailsCast here. I'd recommend just watching the RailsCast to learn how to do this, but I want to mention that there are a couple changes that need to be made. The RailsCasts was filmed using Rails 3, but now that Rails 3.1 is out, we need to make some minor changes.

class Event
   include Mongoid::Document

   field :name, type: String
   field :start_time, type: DateTime
   field :game, type: String

   attr_accessible :name, :start_time, :game

  # :accessible is a variable used to store the game specific dynamic fields so that they can be set through mass assignment via attr_accessible. this allows the attr_accessible to be set dynamically.
  attr_accessor :accessible

  private
  # Overrided so that extra variables can be added to the attr_accessible dynamically to allow mass assignment.
  def mass_assignment_authorizer(role = :default)
    super + (accessible || [])
  end
end

We can now use the accessible variable in the EventsController to dynamically allow mass assignment of game specific attributes. The accessible variable is actually an array, and anything we put in that array will be added to attr_accessible since we overrided the mass_assignment_authorizer method.

One last detour before we get to the final step. We first need to figure out how load the YML file that contains our dynamic attributes in to memory so that we know what attributes to add to attr_accessible. I want to load the dynamic attributes in to a global variable so that we can use it during the lifetime of the server. To do this, we can create a file called event_config.rb in our Rails.root/config/initializers folder. Whenever the rails server starts up, it will automatically load all the files in the initializers folder. We can load the YML in to memory by calling YAML.load() and passing it a file.

module EventConfig
  DYNAMIC_FIELDS = YAML.load(File.open("#{Rails.root}/config/dynamic_fields.yml"))
end

Great, now we can access the DYNAMIC_FIELDS variable anywhere in our code. We have all the tools we need, so let's just jump right in to it. The DYNAMIC_FIELDS is a hash of what is defined in the YML file. So we can iterate through it by accessing the 'starcraft2' key, and then that will return another hash. Inside the 'starcraft2' hash we can acess the 'attributes' key, which will give us another hash that contains the attributes we want. Then we can just iterate through the attributes hash's keys to get the attributes we want to set to be mass assignable.

def create
   # we don't want to pass in the params right away because 
   # we have to set the accessible variable first
   @event = Event.new

   # the keys of the attribute hash are the game specific attributes 
   # we want to be assignable, so set the accessible variable to them
   game = params[:event][:game]
   @event.accessible = EventConfig::DYNAMIC_FIELDS[game]['attributes'].keys

   # lastly, set all the other attributes for the event using the params
   @event.attributes = params[:event]

   respond_to do |format|
      if @event.save
        format.html { redirect_to @event, notice: 'Event was successfully created.' }
      else
        format.html { render action: "new" }
      end
    end
end

That's it! It should work now, but how can we test this really easily? We can test it by using a tool called curl, which allows you to generate and send HTTP requests from the console. The most common use case will be a user filling out a form to create an Event on our server. Curl has a -F parameter, which we can use to emulate a filled-in form and it will send a HTTP POST request to the server.

curl -F 'event[name]=curlTest' -F 'event[game]=starcraft2' -F 'event[regions]=North America' -F 'event[leagues]=Grandmaster' 'http://localhost:3000/events'

This should successfully create a new Event object in our database! We can also test to make sure that our attr_accessible is working correctly by not letting users set attributes that they shouldn't be setting. For example, we can try this:

curl -F 'event[name]=curlTest' -F 'event[game]=starcraft2' -F 'event[regions]=North America' -F 'event[leagues]=Grandmaster' -F 'event[some_random_field]=should not get assigned' 'http://localhost:3000/events'

In the rails server log, you should see this error message, "WARNING: Can't mass-assign protected attributes: some_random_field." And we can see that it did create the Event, but without the some_random_field attribute.

That's it, you are all set and ready to go! I hope that this post was helpful in understanding how dynamic attributes can be a cool and useful feature for your projects. Good luck, and until next time.

Rails 3.1 testing with RSpec, Paperclip, and Mongoid

Coming in to this, I did not know much about how RSpec testing worked. I watched a couple of RailsCasts (here and here) to obtain a quick primer on the subject, but I still felt like it was not enough. So I decided to write this blog post in hopes that it will help anyone else who has had the same troubles as me. After watching the RailsCasts, the first thing I did was set up my test environment by installing the RSpec, FactoryGirl, and Guard gems. I will be writing tests for a model called Event and here is the basic structure of it:

class Event
   include Mongoid::Document
   include Mongoid::Paperclip

   field :name, type: String
   has_mongoid_attached_file :banner_image,
                 :storage => :s3,               
                 :s3_credentials => "#{Rails.root}/config/amazon_s3.yml",                
                 :s3_permissions =>'public-read',                
                 :s3_protocol => 'http',               
                 :s3_headers => {'Expires' => 1.year.from_now.httpdate},                 
                 :url => ':s3_alias_url'

   validates_presence_of :name, allow_nil: false
   validates_length_of :name, minimum: 1, maximum: 50, allow_blank: false
   validates_presence_of :start_time, allow_nil: false
end

So if you installed RSpec before creating this Event model, it should invoke an RSpec generator to automatically create the file Rails.root/spec/models/event_spec.rb, but if you installed RSpec after the fact, then just create the file manually.

require 'spec_helper'

describe Event do

end

So what are the most basic kinds of things that we can test in the model to start off? The easiest things I have found to test first are the model validations. The RSpec syntax is meant to be very human readable as you will see. The first validation to test is the mandatory presence of the name field. First we want to create a mock Event object to test with; we can do that by using FactoryGirl. To set up FactoryGirl, create a factories.rb file in the Rails.root/spec folder. This is where we will define what mock Event objects will look like, like so:

FactoryGirl.define do
   factory :event do
      name 'TestEvent'
      start_time { Time.now }
   end
end

Basically all our mock objects will have the name 'TestEvent' and a start_time of right now. There are many more advanced options and features in FactoryGirl that are available (here), but for now let's keep things simple.

Now back in the event_spec.rb file, I'll create a test called "should require a name", which will test that our validates_presence_of :name is working correctly. First, we need to use FactoryGirl to create a mock Event object. We can do this by calling FactoryGirl.build(:event). Notice that I am using the build method (creates an object, but does not save it to the database), rather than the create method (creates an object, and does save it to the database). We actually want to test that an Event object without a name should not be valid. So to create a mock Event with no name, we can actually set the Event's name by passing it in like so: FactoryGirl.build(:event, :name => nil). Putting this all together we get:

describe Event do
   it "should require a name" do
      event = FactoryGirl.build(:event, :name => nil)
      event.should_not be_valid
   end
end

The rest of the validation tests are similar and straight forward. I want to move on to the controller RSpec tests next. If this file is not already created, you should create it here Rails.root/spec/controllers/events_controller_spec.rb. Initially I had a lot of trouble trying to figure out how to write tests for the controller, but I came across a book called Rails Test Prescriptions: Keeping Your Application Healthy that was quite helpful. Now I didn't buy or read this book, but I downloaded the free sample code from the book here, and it contains some great examples of how to write simple tests for your basic CRUD controller methods. If you're interested in looking at the code, it is located in huddle3_rspec2/spec/controllers/projects_controller_spec.rb.

Let's start with a simple test of making a GET request on the index method of the EventsController. Just like in the model tests, we want to create some mock Event objects by calling FactoryGirl.create_list(:event, 10), which will create 10 mock Event objects. Now when we perform a get :index in the test, it's going to run the code that we have in the controller. Looking at the index method, we can see that in order to get all the events, it calls @events = Event.all, which is great for regular use, but not so much for testing. During testing, we would like to have a controlled environment where we can control the data being returned. We can accomplish this by something called method stubbing. Method stubbing allows you to intercept method calls within a function (so the method in question will not actually be called), and allow you to return whatever data you want. Luckily, this is very easy to do in RSpec. Here is the code:

describe EventsController do
   describe "GET index" do
      it "should assign all events as @events" do
         events = FactoryGirl.create_list(:event, 10)
         Event.stub(:all).and_return(events)

         get :index

         assigns(:events).should eq(events)
      end
   end
end

Let's walk through this code. First we create 10 events that we are going to make the Event.all method return. Then we actually stub the :all method so that it returns the events we created. Now in order to invoke the controller code, we need to send a GET request to the index method, which is what get :index does. Finally, we need to check to make sure that the @events variable (in the index method in the controller) gets correctly set to what we expect it to be, which are the Events we created with FactoryGirl. The assigns() method in RSpec allows you to access the instance variables (variables with an @ in front of it) of the controller. This test should pass because we stub out the :all method and therefore the @events variable gets set to the 10 events that we created in the beginning.

Now that we've had a crash course in RSpec, we can write a test to test if file attachments are working correctly with Paperclip. Before we start, I need to mention that Paperclip comes with RSpec matchers, which allow you test Paperclip specific attributes such as should have_attached_file(:file). This is great, but when using Mongoid, we have to use the mongoid-paperclip gem, which does not support these matchers. I did some research on Google and Stackoverflow, but I could not find a clear cut answer. So what I decided to do was just test if the object had the Paperclip fields defined and were set. When you define a Paperclip attachment in a model, it will create several fields for you. For example, if my Paperclip attachment is called :banner_image, then it will create the following fields in your model automatically: :banner_image_file_name, :banner_image_content_type, :banner_image_file_size, and :banner_image_updated_at; I wanted to test to make sure that these fields were not nil. Now you can just put these checks directly in to your test, but I wanted to be able to reuse these file checks without copying and pasting code. We can do this with something called RSpec Matchers, which allow you to define conditions that get tested after the should method. I won't go in to too much detail about them here, but you can watch this Railscasts for more information. I created a file called banner_image_matcher.rb in Rails.root/spec/support/matchers and put this inside:

# An RSpec matcher used to test the banner_image file attachment of the Event model. It tests whether the Paperclip fields for file attachments exist or not.
RSpec::Matchers.define :have_banner_image_attached_file do |expected|
   match do |actual|
      actual.banner_image_file_name != nil &&
      actual.banner_image_content_type != nil &&
      actual.banner_image_file_size != nil &&
      actual.banner_image_updated_at != nil
   end

   failure_message_for_should do |actual|
      "expected that #{actual} would have a banner_image file attachment"
   end

   failure_message_for_should_not do |actual|
      "expected that #{actual} would not have a banner_image file attachment"
   end
end

More detailed information can be found here, but basically inside the match block, you want to test your condition, and then return true or false. You can also modify the error messages that get printed when a test fails.

Now we can finally write the test. Here is what I did:

it "should assign a newly created event as @event with an image attachment" do 
   event = FactoryGirl.create(:event, :banner_image => File.new(Rails.root + 'spec/support/images/bannerTest.png'))
   Event.stub(:new).and_return(event)

   post :create, :event => event

   assigns(:event).should eq(event)
   assigns(:event).should have_banner_image_attached_file
end

Basically I do the same thing as my previous tests, but I add an extra check for the banner image using the RSpec matcher I just created. Also, I want to mention that you can provide a file to test with by reading one in using File.new. I put a test image in the Rails.root/spec/support/images folder and reference it there. This all works great, but there's on thing we're missing. If you're using Amazon S3 to upload your files to, these tests will actually upload your image every time you run your test. This is something we probably don't want to do, so we can prevent that from happening by stubbing out the method that uploads the file. That method is called save_attached_files, and we can stub it out by using the following code: Event.any_instance.stub(:save_attached_files).and_return(true). So now our test should look like this:

it "should assign a newly created event as @event with an image attachment" do 
   # stub out this method so that this test does not actually upload the image to amazon s3
   Event.any_instance.stub(:save_attached_files).and_return(true)

   event = FactoryGirl.create(:event, :banner_image => File.new(Rails.root + 'spec/support/images/bannerTest.png'))
   Event.stub(:new).and_return(event)

   post :create, :event => event

   assigns(:event).should eq(event)
   assigns(:event).should have_banner_image_attached_file
end

We can run this test (or have Guard run it for us) and see that it passes.

I hope that this post was helpful in getting started with RSpec testing, and also how to test with Paperclip and Mongoid. Feel free to ask questions in the comments and I will do by best to answer all of them.

Getting mysql2 gem to work with Ruby on Rails 3.0 and Windows 7 64bit

Edit: For Rails 3.1, these same steps should still work. I had some issues at first, but try uninstalling (gem uninstall mysql2 -a) all previous instances of the mysql2 gem before installing the latest one.

The mysql2 gem recently added support for Windows (https://github.com/brianmario/mysql2/issues/8), but it sure was a pain in the ass to install. Hopefully this blog post will help save you the time and swearing that this gem has caused me. I had a lot of issues just trying to get the gem to talk with my MySQL server, and it seemed like there was some issues with using the 64bit version of MySQL server, so I eventually downgraded to the 32bit version. Also, you cannot just do a bundle install or gem install mysql2 to get this gem installed. Lastly, once I finally got the dam gem installed, it would always hang when I tried to use it (e.g. rake db:create), and I found that there's an issue on github here: https://github.com/brianmario/mysql2/issues/142, but I was able to still make it work.

First off, here are the versions of my software:

• Ruby 1.9.2p180 (2011-02-18) [i386-mingw32] installed from RubyInstaller
• RubyInstaller's DevKit: DevKit-tdm-32-4.5.1-20101214-1400-sfx.exe
• Rails 3.0.7 or 3.0.8RC2
• MySQL Server 5.5 32 bit (It's really important that you install the 32 bit version of MySQL because the 64 bit version doesn't play nice with RubyInstaller. Refer to here for more information: https://github.com/oneclick/rubyinstaller/wiki/Troubleshooting#gems_mix_bits

Assuming you have all this already installed, here are the steps you need to follow:

1. Copy C:\Program Files (x86)\MySQL\MySQL Server 5.5\lib\libmysql.dll to C:\Ruby192\bin. I think also if you just add C:\Program Files (x86)\MySQL\MySQL Server 5.5\lib\ to your environment variables it might work too.
2. Uninstall all previous version of the mysql2 gem just in case. e.g. gem uninstall mysql2 -a
3. Go to the mysql2 github page and select the 0.2.x branch, then click the Downloads button to download the gem (or you can go here: https://github.com/brianmario/mysql2/zipball/0.2.x). We're going to have to manually compile and install the gem. The latest mysql2 version 0.3.x only works with Rails 3.1 and not 3.0.
4. Unzip in to a folder called mysql2-0.2.7 in your Ruby installation folder here: C:\Ruby192\lib\ruby\gems\1.9.1\gems, so that afterwards it looks like: C:\Ruby192\lib\ruby\gems\1.9.1\gems\mysql2-0.2.7
5. Now load up Git Bash (I couldn't get this to work in regular Command Prompt) and go to C:\Ruby192\lib\ruby\gems\1.9.1\gems\mysql2-0.2.7 and type: gem build mysql2.gemspec
6. Now you should have a file called mysql2-0.2.7.gem in your directory. Let's install the gem now. First type: subst X: "C:\Program Files (x86)\MySQL\MySQL Server 5.5" (or wherever your MySQL Server is installed).
7. Then type: gem install mysql2-0.2.7.gem --platform=ruby -- --with-mysql-dir=X: --with-mysql-lib=X:\lib
8. Congrats, you're done now! We can test it by typing rake db:create or rake db:migrate in one of your projects. If all went well you shouldn't have gotten any errors. Good luck!