Tuesday 18 December 2007

Rails gotchas: All the fixtures in the world

After having spent three hours debugging an error that "couldn't be happening" I have decided to just start loading every fixture for every functional test... Ok maybe not - but that's what it feels like is necessary.

The test in question was there to check if a user with expired quota could start using a feature again once they'd been given more quota. It was failing because I hadn't included the "roles" fixture... vital to the controller when deciding if the user is authorised to get into the controller in the first place... but not really directly relevant to this test, so I hadn't thought about it.

This can also cause the weird effect where an entire functional test suite runs fine when you call it via ruby test/functionals/<whatever>.rb, but several tests fail when you try to run it via rake test:functionals

Lesson: include every fixture that might be accessed by your controller in the performance of its duty - even if it doesn't seem directly relevant.

Friday 14 December 2007

The Economics of IT salaries

I have been reading a book called "The undercover economist"[1], which mentioned the problems with "asymmetric information" in negotiations. It's discussed on pp112-115, if you want to look, but to summarise, the problem is thus: A 2nd-hand car salesman is trying to sell all their cars - whether they are peaches or lemons, and wants to get the best price for each. The buyer is not willing to pay as much for a peach as for a lemon. The salesman knows if any given car is a peach or a lemon, but the car buyer does not. This is the "asymmetric information".

The salesman will, of course, say "all my cars are peaches" (especially if they are not) and to try to charge accordingly. The buyer must then decide if it worth the risk of potentially paying peach-prices for a lemon. If there is no accurate way of telling one car from another, they will often choose not to buy at all.

The problem is that even if the car really *is* a peach - the customer will still not be able to believe the salesman... after all, he'd say that if it was a lemon too.

In this case, both the salesman and the customer lose out, because the salesman is in command of information that the customer is not (ie the "peachness" of the car in question).

The only way out of it is for the salesman to offer undeniable proof of a car's "peachness". The customer can then trust that the car is worth the price - and is more willing to buy.

In Australia, this is done with independant inspections, (eg done by the NRMA). A customer can be fairly certain that a car that has passed inspection meets a basic level of quality. Doing these tests takes more time that the average car-buyer has (especially when they have so many to look over), but it's worth the time for the salesman to get it done, as they then have undeniable proof of peachiness.

But it's relatively easy to verify the quality of car: does it, or does it not have working brakes? Well, test the brakes and see if they work!

What if it weren't so easy - with a set of skills that are subjective, and tests that are at best indirect, and at worst misleading. For example, how do we tell if one IT-worker is more skilled than another? How do you measure skill in IT? and who's doing the measuring? In fact, who is the buyer in this kind of transaction?

Measuring your l33t sk1llz

In the first place, we all know that it's pretty hard to tell how good one programmer is from another. Sure, it's sometimes pretty easy to assess ourselves relative to one another. Obviously David Hanmeier Hansson is easily more skilled than my kid sister. He's certainly better than I am... but am I better than my colleague in the next cubicle over?

I like to think so and I'm pretty sure that my paycheck says so too... but I'm only guessing that... and is a paycheck a reasonable guide anyway - or is that just a cue for a feedback loop?

A year ago I was working for considerably below the average salary[2]. Partly it was because I valued job-security (and thought I had it[3]) but I've been trying to figure out the other part. Why is it that companies with certain types of management don't offer more to their IT workers? Contrariwise - why is it that some companies hire idiots and pay them huge sums of money?

There was a Paul Graham article that explains that the average PHB can't tell if a technology is really good or not. They simply don't have the requisite experience to make an educated distinction.

By corollary, they also can't tell if the technology skill of their workers is really good (after all, if they don't know how to use it, how can they tell if *you're* using it to the best advantage?). The only clues they have to go on are the skills that they can see and judge from their own experience. These tend to be more socially oriented: is the worker well-dressed, articulate yet polite, well-behaved, do they show due deference to the PHB, do as they're told etc. The average PHB can measure these hirself, so they get put to the top of the list of required "professional" skills.

The skills mentioned above may well be a good indicator of whether or not they'll get along well with the manager, but they are clearly not a good indicator of skill in IT.

Now, I'm not saying these skills aren't important. Doubtless, most of these have their importance in a workplace, especially where one must get along with peers. Without any level of skill in the above list, friction will eventually ensue... and yet the original point here was not to tell how good an IT worker would be at making friends in the workplace, but how good they'd be at being an IT worker... and surely actual IT skill should rank up there as an important part of the job requirements.

Now, a good, skilled techie may know reasonably well how they stack up compared to other techies (s)he's worked with - at least within a few degrees of freedom). They tend to know that if they hand a tough task to Jo she'll get the job done quickly and elegantly, but hand it to Sarah and she'll need a fair bit of hand-holding so you'd better give her the easier tasks while you get on with the "real work".

A less-skilled worker, by contrast, may not have a clue how good they are, and you can't compare at all if you've never worked with a person[4]

So maybe they can tell, roughly, by contrast, their own level of skill - maybe it's not accurate, but they've probably got some idea of where they think they stand. In any case they have information that the PHB doesn't have... and, come review time, every worker will say they are a peach - especially if they're a lemon.

And those that have better social skills will be more convincing to the PHB.

What does the PHB think?

Now, a manager that was actually skilled in IT (eg one that had "come up through the ranks") might have a chance to judge accurately. This neatly slams up against another perpetual problem of the IT trade: that if you promote your best IT workers into management - you are wasting the IT skills that made them good in the first place. Not to mention the fact that to be a good manager requires a completely orthogonal skillset to being a good IT worker.[5].

Of course, if promotion is based on IT skill - you're in a chicken-egg situation. How do you judge good skill to promote if you cannot trust the skill-judgements of those around you? Paul Graham calls this "the design problem".

What does this mean for an organisation?

In my experience, the problems from asymmetric information are amplified in monolithic IT organisations (eg the IT departments of banks), which nurture this kind of manager - and also seem to nurture a higher percentage of mediocre programmers.

If a manager cannot tell if their workers are skilled or not - they aren't prepared to pay as well - after all, the worker might really be a lemon, just saying they are a peach. But a real peach may know that they are - and thus won't be prepared to take the low pay... so they have no choice but to leave. Everyone loses.

This is a self-perpetuating cycle. I don't know how it all started, but you can see that it'd be nigh-on impossible to break out of. A mediocre manager would not be prepared to offer competitive pay to his employees - as he's simply not prepared to pay peach prices on the offchance of buying a lemon. This causes all the actual peach epmployees to leave - they know they can get better pay elsewhere... leaving only the mediocre programmers (and lemons) behind in the monolithic organisation.

Which leaves only mediocre pickings for employees to rise through the ranks to become mediocre managers themselves one day... or the hiring of managers, but without the benefit of a sanity check on IT skills from the employees. In any case leading to generally the same cycle - bad managers, bad employees - lemons all round.

Thus the monolithic organisations fail to get or keep good staff, and the only place to find them is in the small, agile development houses, or in contracting - where a peach can have a chance of negotiating more directly with clueful people.

Which matches what we see around us every day...

Is there any way out of this for large organisations?

Not really sure it's possible. I know some do - but my gut feel tells me this only occurs by chance. If they happen to stumble across a peach by accident, or if one decides to take pity on an organisation and work at making it a better place. But it certainly doesn't seem to be the norm.

The only cases I've really known are where peaches don't know their own worth and stay on, thinking they are worth less than they are. In a mediocre organisation this might last for a while - as their worth is determined by people who don't know how to accurately judge worth. But eventually a true peach will learn about themselves and leave - usually by going outside of their colleague-group and seeing what other people are acheiving in their field - ie by getting a better frame of reference.

Any opinions?

Notes

  • [1]By Tim Harford - pretty good read, actually.
  • [2]for a programmer with my experience - as measured by two different salary surveys. Note: for australia many IT managers use the Hays salary survey which is great... except that it leaves job titles somewhat vague - and several of them don't specifically have a "years experience" associated with them (especially the oft-misunderstood term "junior developer"). When going for your pay review, I recommend searching for a variety of surveys - and make sure they have years on them.
  • [3]...an entirely different story that I won't go into here.
  • [4] Paul Graham wrote an article about Great Hackers which also discusses how it's hard for hackers to judge the abilities of other hackers without having actually worked with them.
  • [5]It is not impossible for a worker to have both sets. It is more rare, however, as a skillset takes time and effort to develop... and your IT workers are generally spending more time developing their IT skills than their social ones.

Wednesday 12 December 2007

Rails 2.0

Ok, so Rails 2.0 is out - in case you somehow missed the wild whooping sounds from the Rails community.

At the following site is a comprehensive list of changes in Rails 2.0. Worth studying a few times.

Tuesday 11 December 2007

Rails gotchas: Update attribute... doesn't

Ok, it does... but it doesn't *just* update the attribute - it resaves the whole record. This is a problem if you aren't sure whether or not your record is fresh as you can overwrite other changes with your stale data.

Lesson:

As nasty as it is, you should do a do a self.reload before you do an update_attribute - if it's possible that your record has changed under you (eg by a called sub-method)

Thursday 6 December 2007

Rails Gotchas: fixtures created_at when?

So - I updated some date-sensitive fixtures and re-ran all my rake tests... and suddenly everything went funny.

First off the rack: empty created_at date

The first problem seemed to be that the created_at date just didn't get set at all. I even printed it out in my test case with an "inspect" and it was showing: "created_at" => "0000-00-00 00:00:00". Pretty nasty.

I checked the fixture and even printed out the date I was trying to save (Time.now.utc) to make sure that Time was working in the fixtures... to no avail, until I realised I'd accidentally knocked off the .to_s(:db) from the end. I added that back and magickally my fixtures were loading the dates again (yay).

Next up: intermittent wierdness

So then there was an intermittant, but persistent error that was really weird. The method I was testing boils down to me pulling out a set of objects and ordering them by creation-date and checking the value of the last one in line. I was checking that if I created a new one - I actually got the new value out the other end. Now the problem was - sometimes I did... and sometimes I didn't... sometimes I'd run rake-test and it'd work... and I'd run it straight afterward and it'd break again.

Then I remembered that datetimes only have a one-second granularity... and some of my tests probably ran less than one second after setup. So I used 1.second.ago.utc.to_s(:db) in the fixtures and it all went away. :P

Friday 30 November 2007

Pick your layer

I've been noticing that some of our newer Rails developers are having trouble picking which layer (Model, View or Controller) a method belongs in. Here are some heuristics I've picked up along the way - in no particular order.

1 - Don't put it into a library unless it really has nothing to do with the rest of your site (or you want to package it up to hand to other people).

I've recently noticed a few methods being added to a library I wrote that like this: my_method(@my_object) This is a prime candidate for going into the MyObject model itself. Especially when the body of said method just accesses some related objects eg:

# instead of:
module MyLibrary
  def my_method(my_obj)
    my_obj.sub_classes.map {|sc| [sc.name, sc.id] }
  end
end
# put it on the model
class MyModel < ActiveRecord::Base
  has_many :sub_classes
  def my_method
    return [] if sub_classes.blank?
    sub_classes.map {|sc| [sc.name, sc.id] }
  end
end

2 - A model should know how to deal with itself

I'd like to say that everything that can go into a model should... but that's not quite true. Anything data-related, however, is a prime candidate. A model should know how to deal with it's own data. It should know how to calcuate and derive from its own data. It should know how to pull stuff out of related objects, and to parse other objects into a form that allows it to stuff data into itself.

I've seen too many of these sorts of calculations going into views or helpers, or getting stuck in the middle of controller-code. These will only come back to bite you later, when the client suddenly realises they want the report to also be generated into pdf or graphical form as well...

If it can go on the model, put it there, because you'll always have access to the methods on your model.

3 - Don't put html into your model

By all means put display-names and other, similar methods that produce displayable information... but don't put html into it. After all, right now you may be concentrating on web-only delivery of your data... but who's to say the client won't be asking for xml next week... and pdf the week after that... followed by CSV?

Make your model-based display methods text-only eg:

class MyModel < ActiveRecord::Base
  def good_display_name
    name || "Unspecified"
  end
  def bad_display_name
    "<div class=\"left_aligned_red_box\">#{good_display_name}</div>"
  end
end

4 - Put format-specific display functions in the helpers

If you're displaying a selection of a certain class of objects (eg Dates) in a specific format in multiple places across the site, and want consistency of appearance (a Good Thing), feel free to add a display helper to the ApplicationHelper file, or even update the generic display function for that class in your environment.rb

  # eg add these to application helper
  def display_percentage(num)
    return "" unless num
    "#{number_with_precision(num,2)}%"
  end
  def display_date(date)
    return "" unless date
    date.strftime("%a %d/%m/%Y")
  end
  def display_datetime(datetime)
    return "" unless datetime
    "#{display_date(datetime)} #{datetime.strftime("%H:%M")}"
  end

Note that you can over-ride the default datetime display in rails using the following (in environment.rb or similar) then use "to_s" or "to_s(:datetime)" in your views.

ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.merge!(
   :default => "%a %d/%m/%Y",
   :datetime  => "%a %d/%m/%Y %H:%M"
 )

Monday 26 November 2007

Rails gotchas: ruby-enhanced yaml != rails-enhanced yaml

I you've just added some funky ruby-generated something into your yaml fixture and are suddenly getting this error message:

Fixture::FormatError: a YAML error occurred parsing /myproject/config/../test/fixtures/widgets.yml. Please note that YAML must be consistently indented using spaces. Tabs are not allowed. Please have a look at http://www.yaml.org/faq.html
The exact error was:
  SyntaxError: compile error
(erb):8: syntax error, unexpected ')'
_erbout.concat "  created_at: "; _erbout.concat(( '1 January 2006'.to_date.to_s(:db) -).to_s); _erbout.concat "\n"
                                                                                       ^

Check to make sure you haven't done the "good" thing and used -%>. It appears that YAML files don't like the whitespace-eating ending... you can only use: %>

Monday 19 November 2007

Rails gotchas: Kamikaze data migrations

This is a pernicious little gotcha I've encountered a couple of times, generally by using somebody else's plugins. The problem with it is that it usually doesn't actually affect the developer - just anybody else that uses the system. This can lead to the developer refusing to believe that there is a problem.[1]

So what am I talking about here?

I'm talking about a certain type of data migration that explodes as you run it, sometimes even causing the database to get stuck in a nasty halfway-inbetween state where you can neither continue migrating up... or roll back to a previous stable state.

"But migrations are transactional" I hear you cry. Sure they are. That doesn't stop this particular error from occurring... and *still* leaving the db in a nasty stuck state - honest, just gimme a chance to explain why.

Picture this: You've been happily developing away building the next generation of your category-killer Web 2.0 Widget application when you suddenly realise that your widgets need to refer to pieces instead of parts. So, like a good agile developer you:

  • Create a new Piece resource with all the bells and whistles.
  • Create and run the migration that a) creates the pieces table b) data migrates widgets so they point at pieces instead of parts c) drops the parts table.
  • Update all references in your code from pieces to parts.
  • Remove all the part code from controllers/views etc.

All goes smoothly. rake test runs flawlessly and you check in your code. You continue on your merry way, adding more functionality and reflecting on your brilliance...

...until a user of your code emails you to say that your migration breaks and has left their db in some wierd state that doesn't let them migrate up or down.

The problem is that when you (the developer) ran the migration, you still had the Part class in app/models and the Widget class still had references like "has_one :part" in them. When your user downloads your system, however, it may be several changes later... and the system has no recollection of a Part class or how they relate to Widgets. So when you have a data migration such as:

   Widget.find_all.each { |w| make_piece_from_part(w.part) }

It won't know what a part is - or how you get one off your widget class. The nasty part is that often you will find this kind of data migration stuffed in the middle of the schema migrations thus:

def self.up
  create_table :pieces do |t|
     #... creates a table here
  end
  add_column :widgets, :piece_id, :integer, :default => nil
  Widget.find_all.each { |w| make_piece_from_part(w.part) }
  remove_column :widgets, :part_id
end
def self.up
  add_column :widgets, :part_id, :integer, :default => nil
  Widget.find_all.each { |w| make_part_from_piece(w.piece) }
  remove_column :widgets, :piece_id
  drop_table :pieces
end

I'm told that migrations are transactional - ie if it fails, then everything rolls back to the state before the migration... this doesn't seem to work when data migrations are involved. If the data part of the migration falls over, it stops halfway through the migration with a nasty backtrace, but doesn't perform any rollback. So at that point it's already added the pieces table and the piece_id column to the widgets table, but it hasn't removed the part_id column and the schema is still pointing at the migration number before this one. So if you try to run the migration again, it will fail on the first line, saying something like:

== MovePartsToPieces: migrating ============================================
-- create_table(:pieces)
rake aborted!
Mysql::Error: Table 'pieces' already exists: CREATE TABLE pieces (<piece-table insert statement here>) ENGINE=InnoDB

(See full trace by running task with --trace)

Unfortunately it also won't migrate you back. The bad data migration failed before it had a chance to update the current migration level - so migrating back gives you entirely unhelpful "finished in 0.000183" seconds message (or similar), without doing anything at all.

Recovering from the stuck state

Recovering is possible, but it takes some small amount of hacking in the migration. Basically you need to force Rails to run an empty migration so that it does nothing except to put the version number up. Then you can force it to run the bits of the down migration that cancel the bits that it did the first time you tried running the up migration. In detail:

  1. Comment out every line of code in self.up
  2. Comment out the appropriate lines in self.down that correspond with stuff that never got run when the migration fell over the first time - ie leave anything that cancels out actions that were *successfully* done when you did the up migration. in the example above, you'd comment out the downward data migration and the addition of the piece_id column - leaving only the removal of the part_id column and the drop table - which would leave you back in the state you started in.
  3. Save the file
  4. Now you can run rake db:migrate. This should should leave you with a database that at least has the right version number for you to be able to migrate back down again.
  5. Now run the down migration with rake db:migrate VERSION=<one before this one>.
  6. Now go fix your migration so this doesn't happen to anybody else.

Fixing this kind of data-migration nightmare is the only time that I recommend you go back and edit old migrations rather than creating a new one.

  • [1]Ok, so the real reason I'm writing this post is partly so that I can point said unbelievers at it the next time they swear that nothing's wrong and that I should just quit fussing about.

Wednesday 14 November 2007

Securing Rails

Quick post today. Just found a really great Rails Security Cheatsheet here. Has links to useful articles on all sorts of topics about rails security, including (but not limited to):

  • Why do security for Rails
  • Securing your sessions
  • Useful security plugins
  • Securing files on your server
  • Common web security issues (XSS, SQL injection etc)

Wednesday 24 October 2007

Keeping passwords out of your logs

By default, rails will insert all parameters into the logs. This is useful for debugging, but not so grand when the parameters are sensitive eg:
Parameters: {"user"=>{"login => "mylogin", "answer"=>"my answer", "question"=>"my question ", "terms"=>"1", "new_password"=>"mypass123", "new_password_confirmation"=>"mypass123"}, commit => "Activate my account", "_method"=>"put"} ...
OR
Parameters: {"commit"=>"Log in", "action"=>"create", "controller"=>"sessions", "password"=>"mypass123", "login"=>"mylogin"} ...

It's a one-liner for rails to filter this out. Just add this line to the top of the relevant controller(s):

class SessionsController < ApplicationController
  # filter out sensitive fields from the log
  filter_parameter_logging 'password'
  # rest of controller goes here...
end
class UsersController < ApplicationController
  # filter out sensitive fields from the log
  filter_parameter_logging 'password', 'question', 'answer', 'given_answer'
  # rest of controller goes here...
end

This will turn the required log lines into:
Parameters: {"user"=>{"login => "mylogin", "answer"=>"[FILTERED]", "question"=>"[FILTERED] ", "terms"=>"1", "new_password"=>"[FILTERED]", "new_password_confirmation"=>"[FILTERED]"}, commit => "Activate my account", "_method"=>"put"} ...
OR
Parameters: {"commit"=>"Log in", "action"=>"create", "controller"=>"sessions", "password"=>"[FILTERED]", "login"=>"mylogin"} ...

Note that it's even smart enough to automatically filter out the "password_confirmation" field without requiring a specific reference to it.

Tuesday 23 October 2007

Extending ActiveRecord::Validations for positive numbers

I noticed I could DRY up a lot of my def validate items by finding a better way to check if a numerical field was a positive number (when provided). I figured this would be a perfect candidate for extending the "validates_numericality_of" method. But then I had a play with extending it and got stuck on "how exactly do you extend a class method?"

I've successfully added new validations by reopening ActiveRecord::Base, but how do you open an exising method and add more bits on?

I tried alising the method... but the question became: how do you alias a method that is defined as self.validates_whatever?

This excellent post describes how to extend class methods by using base.class_eval and putting the alises into class << self

Note: you definitely need to double-alias your function (as in the class extension section below) or you will find yourself instantly spiralling into a "stack level too deep" exception the first time you call it.

Here's how to extend a validation in Rails (includes my extension to validates_numericality_of, and also my preivous validates_percentage method):

module MyNewValidations
  # magic to allow us to override existing validations
  def self.included(base)
    base.extend(ClassMethods)
    base.class_eval do
      class << self
        alias old_numericality_of :validates_numericality_of unless method_defined?(:old_numericality_of)
        alias validates_numericality_of :my_validates_numericality
      end
    end
  end

  module ClassMethods
    # extends the "validates numericality of" validation to allow the option
    # ":positive_only => true" or ":negative_only => true"
    # This will validate to true only if the given number is positive (>= 0)
    # or negative (<= 0) respectively
    # Otherwise is behaves exactly as the standard validation
    def my_validates_numericality(fields, args = {})
      ret = old_numericality_of(fields, args) # first call standard numericality

      pos = args[:positive_only] || false
      neg = args[:negative_only] || false

      if pos || neg
        msg = args[:message] || "should be a #{pos ? 'positive' : 'negative'} number"
        validates_each fields do |model, attr, val|
          if (pos && val.to_f < 0) || (neg && val.to_f > 0)
            model.errors.add attr, msg
            ret = false
          end
        end
      end

      ret
    end

    # validates whether the given object is a percentage
    # Can also take optional args which will get passed verbatim into the
    # validation methods. Thus it's only really safe to use ":allow_nil" and
    # ":message"
    def validates_percentage(fields, args = {})
      msg = args[:message] || "should be a percentage (0-100)"
      validates_each fields do |model, attr, val|
         pre_val = model.send("#{attr}_before_type_cast".to_sym)
         unless val.nil? || !pre_val.is_a?(String) || pre_val =~ /^[-+]?\d+(\.\d*)?%?$/
           model.errors.add(attr, msg)
         end
       end
      args[:message] = msg
      args[:in] = 0..100
      validates_inclusion_of fields, args
    end

  end
end

class ActiveRecord::Base
  # add in the extra validations created above
  include MyNewValidations

  #other stuff I'd defined goes here
end

Friday 21 September 2007

quarter-master

Continuing my series of "why isn't this in ruby already?", we have some date functions that allow us to add and display the current financial quarter. I have even overridden "strftime" to accept the symbol '%Q' to display the quarter.

class Time
  # save the original strftime so we can call it later
  alias original_strftime :strftime
  # overload strftime to accept '%Q' to display the quarter
  def strftime(format)
    format = format.gsub('%Q', "Q#{self.quarter.to_s}") if format
    self.original_strftime format
  end

  # create a utc time using the year and financial quarter
  def self.utc_fq(year, fq = 1)
    self.utc(year, self.quarter_to_month(fq))
  end
  # create a local time using the year and financial quarter
  def self.local_fq(year, fq = 1)
    self.local(year, self.quarter_to_month(fq))
  end

  # turns a financial quarter (1-4) into the first month of that quarter
  def self.quarter_to_month(fq)
    ((fq-1)  * 3) + 1
  end
  # turns a given month (1-12) into the financial quarter
  # (1=jan-mar, 2=apr-jun, 3=jul-sep, 4=oct-dec)
  def self.month_to_quarter(month)
    ((month - 1) / 3) + 1
  end

  # returns the financial quarter of this date
  # (1=jan-mar, 2=apr-jun, 3=jul-sep, 4=oct-dec)
  def quarter
    self.class.month_to_quarter(self.month)
  end
end

Example of use:

>> order_fq = Time.utc_fq(order.created_at.year, order.created_at.quarter)
=> Sun Jul 01 00:00:00 UTC 2007
>> p "Order's financial quarter: #{order_fq.strftime("%Y-%Q")}"
"Order's financial quarter: 2007-Q3"
=> nil

Wednesday 19 September 2007

methods in_time

On the subject of "why isn't this in Ruby already?". I've written up a set of conversion methods that I felt sure were there already, except that I couldn't find them anywhere.

class Numeric
  # assumes i am expressed in seconds - converts me to minutes
  def in_minutes
    self / 60
  end
  # assumes i am expressed in seconds - converts me to hours
  def in_hours
    self / 3600
  end
  # assumes i am expressed in seconds - converts me to days
  def in_days
    self / (3600 * 24)
  end
end

So now I can do things like: 3600.in_hours, but also the more useful: (prevTime - Time.now).in_days

Monday 17 September 2007

float precision

So, I want to test that some floating-point helper function returns a certain value - but I only want it to check to a certain precision (so I don't get hit with negligible rounding errors).

Add this to config/environment.rb (and why doesn't this already exist in Ruby?)

class Float
  def precision(pre)
    mult = 10 ** pre
    (self * mult).round.to_f / mult
  end
end

Then you can do things like this:

assert_equal -5087.35, order1.profit_loss.precision(2)

[Edit: I've replaced 'truncate' with 'round' - it's the only change I've made in two years and I'm still using this code... maybe I should put it into the Rails core-extensions instead?]

Monday 27 August 2007

Truth in Fiction

So, I was having dinner with my cousin the other day[1]. Now my cousin and I are both part of an international book-swapping club, so we often swap books with each other, or chat about the books we're currently reading. Anyway, she had a book to lend me, and gave me a brief description of what it was about and why she liked it. What struck me was her use of the word "inoffensive" as a descriptive for the plot.

Now, I like to think I don't really judge a person by their book-taste. I mean everyone's different, and that's just fine. If you want to waste your time reading endless soppy romances, or brain-dead action books with no plot, but lots of sex... that's not my choice and doesn't impinge on my life one jot, so go for your life.

But it got me thinking about what it is that I *do* look for in a book. After all, my taste is, you could say "highly eclectic". I have fiction, non-fiction, fantasy, mysteries, science-fiction, speculative fiction, historical fabrications, and even the odd romance or action book.

With all that diversity, is it possible for there to be an overall theme running through here? Clearly not in genre, but the contrast with the word "inoffensive" really got me thinking.

I don't care if a book is offensive, what I value in a book is its sense of Truth.

Now, clearly I don't mean "it must be a true story" - after all, I read a hell of a lot of fiction (about 40-60 books a year on average). I also don't mean "it must mirror reality" as I read very little mainstream (the realm of "near reality" books) by comparison with my science fiction and fantasy.

After all, "The Lord of the Rings" has elves, and orcs and and magic rings galore, so you couldn't call is a true story, but in one sense it is a True Story. It thoroughly explores the reactions and relationships between the characters, and thereby delves into the Truth by exploring how to hold onto friendship through a struggle against near-overwhelming odds. About the Truth of courage, how it doesn't mean you are unafraid, but that you are prepared to struggle on for a cause that is worth fighting for, even when you are certain that you will not see the end of it.

It is this kind of Truth that I value in (what I consider to be) a good book.

Once I got onto that train of thought, I also realised why some other books really didn't appeal to me. I've recently tried to read my way through a few of the classic "Stainless Steel Rat" stories (by Harry Harrison). Mainly because several SF-fan friends of mine (mainly male) have enthused about how they loved them as kids. When I came to read them I found them to be bog-standard "boy's own adventure" stories... just set in space. Which were *so* not to my taste.

In the light of my new discovery I can see what was lacking in them that turned me off. A boy's own adventure works like a fantasy/day-dream. Everything goes just perfectly. Even when things are going wrong, you know that some Deus ex Machina is waiting in the wings to save the day. I get the same feeling from Traci Harding (though that is fantasy of the female kind).

Some authors bounce between the two ends. Laurell K Hamilton can push too far to the fantasy side of things, her recent books have this side clearly overshadowing the plot. Whereas her earlier books showed a great ability to get to the Truth of the relationships between the characters. Her main character (Anita Blake) is a vampire-hunter, hunting rogue vampires in a world where vampires have been made into "legitimate citizens". Unfortunately, vampires have several "natural" abilities that can cause problems in humans (especially those creating an irresistable "draw" or desire). Laurell writes poignantly about Anita's internal emotional struggle against the desire she feels for a particular vampire while simultaneously having to deal with the physical struggles she goes through in the course of her job.

Of course this also aparrently involves a lot of sex...

So, to get back to the original point, I wonder a bit about what my cousin meant by describing a book as "inoffensive". Inoffensive to what?

I can only guess that she meant inoffensive to her Christian values, and to her sense of well-being. Like many people, she probably doesn't want a provocative book that jolts her out of her current orbit. This is reasonable. She's a mother and definitely has a lot on her plate already... but this kind of thought-provoking, stimulating book is exactly what I crave.

So, I don't hold out a lot of faith that I'll really find her book to be as interesting as she finds it. But I'll definitely give it a go. Who knows? Maybe I'll be pleasantly surprised.

Notes

[1]Surjit's Indian Restaurant: 215 Parramatta Road Annandale (Sydney). I thoroughly recommend this place - they do fantastic Indian. Aparrently the Sri Lankan cricket team visits there when they are over here.

Thursday 9 August 2007

validates percentage

So, we wanted to check that a user could enter both "20" and "20%" and "20.000%" and have them all perfectly valid for a field on our models. I finished up with this validation method (below) to add to ActiveRecord::Base.

  # validates whether the given object is a percentage
  # Can also take optional args which will get passed verbatim into the
  # "validates inclusion" method.
  # It will safely take the usual set of fields for that - but if you pass
  # in a new message - it will get used for both validations.
  # The default message is: "should be a percentage (0-100)"
  def self.validates_percentage(fields, args = {})
    msg = args[:message] || "should be a percentage (0-100)"
    validates_each fields do |model, attr, val|
       pre_val = model.send("#{attr}_before_type_cast".to_sym)
       unless val.nil? || !pre_val.is_a?(String) || pre_val =~ /^[-+]?\d+(\.\d*)?%?$/
         model.errors.add(attr, msg) 
       end
     end
    args[:message] = msg
    args[:in] = 0..100 
    validates_inclusion_of fields, args 
  end

Friday 13 July 2007

Assert no errors

Small snippet today... I've mainly been working on updating the test-coverage for our app the last couple of weeks - so nothing particularly useful to share. I have, however, found the following tiny test-helper method to be useful.

Many tests require you to check for a particular error on a specific field on an object... but you also tend to check (over, and over, and over) that there aren't any errors on a given object. So I extracted that out.

Put this into "test_helper.rb" for increased DRYness

  # Assert that the handed object (thing) has no errors on it.
  def assert_no_errors(thing)
    fail("should have passed an object") unless thing
    e = thing.errors.full_messages
    assert e.empty?, "Should not have error: #{e.to_sentence}"
  end

Tuesday 19 June 2007

Slice and Dice

I've been playing around with sorting and scoping to neatly and easily sort and filter an index page. I'm finally happy with a generic set of scoping/sorting methods I can quickly apply to a new resource. It uses the autoscope plugin I discussed earlier, as well as the smart column sorting I played with.

Scope up

To start with, drop some scoping into your model object - and add the three required methods below (with suitably appropriate modifications for your purposes).

  auto_scope \
       :active => {:find => {:conditions => ['destroyed_at IS NULL']}},
       :archived => {:find => {:conditions => ['destroyed_at IS NOT NULL']}}

  # Available scopes - shouldn't this be available via autoscope?
  def self.scopes
    [:all, :active, :archived]
  end
  # Defaults (put into the model object)
  def self.default_scope
    :active
  end
  def self.default_sort
    '(destroyed_at IS NULL) DESC, login ASC'
  end

Next, drop these methods into your application.rb

  # Generate a quick-and-dirty description of the chosen sort order (for displaying in the template)
  def sort_desc
    "#{params[:dir] == 'down' ? 'descending' : ''} by #{params[:col_name] || params[:col]}"
  end
  # Generates a SQL order-by snippet based on requested sort criteria (or given default). 
  #
  # Adapted from the following blog post:
  # http://garbageburrito.com/blog/entry/447/rails-super-cool-simple-column-sorting
  def sort_order(model, default)
    orderby = "#{params[:col]} #{params[:dir] == 'down' ? 'DESC' : 'ASC'}"
    return sort_desc, orderby
  end

  # Uses model scoping to generate a scoped subset of the required objects
  # for use in the index view.
  # Returns a sorted list of the object .
  # Assumes existance of three methods on the model: 
  # scopes:: returns an array of acceptable scopes for this model
  # default_scope:: returns the scope to use when none has been selected
  # default_sort:: represents an SQL-appropriate string for this model
  #   representing the default way of sorting this model object
  def scoped_search(model, scope)
    scope = model.default_scope unless model.scopes.include?(scope.to_sym)
    sort_desc, orderby = sort_order(model, model.default_sort)
    # a description of the search/sort to display in the view
    filter_desc = "#{scope.to_s} #{model.name.pluralize.downcase} #{sort_desc}"
    return filter_desc, model.find(:all, :order => orderby) if scope.to_sym == :all
    return filter_desc, model.send(scope.to_sym).find(:all, :order => orderby)
  end

Using it is now easy. Drop something like this into your controller and call on it for all your collection-based actions.

  # applies user-specified filters and sorting to the specified collection
  def filter_users
    scope = params[:user_scope].to_sym if params[:user_scope]
    @filter_desc, @users = scoped_search(User,scope)
  end

If you need to paginate, you can always paginate-collection, or add the code into the scoped_search function.

Tuesday 12 June 2007

Super Cool Smart Column Sorting

So, I wanted to drop in some basic column sorting without having to rip out the innards of something like streamlined. I googled and came across this blogpost on SCSCS - which is so worth it for super-cool simple column sorting...

But I'm just not satisfied by what it can do. As ever, I want more. In this example, I wanted to be able to display neato little up/down arrow widgety thingies on the column heads by giving the th the appropriate "up/down" class - so the user can see which column it's all sorting by.

Arrowy goodness

We want a way to stuff a CSS class into the table's column-heads so we can show the sorty arrows. We need to indicate two things. Firstly, when the data has been sorted by a particular column, we want to indicate "this is the current sort colum". Secondly, we want to be able to indicate the sorting direction of a column (up, down or indifferent).

So the first step is to create (or appropriate) arrow-images. I'm not going to show an example here (you can figure that out for yourself) but I'll assume you have three images: SortUp.gif, SortDown.gif (to show that the current column is sorted up or down), and SortAny.gif (a double-arrow to indicate that the column could be sorted if they want to).

Stylish stuff

Then you need to put something appropriate in your stylesheet. eg:

/* what an ordinary column-head looks like */
.dataTable th {
   background-color: grey;
   color: white;
   padding-left: 15px;
   padding-right: 4px;
}
/* colour for currently sorted column */
.dataTable th.current {
  background-color: black;
  color: white;
}
/* the sorting link */
.dataTable th a {
  color: white;
  text-decoration: none;
  display: block;
  width: 100%;
}
.dataTable th.current a {
  color: grey;
}
/* pretty colours when hovering */
.dataTable th a:hover {
  color: blue;
  background-color: grey;
}
.dataTable th:hover {
  background-color: grey;
}

/* display of up and down arrows for sortable columns */
.dataTable th.up {
  background-image: url("/images/SortUp.gif");
  background-repeat: no-repeat;
  background-position: center left;
}
.dataTable th.down {
  background-image: url("/images/SortDown.gif");
  background-repeat: no-repeat;
  background-position: center left;
}
/* display for "any" sort - eg up and down arrows */
.dataTable th.any {
  background-image: url("/images/SortAny.gif");
  background-repeat: no-repeat;
  background-position: center left;
}

Stuff to note:

  • The "current" column head is a distinctively different colour to a normal column - this makes it stand out.
  • The "hover" colours provide good feedback for users - they pick up the idea that clicking here will probably do something.
  • The anchor within the column head has display:block; and width:100%; this makes the anchor stretch to the full width of the column-head - which is nicer than having to click just the word itself.
  • There's a padding-left on the anchor columns - this gives room for the arrow images. I've specified 15px as it's big enough for the images I use - YMMV.

Classy heads

So now we need to adapt the sort_link method to make it actually use this stuff. But we don't want it just on the link - the class needs to go on the whole column-head. So write a wrapper-function that will generate the whole <th> for you (class and all):

  # generate appropriate class for current sort options
  def sortlink_class(col)
    return "any" unless params[:col] || params[:dir]
    return "current #{params[:dir] == 'up' ? 'up' : 'down'}" if params[:col] == col.to_s
    "any"
  end

  # Generates a column head tag that contains a sort link and is given the
  # appropriate class for the current sorting options.
  def column_sort_link(title, column, options = {})
    content_tag 'th', sort_link(title, column, options), 
                      :class => sortlink_class(column)
  end

  def sort_link(title, column, options = {})
    if options.has_key?(:unless)
      condition = options[:unless] 
      options.delete(:unless)
    end
    new_sort_dir = params[:dir] ==  'down' ? 'up' : 'down'
    link_to_unless condition, title, request.parameters.merge(options.merge(:col => column, :dir => new_sort_dir))
  end

So, the currently sorted column will get something like class="current up", wheras most of the columns will be class="any". You'll also note I updated the badly-named "d" and "c" to "dir" and "col" so that unsuspecting maintenance people might have a clue as to the actual purpose of the options. Self-documenting code being a Good Thing.

Finally, I've modified it to pass through any extra options to the controller by merging the options into the link. This can be used to pass through requests for special sorts, extra parameters etc.

Friday 8 June 2007

Sliding Door Tabs take 2

So, a while back I wrote a quick widget for making sliding door tabs. But it wasn't enough, I wanted *MORE*. I wanted them to be responsive to conditions.

For example, I have a resource User, and I want the "users" tab to stay open no matter which user page I'm on (index, show, edit etc)... *unless*, of course, it happens also to be the "my account" page (ie I'm viewing myself)... for which I already have a tab. This is a cut above just being "on" if we happen to perfectly match the current url.

So I rewrote the helper methods. Now my tabs come with optional separators (that can be independantly styled), and with the ability to pass in a conditional to override whether or not to show the tab as "on".

  def make_tab(t)
    # use the conditional if one was passed - default to current page 
    cond = t.has_key?(:cond) ? t[:cond] : current_page?(t[:options])
    content_tag "li", link_to(t[:name], url_for(t[:options])),
                             :class => (cond ? 'current' : nil)
  end

  # Builds a tab that is just a separator
  def make_separator
    "<li class='separator'></li>"
  end

  # Make tablist out of a given set of tabs.
  def build_tablist(tabs, do_sep = true)
    list = "<ul>"
    tabs.each_with_index do |t,i|
      list << make_tab(t)
      list << make_separator if do_sep && i != tabs.size - 1
    end
    list << "</ul>"
    list
  end

And an example of use (that assumes existance of restful_auth)

<div id="topnav" class="tabNav">

    <% # standard set of tabs
      tabs = [] 
      if logged_in?
        tabs << {:name => 'dashboard', :options => dashboard_url}
        tabs << {:name => "my account", :options => user_path(current_user)}
        if current_user.is_admin?
          tabs << {:name => "widgets", :options => widgets_path, 
            :cond => (current_controller?(hash_for_widgets_path))}
          tabs << {:name => "users", :options => users_path, 
            :cond => (current_controller?(hash_for_users_path) && 
                      !current_page?(user_path(current_user)))}
        end
        tabs << {:name => 'logout', :options => logout_url}
      else
        tabs << {:name => 'login', :options => login_url}
        tabs << {:name => 'forgot password', :options => forgot_password_url}
      end
-%>
      
  <!-- tabbed browsing of main site areas --> 
  <%= build_tablist tabs, false -%>
</div>

Note that it also uses a convenience function: "current_controller?" This is below and could almost certainly be done better... but it currently serves to check if the given url-options match the current controller.

  # determines if the given set of options contains the current controller
  def current_controller?(options)
    return true if current_page?(options) # trivial case
    if options.respond_to?(:has_key?) && options.has_key?(:controller)
      return options[:controller] == controller.controller_name
    end
    # try matching it - assumes it's a string url
    options =~ /#{controller.controller_name}/
  end

Friday 25 May 2007

Scope out your models

Just found a nitfy plugin called AutoScope. It lets you define scopes for a model. For example:

class User < ActiveRecord::Base
   auto_scope \
       :fresh => {:find => {:conditions => ['activated_at IS NULL AND destroyed_at IS NULL']},
       :active => {:find => {:conditions => ['activated_at IS NOT NULL AND destroyed_at IS NULL']}},
       :archived => {:find => {:conditions => ['destroyed_at IS NOT NULL']}}
end

These set up some scoped methods

 new_count = User.fresh.count
 old_users = User.archived.find(:all)
 User.fresh.find(:all).each {|u| u.update_attribute(:activated_at, Time.now.to_utc)} 

I've found these great for filtering collection views that group users distinctly, but span across multiple columns.


    def filter_users
      return default_search unless params[:user_scope]
      scope = params[:user_scope].to_sym 
      case scope
      when :fresh, :active, :archived
        # scopes we recognise
        users = User.send(scope).find(:all)
      else
        users = User.find(:all)
      end
      users
    end

Do note, though, to be careful what to name your scopes. I started out by calling my "fresh" scope "new". Unfortunately this overwrote the "new" function on my User model... not a Good Thing.

Tuesday 22 May 2007

Don't flash your partials!

Random Rails tip #1: Don't use a partial named "flash" (well: "_flash.rhtml" to be specific) if you don't want to be flooded with deprecation warnings.

Rails auto-magically creates an instance of @flash for you. This triggers an avalanche of warnings for you in your functional tests :P

Tuesday 15 May 2007

RubyFit that works...

We're setting up a FITnesse server and intend to use RubyFIT for it. I tried using Cory Foy's tutorial to help get them talking to each other - but found it was out of date. The differences:

  • Get your Rubyfit from sudo gem install fit instead of "grabbing ruby.zip" (because ruby.zip no longer exists on the Interweb)
  • Instead of !define COMMAND_PATTERN {ruby -I %p ruby/bin/FitServer.rb -v} use !define COMMAND_PATTERN {ruby -I %p <path to your fit gem's bin>/FitServer.rb -v}
  • ignore the fact that FITnesse seems not to be as verbose with reporting the details of exceptions as Cory Foy's tutorial suggests - just move on to the "path" step

Sunday 13 May 2007

Exuberant programming

My colleague (Peter Merel) is the sort of person to get enthused at trying new stuff out. He is an Agile-evangelist, and has been trying all different ways to make this process fit into corporate-land. The main issues involved in this tend to be getting buy-in from the usual management-types who have never been exposed to this methodology, and finding ways to interpret it into the sorts of controls and reporting guidelines that they are used to.

Don't blow up the universe

The kind of "No, no! everything you're doing is wrong, get rid of it all and work our way!" approach tends to be frowned-upon as disruptive to standard operating practice. For good reason - it is.

Changing over an entire system of business practice (such as how to develop software) is a Big Thing for any company. Techie types will often scoff and refer to the inertia of corporate behemoths, entrenched dinosaurism and a corresponding lack of corporate flexibility, but the managers at these places actually have a good point. Changing systems involves time, effort and risk of failure (whether total failure of the system or simply setbacks that cause loss of market share).

A bigger company has more systems and people involved - and so more chance of breaking something valuable. They will generally have an entrenched and provably working system of practice. It may not be the best that is out there, but it has gotten them to where they are now. It's up to you to prove to them that what you're offering is not only better, but that it's worth more than the pain and effort involved in changing over.

Managerese

A complete re-write of corporate practice tends only to come onto the books once the proverbial hits the fan. At which point the company is desperately flailing around for any lifeline they can reach. But most companies actually aren't in this position, which means you have to actually, you know, convince them that what you're doing is a good idea (wierd huh?).

This requires an ability sadly lacking in most techie-types... the ability to speak with management in their own language.

There are a number of supporting tools that assist managers to get a handle on a project - tools they are used to and understand. The requirements spec, the gantt chart, todo lists and regular meetings. They not only understand thse, but know how to integrate them into a working whole. They use them to to juggle tasks and developers, to track project progress and momentum and to basically Get Things Done.

If suddenly you arrive on the scene like some animated furry-toothed geek enthusing wildly about "user stories" and index cards and stand-up meetings; sneering at the locals with their endless requirements specs, gantt charts and deliverables; refusing to work with management unless they change all their habits to match your own... you'll undoubtedly gather some resistance to your methodology ideas... even if they would be better off in the long run.

Bridging Mount Learning-curve

So what can you do to bridge the gap? Is there a way to implement all of our agile programming practices, yet still present the incumbents with the tools they are used to seeing? Can we apply the Principle of Least Surprise to our stakeholders too? Can both worlds co-exist in joyful harmony?

This is what we're currently working on. We call it "exuberant programming". It's a set of tools that allows us to be "abundantly fruitful" with our productivity (using the latest in agile methodology), while still giving joy to the more traditional managers.

The project managers are already productive and don't have the time to scale Mount Learning-curve. So let them have their requirements specs, but instruct them how to structure them using user stories, with each sentence a function point that can be fed into the priority planning-game. Let them have their Gantt charts - but use them to show User Stories. Update them at the stand-up-meetings and generate burn-down charts for the managers to let them see the project velocity.

We can work together. We can have peace in our time! We are the world... <waves lighter in air>

er, sorry, got a bit carried away there...

You can see the potential of what we're proposing. We're still working on the details. I'll post more about it as we develop it.

Saturday 28 April 2007

Exploding contracts == dilemma

Two weeks ago I was bemoaning the fact that my cash flow was in the red and my savings were running away in a torrent. I was then approached by a friend for a contract. Shortly after my interview with them I had four other clients approach me. So suddenly I'm flooded with offers!

One of the other clients was a good friend of mine as well. So I had a real dilemma on my hands. I didn't want to disappoint either one.

The first contract came first - but they hesitated on giving me a "Yes definitely" answer until after I'd started to look at the second one. I felt I had to look at each of them in good faith.

The first contract offered sexy video on the web, the second a really big name financial company

The first contract offerred a relaxed, sociable startup feel with some onsite some offsite. The second offered relaxed, sociable academic feel, onsite-only, but I'd have the chance to teach the rest of the team Rails skills.

It was pretty much neck-and-neck. It was a tough decision, from the perspective of preserving friendships. But when I weighed all the pros and cons objectively, the business decision became clear. In the end, for me, it came down to the fact that the first contract could only offer three days a week, while the second offered full-time.

I start on Tuesday.

Tuesday 24 April 2007

Presenting Rails

Ok, so this post was going to be how I gave an absolutely rocking presentation on Rails to my local LUG group (SLUG).

Unfortunately, I was giving the second talk and the first one began late and then ran way over-time. So my talk was pre-empted. The SLUG team were really nice about it (they even bought me dinner), and I'll be running it next month instead.

I certainly can't fault them - the first talk was on the cool OLPC project. I was interested in listening myself. Besides which, an extra month will give me more time to polish the talk and prepare that demo I'd meant to organise...

Anyway, I did want to mention a really cool talk I stumbled over while researching other Introductory Rails talks. Jim Weirich gave this talk and has a lot of reference material available here, but what I wanted to point out the neat slide software he used.

It's called "The Takahashi method" and is a XUL file that runs in your browser. It lets you write simple and effective slides with an image or a few words on each. These give great impact to what you're trying to say.

The data file for the slides is simple plain-text markup, so it's ultra-quick to write up your slides from your presentation notes.

Here is a link to Jim's slides and data. See how punchy that is.

Anyway, that's what I would have used... if I had the chance ;)

Friday 20 April 2007

Coding through treacle

Coding while sick is like trying to walk through fairy-floss

I once tried to explain to my aunt (a respected Dietitian) about how you need your whole brain firing on overdrive to write truly awesome code... it was difficult to explain clearly and I didn't pull it off successfully. Apparently I left her with the feeling of "and you don't think I use my brain in my work?" which was *so* not the point I was trying to make.

In my experience, most jobs (ie non knowledge-work jobs) have an element of drudgery. Tasks that you can do with only half your brain involved. You might not be able to do them while sitting in front of the tv - or while chatting with a friend... but you can do them when you're feeling less than 100%.

Coding isn't like that. Neither are things like mathematics, theoretical physics and creative writing.

At least 95% of the tasks you do require full attention. It's not like you suddenly forget how to code (or write, or whatever), it's just that if you do that work when you're not "all there"... it'll be far more buggy and inelegant. You might be able to salvage most of it, maybe with only a bit of re-writing, but if you're seriously out of it, you'll have to re-write so much of it that you might as well write it from scratch... ie you'd be better off having not written it at all.

The last week and a half has been like that for me. I've been trying to see through a near-impenetrable fug of virus-induced head-cold.

I've watched a lot of television. I've made significant in-roads into my post-easter chocolate collection. I even began a new knitting project... but I haven't coded much.

I even had trouble concentrating enough to get through my email. When you find yourself blankly staring at the screen, and realise that you've been sitting there, doped out for ten minutes doing nothing... you know there's no point firing up the text-editor.

Still, the fog has begun to clear over the last few days, so I'm slowly picking up where I left off. Which is lucky as I'm helping a friend out - he's sub-contracted a few basic rails tasks to me on his current project. Hopefully I'll be back in full-gear next week as I have a contract-interview and a talk on Rails for my local LUG.

Tuesday 3 April 2007

Plugins and engines and gems, oh my!

So I was talking to friend of mine about what I was doing and how a gem was too heavy for what I wanted. He pointed out that I should consider doing it as an engine instead of a plugin.

What's the difference between a plugin and an engine you ask?

I didn't know either... engines seem to be the forgotten children of Rails in all the current plugin-love that's going around (not knocking it - I like plugins too).

Anyway, it turns out that an engine is a plugin - with some differences. An engine requires that you install the "engines" plugin first - you can't just script/install the plugin itself. However, adding that engines plugin allows your own baby to do a hell of a lot more than your standard plugin-ly functions.

An engine is a complete, vertical slice of MVC. An ordinary plugin, by comparison, is generally a small chunk of functionality such as a helper library. Engines can have an /app/ directory structure identical to a full rails site. You can also add routes and migrations (with some user-tweaking required). This allows a fully encapsulated chunk of functionality (such as what I was proposing for my little blog).

Now, mephisto and typo are both gems. This is because they contain a huge amount of functionality (they well deserve their leading status in the Rails-blog field). They contain more functionality than what most people associate with a simple plugin... more than what people would associate with an engine (if people used engines more commonly) - so they are distributed as gems.

So, to sum up:

  • plugins are for small bundles of helpful code
  • engines are for encapsulated slices of vertical functionality and
  • gems are for complex, multi-functional systems.

So how do I make an engine?

The information on how to make an engine is a bit scattered about the web. There doesn't seem to be a whole lot out there in the way of tutorials (at least not compared to plugins).

Some of the info seems to be downright incorrect. The article that looked most promising was: the alterthoughts one, but it seems to be based on an old version of engine-construction that Just Doesn't Work. I couldn't get script/generate engine my_blog to give me anything but "can't find a generator for engine". This article is clearly built on an earlier version of both Rails and engines

The railsdoc on engines is fairly complete (though it's not a step-by-step tutorial). It turns out that you can just start out with a rails app - and turn it into an engine. Two things to remember:

Routes
Copy your engine's routes.rb into the root directory. Then instruct your users to add map.from_plugin :your_plugin into their routes.rb.
Migrations
If you have migrations, make sure they are numbered from 001. Then leave instructions for your user to run script/generate plugin_migration, then rake db:migrate.
Shared plugins
A blog needs to share the authentication plugin. But which authentication mechanism are you using? I could assume you're using the same as me (RESTful auth right?) but that's not likely. The best option is to write a wrapper library that allows people to override the functions with their own authentication system, if they choose.
Other plugins
If your plugin depends on other plugins there may be some duplication. I'm sure there must be a way that minimises double-loading of plugins eg if your plugin uses acts_as_taggable, and so does their site - maybe your install.rb could check for that and not bother installing it - but I think that may be over-optimisation.

I'm sure there must be a way to put all these tasks into "install.rb" - but I haven't had a chance to do that yet.

So, what have you done?

So I wrote a very tiny blog app (currently all is does are posts). It was enough to demonstrate the point for me while I figure out how to do the engine thing. I'm still playing with it in my spare time - but I probably won't get in installed until after the Easter long-weekend. At that point I'll also find somewhere to upload it so people can have a play with it.

Friday 30 March 2007

A blog to call my own

So, I've been thinking about shifting my blog over to my homepage recently. Why? well blogger's great, but I really need to move out and find a blog of my own. I want a tighter control over the content, I want to integrate it with my website's styling and layout and maintaining it in two places violates the DRY principle...

Ok, so I'm lazy and don't want to have to maintain stylesheets/layouts in two places :P

So anyway, in Rails, the major blog movers-and-shakers atm are Mephisto and Typo. They're both great. They provide heaps of functionality and have a thriving developer community. But they're a little on the top-heavy side. All the bells an whistles, Mephisto is almost a full CMS... you have to pretty much install either one as a new app (or hack it a lot ot fit with your existing one).

This is great if:

  1. you're starting an entirely new website from scratch
  2. you like using a CMS-approach to organise your site for you
  3. you're ok with your blog being a completely independant rails app.

In my case:

  1. I already have a pre-existing website. It might not be much, but I like what I've done and I don't want to have to re-hack it to fit around my blog software. Conversely, I don't want to spend hours hacking my blog software to fit in with my website - and risk destabilising the upgrade path (which kinda defeats the purpose).
  2. I'm a rails freelancer - not a stay-at-home mum. I like to get my hands dirty in the innards of my site. I don't like having the structure of said site dictated to me by some upstart CMS (no matter how shiny). I like to call the shots on my own turf
  3. We're seriously violating the DRY principle here. Why on earth would I want to maintain two separate sets of layout/styling? or two separate authentication systems? Single sign-on is a PITA. Yes, there are solutions, but why invite the problem into your home? and you couldn't integrate the layouts (eg put your latest post on the homepage) without contortions. No RESTful articles_path for you!

What I really want is to be able to do something like
script/plugin install blog
and have it all Just Work(tm). That doesn't seem available anywhere.

So, I've decided to write my own.

It won't be anything flash, just something bog-simple to allow me to post articles, tag them, let people comment and generate an RSS feed. It won't have whizz-bang flickr integration (well, probably not), but it's all that most people seem to need.

I'll post updates here as I progress.

Tuesday 27 March 2007

No to death threats!

I am shocked and horrified. By this post outlining sexually explicit death threats levelled against Kathy Sierra. This behaviour is outrageous and should not be tolerated. Kathy is a wonderful person, an excellent speaker and a giving member of the tech community. She has my full support. My thoughts and best wishes go out to her as she copes with her current situation.

Let's not let it happen again. If you notice such unruly behaviour forming in your own blog/forum/community - head it off. Inform the person that this behaviour is not appropriate. If that doesn't work - then do something more drastic (ban them from the site and call the police).

[Update: Andy Carvin has posted (on "stop cyber bullying day") a social network he has set up to help people combat cyber-bullying here.]

Sunday 25 March 2007

Social overload

So, just how many Rails social sites are out there? Seems everyone and his dog is in the process of creating a new one. I suppose this shouldn't be really surprising given how deeply entwined Rails is with the "Web 2.0 revolution"... but still.

Here is a shortlist of my current profiles:

Working With Rails
By far the best of the lot. It seems to have been around the longest - and if you're not on it, you're not anybody. This is the site that features the luminaries of the Ruby and Rails world. It also seems to be in active upgrade mode atm. They've just added a whole bunch more features.
Network of Ruby Freelancers
You can't access this site without registering. The concept is that this site marrys up freelancers with firms looking to hire. It's still a bit rough-and-ready, but it's only really just been launched. Still, there are already over 50 freelancers listing on it. They also cater for sub-contracting, allowing you to quote your sub-contract rate so other freelancers can pass their work on if they're overloaded.
Rails for all
I find this site to be way too complicated. I really like the navigation options (something that WWR is missing), but once you start trying to get to actual data, things aren't intuitive. It's also not as stable as it could be (the search fell over for no reason). There's a lot of potential here, but I think the developers tried to swallow too much at once. It has about the same level of functionality as WWR, but is far less dependable.
Freelancers on rails
Not a social-site per se, but a google-group intended for conversations about freelance rails contracting. Mentoring, tips and experiences. The members are pretty helpful and the topics cover a wide range of issues within freelancing. Worth joining if you're a freelancer.
Technorati
Ok, so this definitely isn't a rails-specific social site, but there are a lot of rails-specific blogs available here. I probably don't need to say more.

Friday 16 March 2007

Choices list

A basic CRUD list page tends to have various ways of manipulating the content - sorting, filtering etc. Most of these apply to a single column of data, so the UI will generally end up with a button, link or other input in the column-head. I found that I often needed a list of choices where some referred to a single column and others didn't (eg "Sort by: date, popularity, controversy") or choices that have no bearing on the data at all (eg "View as: User, Moderator, Admin").

It was annoying to have to type out the HTML for these lists of choices each time, with appropriate styling to highlight the "current item" etc so I wrote up a helper. My example uses RESTful paths - but could be quickly modified for legacy paths too.

    def make_choicelist(title,fieldname,choices,path_hash)
    # creates a horizontal list displaying a set of choices the user can
    # pick from including urls for each choice.
       cur_val = params[fieldname]
       str = "<div class=\"hlist\"><span class=\"head\">#{title}</span>"
       choices.each do |label,value| 
         path = url_for path_hash.merge(fieldname => value)
         str << "<span class=\"#{(cur_val && cur_val == value.to_s) ? "current" : "item"}"
         str << "\">#{link_to_unless_current(label, path)}</span>"
       end 
       str << "</div>"
       str
    end

You can then pass it:

  • The title of the choice list
  • The name and current value of the field that saves the state of this item
  • The set of choices (label+value) that this field can take
  • The basic URL of the page apart from this field - as a hash (so we can merge in the new value)
    
   <%  choices = [["Admin", :admin],
                ["Moderator", :mod]
                ["User", :user]] -%>
  <%= make_choicelist "View as:", :user_type, choices, 
                 hash_for_widgets_path -%>

The code will generate the list with the given title, and will add each option after it - highlighting whichever is the current choice. You can pair this with your stylesheet to restyle the choice-list appropriately. A really basic style example is below:

/* used for horizontal lists - generally for depicting choices */
.hlist {
  padding: 0px;
  background-color: #EEE;
  color: #333;
  border: 1px solid #333;
}
/* short, descriptive heading for the list */
.hlist .head {
  padding-left: 10px;
  padding-right: 10px;
  font-weight: normal;
}
/* list items */
.hlist .current, .hlist .item {
  font-weight: bold;
  margin-right:3px;
}
/* the currently-selected item */
.hlist .current {
  padding: 3px;
  border: 2px inset black;
  background-color: #333;
  color: #CCC;
}
/* an unselected item */
.hlist .item {
  background-color: silver;
  font-weight: bold;
}
/* the link for an unselected item */
.hlist .item a {
  border: 2px outset black;
  color: #333;
  background-color: silver;
  text-decoration: none;
}
.hlist .item a:hover {
  border: 2px inset black;
  background-color: #444;
  color: #CCC;
}

Which will give you something that looks like this:

Tuesday 6 March 2007

Sliding Doors tabbing in Rails

So I wanted to find a way of doing tabbed navigation in Rails for my own website. Naturally I asked Aunty google for help, who came up with many links to this article on tabnav. It looks pretty neat and I had a play with it. It is pretty easy to set up and works quite nicely *however*, it automatically puts, not only the tabs, but the page-content div into the page as well. This means you have to jigger with the css and can't just use the tabs (without the content div) if you don't wish to.

I also really like the classic Sliding Doors CSS look. So I decided to carve my own path in the bush, so to speak. I plan to clean it up a bit and turn it into a generator/plugin of my very own. For now, here are the lumps o' code I used to get it working together.

Sliding Doors styles

This is the standard set of sliding doors styles taken almost directly from the tutorial (and tweaked a little for the images I'm using). Clearly you also need the proper images - have a look at the tutorial for what's needed here.

/* these are used for dHTML positioning and style of tab menu. Much of this
 * was taken from "A list apart"s article on the "Sliding Doors technique".
 * http://www.alistapart.com/articles/slidingdoors/
 * This is an amazing article and well worth the read.*/
#header {
  /* the tab div needs to match the page background */
  background: #38cbff url(http://example.com/page_bg.png) top left repeat-x;
  
  float:left;
  width:100%;
  font-size:85%;
  line-height:normal;
  margin: 0px 0px 0px 110px;
  padding-right: 30px;
  border-bottom: 2px solid;
}
#header ul {
  margin:0;
  padding:5px 0 0 0;
  list-style:none;
}
#header li {
  float:left;
  background:url("/images/ButOffRight.png") no-repeat right top;
  margin:0;
  padding:0 15px 0 0;
}
#header a {
  float: left; /* one half of IE5 hack */
  display: block;
  background: url("/images/ButOffLeft.png") no-repeat left top;
  padding: 10px 0px 5px 15px;
  text-decoration: none;
  font-weight: bold;
  color:#fed;
}
#header a:hover {
  color:#fff;
}
#header #current {
  background-image:url("/images/ButOnRight.png");
}
#header #current a {
  background-image:url("/images/ButOnLeft.png");
  color:#222;
  padding-bottom:5px;
}

/* Commented Backslash Hack
   hides rule from IE5-Mac \*/
#header a {float:none;}
/* End IE5-Mac hack */

/* IE6 hack for padding around tabs */
* html #header a {padding: 0px 0px 5px 15px;}

Layout item

I kept the tabs in a partial template so they could be shared by multiple layouts (if necessary). Standard Rails practice puts shared partials into /app/views/shared Thus, to pull the tabs into the layout requires:

<%= render :partial => 'shared/tabs' -%>

Tab helper function

Each tab is generated using this helper. It creates the list item given the name and preferred link options, hilighting the tab if the user is on the actual page already. I'm tossing up whether to allow highlighting even when your on the main page of a group of pages - it's a GUI question that has pros and cons.

    def make_tab(t)
    # creates a navigation tab out of the given information.    
      tab = "<li"
      tab << ' id="current"' if current_page?(t[:options])
      tab << '>'
      tab << link_to(t[:name], t[:options])
      tab << '</li>'
      tab
    end

Tabs partial template

This bit is probably the messiest part. I'm just using a temporary variable to hold the tabs here - it should be passed into the partial, so the partial can be re-used with different tab-sets (right now I only use one, so it doesn't matter). Later I plan to make a Tab class and have a standard set of tabs loaded from a config file. This standard set can then be added-to or overridden just like columns in the AJAX scaffold.

<div id="header">
  <% tabs = [{:name => 'home', 
                :options => {:controller => :home, :action => :index}},
             #{:name => 'portfolio', 
             #   :options => {:controller => :portfolio}},
             {:name => 'resume', :options => 'resume.pdf'},
             {:name => 'blog', 
                :options => 'http://rubyglasses.blogspot.com/'},
                ] -%>
      
  <!-- tabbed browsing of main site areas --> 
  <ul>
    <% tabs.each do |t| -%>
      <%= make_tab t -%>
    <% end -%>
  </ul>
</div>

New Zealand and Flat HTML

I've spent a few weeks in New Zealand on holiday, which is why I've been so quiet for the past month. BTW, New Zealand is breathtakingly beautiful. But now I'm getting back into work mode again - and trying not to think of the extra kilos I've put on from all the insanely good food over there.

I've been spending some time working on my site - though, as yet, have nothing to show for it. I'm currently replacing what I have already there - trying to make it a bit more dynamically-generated. But I haven't actually deployed that yet. I'll be blogging about my sliding doors tabbing system in a moment, though, so I figured I should mention it.

Mainly I feel good as I set up a really quick website for my Aunt and Uncle. My Aunt is a Dietitian and together they have a business doing HACCP accreditation. This is something to do with making sure people are handling food safely in their kitchens (eg in Nursing Homes). They wanted a really bog-standard website up so people know they're Out There.

This is what I came up with: Simply HACCP

It's actually a flat-html site atm, but I used rails to generate the pages as a sort of overkill CMS. I wanted to see if it was worthwhile doing something like this - and it seems to be just fine. It meant I could work independantly on the layout/styling/content. When I was happy I just dumped it all to flat html and saved each file. There are only four pages, so that part wasn't hard.

In all it only took me about 2.5 hours of work - though, admittedly I had done some of the styling at an earlier date - including finding and splicing together the foody pictures from the creative commons. Maybe I can do ordinary websites as well as web apps. Of course, my web design is exceedingly basic - but it was reasonable enough for my Aunt to be happy. It's never going to win a design award, but compared with the crud that can come out of your standard FrontPage site it's even reasonably professional.

Friday 26 January 2007

Attracting women

After the LCA dinner, a friend-of-a-friend asked me that perennial question: how do you attract women to your FOSS group? Here are my ideas.

Getting women into your group

The best way to attract women to your group - is to have women in it already.

Particularly women that are great role-models: active women that contribute and organise, like Pia Waugh (LCA2007 "Seven team"). Unfortunately, for many groups this just sets up a chicken-and-egg situation. If you don't have women already, how do you get your first ones to come along? Luckily you can leverage women in other groups. Some ideas:

Start by asking your own members. Do they have wives/girlfriends or colleagues in IT? What about other local IT communities eg local businesses? This is just like any other FOSS evangelising: these women are already in IT, but maybe have never heard of FOSS (or never realised that it could actually benefit them to come to a group like yours).

Then ask groups like yours in that are in other regions. Do they know any women that live in your area that might be interested? Would they be interested in travelling to your area and bringing their friends along, to get the ball rolling?

Look into the local, regional and national "Women in IT" groups and advertise yourself there. Promote what it is that you do, and why it would be interesting - just as you would to any other newbie.

LCA's great success this year was the large percentage of women that attended (10%). This was fuelled by the work put in by Mary Gardiner and other members of the LinuxChix group. These groups exist to help women do interesting stuff in IT - your group counts!

Even if these groups don't serve to find you any members, consider asking the women to come and speak at your group. Especially on gender issues. The linuxchix miniconf was a great success with women speaking on issues such as negotiation and the gender pay-gap issue. Women are interested in these issues... and so are men.

Retaining your women

One of the broader issues for women in the corporate world, is that women and men have their own cultures. Be mindful that women's culture is likely to be distinctly different (though no less important) than the culture you already have in your group.

Obviously I am speaking broadly and individuals differ widely, but there are some common issues with women, and understanding these issues will help you integrate.

Firstly, don't assume that your culture is better. If you have a culture in which people brag about their exploits and the loudest shouter wins the floor... don't be surprised if women stay away in droves. Women come from a culture in which it is considered polite (and expected) to await an opening before speaking. They will not jump in and say their piece just because everybody else does. If an opening is not made for them, then they may never speak at all - to the detriment of everybody.

Female culture thrives off positive feedback and encouragement. If you notice that a woman in your group has done something you think is cool - encourage her to tell people about it. Do not be surprised to find that she thinks nobody wants to know. This is a place where it's ok to tell her that she is wrong. :)

Most women are self-deprecating (by inculturation) and will often down-play their experience. Poor self-esteem is common, even in those that have many successes behind them. You may need to treat your women like "the quiet kid" who needs some encouragement to speak up.

With the particularly shy and retiring - take them under your wing a bit. If she is too afraid to do it herself, promote her successes for her and show her how much her contribution is valued. Prompt her to talk about what she did and what she learned. When she sees that you're not all big, scary people that are going to laugh at her... she'll be more likely to stay.

Don't be big, scary people that laugh. ;)

Newbie women are afraid of the same things that any newbie is afraid of. They're worried that they will look stupid amongst all these shining gods of IT. They think their dumb questions and newbie mistakes will be laughed-at. It's up to you to make sure they know that it's ok to be a beginner. This also goes for experienced women with low self-esteem - who often rate themselves as a beginner even when they are an expert.

Share your own stupid questions and mistakes. Show them that you gods are really human too and were once a newbie with dumb questions (and, in many cases, still are). Let them know that everybody asks stupid questions and nobody thinks the less of them. Make them understand that what they are going through is not only normal, but expected. This can be as simple as saying "I'm glad you asked that", when they do pluck up the courage to ask.

Finally, support (or start) a women's chapter for your group. The intent is not to have a separatist group, favourable only to a minority. It should instead be a safe haven for newbie women to socialise and network with their peers as they adjust to a new and alien culture.

Getting women into your group (and keeping them there), is a Hard Problem. But there are lots of things you can try. The effort will mostly be in the initial stages - once the ball is rolling it'll take on its own momentum, and the results (I think) are worth it.

Want to know more?

Read Val Henson's article on encouraging women in linux. It's very thorough.

First contract

So I finished up my first ever contract yesterday. It was a good start. I had been hired for a week of work to implement a payment gateway to allow people to subscribe to the site.

It started out much slower than I hoped as I managed to get the bad luck of coming down with the flu on the day I was supposed to begin. I negotiated to take the first day off, but even after I started my head was pretty fuzzy. Not good for staying at the height of my game.

So I worked some longer hours than I would have done - especially in the later days of the contract, just to try to make up for the early slow-down.

Anyway, in the end the system was successfully subscribing and unsubscribing people. I was hired for a couple of extra days to do some other stuff, so the client obviously wasn't unhappy with my work. :)

Wednesday 24 January 2007

Linuxchix Rails talk

I gave my first ever talk at Linuxchix on Monday. It was an overview of Rails. I kept it informal as we're never sure of the venue or how many people will turn up. As it turned out there were around ten people and we sat in a fairly noisy area outside of Gloria Jean's at St Leonard's.

It apparently went quite well, given the location and the necessary informality. I briefly went over why Rails is so appealing to businesses and what it actually does to help a developer build a web app. I ended with a vague discussion about MVC architecture and took a few questions. People seemed particularly interested in the causes behind Rail's reported slowness: which unfortunately I don't actually know. So I'll have to research that for next time.

I had a few positive tips for improvement come out of it too. Mainly it was suggested that I tie the talk together with a narrative element - as people naturally respond to stories. It gives people a hook to hang onto as they follow the points you're presenting. Slides or a demo would have been good too - but that was a limitation of the venue.

Afterward, of course, we all headed off to SLUG and some interesting talks, including one on how AV was used during LCA, and another on the Google Summer-of-code project.

BusyBusy

I haven't died - I've just been busy.

First with my new contract with SugarStats.com I sadly fell ill on the first day, and left myself that day to try to recover. But I gave in and just struggled to work through the mild flu for the rest of the week - I can't sit abed for too long when there's work to be done. That filled in the rest of the second week of January.

Secondly was LCA - what a fantastic week! Keynotes and lectures from amazing speakers, cool shiny toys to play with and wonderful social evenings amongst the cool hackers that flocked to the con. I think I'll go every year from now on! I have yet to watch through all the amazing talks that I missed (while watching other amazing talks, of course). I also have some cool ideas for stuff I can contribute to Open Source... as well as several ideas for blog-posts that I'll type up - as my backlog.

The beginning of this week has been filled with the housework needed to get my house back into shape after a week of 8am-10pm days, followed by several more days working on my contract (which has been extended so I can do a few more things). In any case, I'll try to get down a few more posts in the next couple of days.

Thursday 4 January 2007

File uploads

I wasted time today.

I didn't intend to - I was hopng to change an admin CRUD interface over to use AJAX scaffold - which is prettier than what I had before. But when it was all changed over suddenly all my file-uploading stopped working.

I checked to make sure all the file-saving code was still in place, the filenames were correct, everything but it kept simply not finding the StringIO that was clearly there in the form.

It took me way too long to discover that you Can't upload files with AJAX. It seems the JS doesn't accept the multi-part get/post data. :(

So I changed back...

but in the meantime (while I was checking all my code) I discovered the file_column plugin which is a neat little library that does all of your file-uploading for you. It even integrates with RMagick to let you do some speccy resizing etc. So it wasn't *totally* wasted effort... just not what I'd planned spending the morning working on :P