My hovercraft is full of eels

How I tell a story

2010-08-05T00:00:00-07:00

As far as I can tell, the most widely accepted format for agile user stories looks like this:

As a [role]
I would like to [action]
So that [reason for action]

I’m not entirely certain of the origins. Some have suggested Dan North and others have have suggested much earlier use. In any event, it’s use is so common I can’t recall a single “agile” project where that wasn’t the format but I’ve also never really been that comfortable with it and I’ve not really known why until recently.

Here’s an example:

As a domain novice
I want the domain experts names made prominent on a domain listing
So that I know who to turn to for help on a given domain

Though entirely contrived for the purpose of this discussion, it demonstrates the kind of stories people are inclined to write.

On the positive side, the format seems quite natural – people often have an implementation in mind and they can articulate it quite easily but find it much harder to articulate the intent. This seems to be reflected in the format with the primary focus (“I want”) expressed as the desired behaviour and the intent (“So that”) of conceptually lesser importance. The problem I have is the story centres around how the user will interact with the system and only then describes why. It presupposes an implementation with a cursory nod as to the user’s intent.

In my various roles from analyst to developer, I’ve always wanted to understand why we’re building a new feature. Experience has taught me that when I understand the why, I am better able to build a solution that satisfies the real need. As a consequence, when running workshops, I’ve always tried to have the intent stated as the primary desire/need. For example:

As a domain novice
I want to know who to turn to for help on a given domain
So that .... [what the hell goes here?]

When people go to the trouble of putting the intent first, the “So that” becomes a bit of a WTF, often resulting in a simple re-statement of the intent just to satisfy the template:

As a domain novice
I want to know who to turn to for help on a given domain
So that I can learn more about the domain [well d'uh!]

Moreover, we’ve now lost the implementation detail which, although we’re more interested in the intent, is still of value by providing context for further discussion/understanding.

What I’ve wanted was a way to encourage people to describe their intent and leave the implementation details to be decided closer to the iteration in which the story will be delivered. Yes, training is one way to achieve this but my experience is that the standard format effectively devalues the intent in favour of implementation detail. What I’d really like is to somehow nudge people into writing a story where the primary focus is the intent.

Enter the format I’ve been using recently and really liking:

As a [role]
I want to [intent]
[For example] By [action]

It’s a small but subtle change. To some so subtle that it verges on the trivial. However, my experience is that it’s similar enough to the original format so as not to seem too radical a change and at the same time different enough to encourage the behaviour I am looking for. By way of example:

As a domain novice
I want to know who to turn to for help on a given domain
e.g by having the domain experts names made prominent on a domain listing

The intent is now first and clearly specified as a higher order issue. It makes no sense to express the intent as “For example by wanting to know who to turn to for help on a given domain” so the only sensible place for it is as “I want …”.

That the last line begins with “By” identifies it as an implementation detail, acknowledging that most people have some idea of how they want it implemented and allowing them to express that without feeling awkward. With the addition of “For example” or “e.g.”– as suggested by Mike Williams – we further clarify that it as up for debate once the story hits an iteration. Finally, by placing it last and making it optional we indicate that it lends less weight than the intent.

The more I use this format the more it becomes clear when the intent is expressed as implementation, my stories tend to read much more like a plot or a narrative of the software, and I’m finding it nudges/leads me to write stories that express the intent rather than work backwards from the implementation. I’ve also noticed I’m developing some heuristics for “validation”:

When the implementation contains a conjunction (and) it probably indicates the story is an epic.
When the intent contains a conjunction, it probably indicates more than one story/epic.

Trading Design Pain for Runtime Pain

2010-06-03T00:00:00-07:00

So¹, since my post on functional programming in object-oriented languages I’ve continued to tread the path with a mixture of gratification and despair. This morning the latter became overwhelming, I’d just had enough. My brain hurt and I just wanted to pump out some code, run it. I threw out the concepts I had been using as my guide and fell back on years of “old-school” object-oriented code.

Unfortunately, I was no more productive. In fact, I’d argue I was less productive. Things began to fail in weird and unexpected ways. The number of tests I needed to write to catch errors at least doubled. I soon returned to the comfort of my hybrid world.

In hindsight, I had traded design pain for runtime pain. All the mental gymnastics that went into working out how to build classes that are inter-related and at the same time immutable, etc. was replaced with time spent writing tests for anticipated edge cases as well as debugging the unexpected ones.

I concluded that the “pain” I had been experiencing was largely the result of being forced to deal with the complexity of the underlying problem. Once solved however, the code fell out with few or no bugs. By contrast, when I reverted to my previous approach, the code flowed far more freely but I spent a lot more time working out how to ensure the code didn’t do nasty things to itself.

What’s perhaps as interesting to me is that my designs are resulting in smaller and smaller classes. The more I think about problems in a functional way, the more I’m am able to design solutions that are essentially pipelines. The irony being that even though we think of imperative code as being step-by-step, it more often than not turns out to be a big, intertwined blob. Functional code on the other hand is almost by definition a series of steps, or transformations applied one after the other on some input.

These two observations are drawing me ever closer to just “getting over it” and using a functional language. The issue for me is the only FP language I know and actually like is Haskell and the only FP language I’d be likely to get into production is Clojure. Which is all I’ll say in public as I have no desire to start a flame war :)

A Connective Word Takes the Lead
↩

Functional programming in object oriented languages

2010-03-01T00:00:00-08:00

In my current job, I spend about 40% of my time with my underpants on the outside, digging around in production code, generally making stuff better. The other 60% of the time is R&D. The R&D part has some very concrete objectives but there is certainly leeway to explore different ways of developing software.

Like many programmers, my first formal introduction to OO was all about classes and inheritance. What mattered most was getting the “structure” right. Next, I came to understand the importance of encapsulation, and after that polymorphism.

Over the past 6-12 months or so I’ve become more and more interested in functional programming concepts if not functional programming languages. I’ve always been a big fan of declarative programming, business rules, etc. and yet I’ve also always been a big fan of OO. Even when I was an assembler programmer I tended to structure my code and data as if it were object-oriented, even if the self pointer was explicit.

Mor recently I re-read SICP, learned Haskell which I unashamedly love, played with Clojure on and off, and briefly looked at Scala. Co-incidentally, I also read Domain Driven Design, and Clean Code along with a number of other very interesting articles on functional programming, immutability in general and recursion in object-oriented languages.¹²³⁴ All of which got me thinking, once again, that perhaps I’d been doing this OO thing all wrong.

Here I’d like to present a few observations from my exploration into functional programming in an object-oriented world.

Immutable objects good; Mutable objects bad
An object is a collection of partially applied functions
An object’s API provides a clear separation between Commands and Queries
An object is a snapshot of state and possible outcomes
An object is a persistent data structure

Clear as mud hey?

Immutable objects good; Mutable objects bad

“Classes should be immutable unless there’s a very good reason to make them mutable….If a class cannot be made immutable, limit its mutability as much as possible.” – Joshua Bloch, Effective Java

I won’t go into a lot of detail as to why I believe this to be true. There are plenty of arguments to be found both for and against. Suffice it to say, the direction I’m taking and the conclusions I draw from my experiences are predicated on this belief. What I will say however, is that the effect it has on my designs quite often gives me the same sense of satisfaction as when practising Test Driven Design.

It’s not as though immutability in object-oriented programming is anything new. Many years ago I wrote code where all my value objects were immutable as was the occasional service class, etc. The difference in my current approach is that everything is immutable except for a tiny layer at the fringes where I need to save data for later retrieval, consume HTTP requests, etc. Yes, even my “entities” are immutable.

An object is a collection of partially applied functions

“The ideal number of arguments for a function is zero … More than three requires very special justification – and then shouldn’t be used anyway.” – Bob Martin, Clean Code

In functional languages, partial application of a function allows us to define a new function by “pre-populating” some of the arguments. For example, we can take a very simple Haskell function that calculates the amount of applicable sales tax:

applicableSalesTax percentage amount = (percentage / 100) * amount

and then partially apply it to create another function that has a fixed sales tax:

applicableGST = applicableSalesTax 10

The function applicableGST partially applies applicableSalesTax with the value 10. Anytime we call applicableGST it will invoke applicableSalesTax with a rate of 10% and whatever amount we pass to it.

Now consider an object-oriented approach (in Ruby). Imagine a very simple SalesTax class that holds the rate:

class SalesTax

  def initialize(percentage)
    @rate = percentage / 100.0
  end

  def applicable_for(amount)
    (amount * @rate)
  end

  def included_in(amount)
    amount - excluded_from(amount)
  end

  def excluded_from(amount)
    amount / (1 + @rate)
  end

end

Here, the constructor sets up the context within which each of the methods then operates, just the way we always read good OO code should be.

I’ve starting to think of constructor arguments as the mechanism for partially applying all the methods on an object. Considering an object as a partial application of a set of methods is really quite interesting to me. It almost dictates that methods MUST operate, in some way, on the state of the object – just as we always read good OO code should – only there’s a nice explanation as to why: If they didn’t operate on the object’s state, they wouldn’t be partially applied.

When I apply this principle in my designs I find I have smaller, more cohesive classes – ie the methods are all related more closely to the shared state. I also find I have far fewer private methods and more often than not, none at all. (This coincidentally fits in nicely with my relatively long held belief that private methods are a smell though this is a perhaps a topic for another discussion.) I also find that the constructor then becomes a meaningful, nay critical, part of my API.

An object’s API provides a clear separation between Commands and Queries

“Methods should return a value only if they are referentially transparent and hence possess no side effects.” – Bertrand Meyer

Interestingly, when we deal with immutable objects we really have little choice but to do just this.

If we want an object to “answer something” (a query) no modification is expected and we simply return a (potentially calculated) value:

def full_name
  "#{@first_name}, #{@last_name}"
end

If we want an object to “do something” (a command) we’ll be expecting some kind of representation of the new state as the result:

def set_first(new_last_name)
  Person.new(@first_name, new_last_name)
end

If we assume we only want to return one thing from a method, and that a change in state necessitates returning a handle to the new state, then a method can only ever be either a query or command but not both.

After discussing this with a colleague, they suggested that in a sense commands are now queries, eg. that ask the question: “what would a system look like if I asked you to do something?” Given this definition of Commands and Queries I can certainly see it from that perspective.

In the example just given we merely returned a newly created object but it could just as easily have inserted a record into a database. So, I still like to think of commands in the original sense except they now return a handle to the new state. Much like an HTTP PUT/POST/DELETE request does when following the principles of REST.

(As an aside, one consequence of this approach is that objects are closed under modification. That is, whenever we modify an object, we receive the result in the form of a new object of the same type.)

An object is a snapshot of state and possible outcomes

“Domain Events represent the state of entities at a given time when an important event occurred and decouple subsystems with event streams. Domain Events give us clearer, more expressive models in those cases.” – Eric Evans

I presume it would be largely uncontroversial to describe immutable objects as a snapshot of some state. When I also consider an object as a collection of partially applied functions, I begin to think of an object – and therefore the system as a whole – as a snapshot not only of state but also possible outcomes.

I suspect thinking about the difference between a system that undergoes state changes in situ and one that in effect represents every possible outcome simultaneously without needing to materialise them all in advance leads to a very different way of modelling a problem.

While pondering this, I recalled that as a kid I loved Choose Your Own Adventure books? At the end of each page (or few pages) you are presented with a set of choices: Do you turn left? Do we Open the door? Do you run away? Depending on the choice we make, the story takes a different course. Objects seem a bit like pages in a Choose Your Own Adventure Book, frozen in time. The methods are like the choices we make – each one takes us to a different page, itself frozen in time. I used to bookmark pages with my thumb, like a save-point. If I didn’t like the way the story was headed I simply turned back to the last known “good” point in the book.

An object is a persistent data structure

A persistent data structure is one that efficiently preserves the previous version of itself when it is modified. One of the simplest persistent data structures is the singly-linked (cons) list but almost any tree structure can be adapted to be persistent.

If we’re creating snapshots in order to preserve the initial state we’ll need to make a copy that can be modified. It’s reasonable to assume that all these redundant copies will take precious memory and computing cycles and in doing so create quite a bit of garbage. Assuming a naive copying strategy that makes deep, nested copies, this will certainly be the case. Thankfully, there are some pretty neat optimisations we can make by because every object is immutable.

When thought of as a tree with the references to other objects as the children, an object can be treated just like a persistent data structure. If we need to make a copy and modify something, all we need do is create a shallow copy and change the necessary fields. In this way, each copy is like a delta from the previous, sharing as much state as possible and reducing not only the time to copy but also the amount of memory needed.

When I first started down this path, I used constructors for this purpose. For example, if I wanted to create a new Money object, I’d create a new version using regular object construction:

class Money

  attr_reader :currency, amount

  def initialize(currency, amount)
    @currency = currency
    @amount = amount
  end

  def add(other)
    Money.new(@currency, @amount + @currency.convert(other.currency, other.money))
  end

end

I found this approach worked fine for small, value-object sized, classes however for objects that reference more than two or three values, constructing new instances became cumbersome. Moreover, if an object had internal state that was hidden but necessary when copying, the constructor API would become polluted. Even in languages such as Java that provide method overloading, or Ruby with first-class associative arrays, it still felt overly complicated to construct a new instance just to change a single value.

Instead I’ve started using an approach that involves cloning. Whenever I need to make a change, I perform a shallow copy, update the appropriate fields and return the result. In a language such as Ruby, this is quite simple to implement and make safe. Instead of using object construction, anytime I wish to make a change I do something like this:

def add(other)
  transform { @amount = @amount + currency.convert(other.currency, other.money) }
end

The add method now looks similar to a method you’d write if you were able to modify state; we assign a new amount based on some calculation. The transform method⁵ is the key here by making a shallow copy of the object, running the block within the context of the copy, then freezing the result to prevent modification before returning it to the caller. I’m finding this approach to have a number of advantages.

My constructor API isn’t “polluted” with internal implementation concerns. The constructor remains a part of the public API.

The construction of the modified copy isn’t leaking into the implementation of the method itself. Instead, the method can focus on the job at hand.

Because the method only ever needs to concern itself with the data upon which it operates, the rest of the class can vary relatively independently. When we were explicitly constructing a new object, the add method had to concern itself with also copying the currency. When using transform that problem goes away. No matter how many other fields need to be copied, the add method remains unchanged.

In a sense each method describes the delta between the current state and the new state. Just like a persistent data structure. It reminds me of branching in a Version Control System.

More to explore

As I mentioned very early on, I’m working with designs that are almost completely immutable, even at the entity level. This approach results in some really positive benefits but also presents some interesting implementation challenges as well as challenging some of my long held beliefs.

Immutable designs feel easier to reason about. I have no empirical evidence for this just a gut feel. When I’m trying to work out why something isn’t working as expected, more often than not I can just read through the code and work out what happened. I’m not trying to keep a mental mode of all the side effects.

When dealing with any kind of RDBMS, immutable objects lend themselves to a model where changes to the database are written as events rather than updates to individual records. Unlike a traditional ORM, I don’t have the “luxury” of modifying an object to assign an ID. This hasn’t presented much of a problem as yet though I suspect it might become more interesting as the object model increases in complexity.

I’ve been contemplating trying out an OODB such as MagLev to see how that might fit in. I suspect it should be no more difficult than with mutable objects and perhaps even simpler.

I’m also working on a project at the moment that uses an immutable RDF store wrapped in an object model to make it actually useable. So far it’s fit really nicely.

I’m weary of my code turning into a collection of function objects. Ie. objects that effectively have a single doIt method. It doesn’t happen too often to be a concern but it certainly does happen and I’m still not certain how I feel about it.

I sometimes find myself exposing state I wouldn’t otherwise have needed when I used mutable objects. It feels a tad icky but thus far I haven’t found it to be much of an issue.

Testing has also been interesting. I find I’m doing far less mocking and much more state-based testing. I find the tests I’m writing to be far more declarative than when I do interaction based testing. Define the initial state of the system, run the code, and compare against the expected state. Even (especially?) at the unit level this has been very effective. Again, I’m not sure how I feel about this but so far, it’s worked out well.

Of course not everything is immutable. The database isn’t nor is the file system though in both cases I try my best to treat them as if they were by only ever writing new records/files. The system runtime isn’t immutable either – the act of creating a new object proves that.

Perhaps it really is a natural progression from here to the use of more functional languages as some of my colleagues tell me I should. However there does seem to be a general lack of distinction between the benefits of functional programming concepts and functional programming languages. I’m thoroughly enjoying incorporating functional programming concepts into object-oriented languages.

Tests as documentation

2010-02-11T00:00:00-08:00

Whilst I’ve been playing around with immutable collection classes in Ruby, I’ve also been working on ways to document behaviour without writing loads of RDOC that goes stale really quickly.

Tests have always been touted as a form of documentation but I’ve rarely – if ever come to think of it – seen that work in practice. Cucumber comes very close but I wanted something a little closer to the metal, something that allowed me to write unit tests with something like RSpec.

For sometime now, I’d been structuring my specs with this in mind and I thought I was doing a reasonably good approximation. Then today I finally had cause to test my work. As part of a feature I was implementing in another project, I wanted to use a list method I thought provided just what I required but I couldn’t remember exactly how it behaved or what the interface was so, naturally, I consulted the documentation. D’oh! But wait, I thought smugly, I’ve been writing these specs and as we all know, specs are documentation. Moreover, I’ve been putting in quite a bit of effort to make them read as such so why not go read the specs?

Suffice to say, they didn’t live up to my expectations. After running spec -f nested spec/hamster/list/span_spec.rb the result wasn’t bad but wasn’t great either:

Hamster::List
  #span
    is lazy
    on []
      with a block
        preserves the original
        returns a tuple with two items
        correctly identifies the prefix
        correctly identifies the remainder
      without a block
        returns a tuple with two items
        returns self as the prefix
        leaves the remainder empty
    on [1]
      with a block
        preserves the original
        returns a tuple with two items
        correctly identifies the prefix
        correctly identifies the remainder
      without a block
        returns a tuple with two items
        returns self as the prefix
        leaves the remainder empty
    on [1, 2, 3, 4]
      with a block
        preserves the original
        returns a tuple with two items
        correctly identifies the prefix
        correctly identifies the remainder
      without a block
        returns a tuple with two items
        returns self as the prefix
        leaves the remainder empty

For a start, there was no narrative, nothing telling me what the desired outcome was; why do I want to use this method? Secondly, whilst the individual assertions seemed to make sense when reading the spec code, once they were in this purely textual form they were somewhat useless in helping me understand what to expect. And lastly, a purely aesthetic complaint, I didn’t really like the indentation so much. Right when all that hard work should have paid off, it failed me. But not completely. I was still convinced there was some merit in what I wanted and perhaps a little more tweaking could get me closer to my ideal.

After a few iterations of modifying the code, running the specs, and reading the output, I finally hit upon something I think is pretty close to what I’ve been after:

Hamster.list#span
  is lazy
  given a predicate (in the form of a block), splits the list into two lists
  (returned as a tuple) such that elements in the first list (the prefix) are
  taken from the head of the list while the predicate is satisfied, and elements
  in the second list (the remainder) are the remaining elements from the list
  once the predicate is not satisfied. For example:
    given the list []
      and a predicate that returns true for values <= 2
        preserves the original
        returns the prefix as []
        returns the remainder as []
      without a predicate
        returns a tuple
        returns self as the prefix
        returns an empty list as the remainder
    given the list [1]
      and a predicate that returns true for values <= 2
        preserves the original
        returns the prefix as [1]
        returns the remainder as []
      without a predicate
        returns a tuple
        returns self as the prefix
        returns an empty list as the remainder
    given the list [1, 2, 3, 4]
      and a predicate that returns true for values <= 2
        preserves the original
        returns the prefix as [1, 2]
        returns the remainder as [3, 4]
      without a predicate
        returns a tuple
        returns self as the prefix
        returns an empty list as the remainder

This time there’s a narrative describing what the method does, followed by a series of examples not only describing the behaviour but also providing concrete values. Now the output reads more like documentation only rather than duplicated as RDOC that rapidly becomes disconnected from reality, it’s generated from the tests and automatically stays up-to-date.

The underlying spec is not perfect by any stretch – there is certainly a modicum of duplication between the test code and the descriptive text – but I think it strikes a reasonable balance between tests that are readable as code as well as plain text documentation. I’d certainly love to know what, if anything, others have done.

require File.expand_path('../../../spec_helper', __FILE__)

require 'hamster/list'

describe "Hamster.list#span" do

  it "is lazy" do
    lambda { Hamster.stream { |item| fail }.span { true } }.should_not raise_error
  end

  describe <<-DESC do
given a predicate (in the form of a block), splits the list into two lists
  (returned as a tuple) such that elements in the first list (the prefix) are
  taken from the head of the list while the predicate is satisfied, and elements
  in the second list (the remainder) are the remaining elements from the list
  once the predicate is not satisfied. For example:
DESC

    [
      [[], [], []],
      [[1], [1], []],
      [[1, 2], [1, 2], []],
      [[1, 2, 3], [1, 2], [3]],
      [[1, 2, 3, 4], [1, 2], [3, 4]],
      [[2, 3, 4], [2], [3, 4]],
      [[3, 4], [], [3, 4]],
      [[4], [], [4]],
    ].each do |values, expected_prefix, expected_remainder|

      describe "given the list #{values.inspect}" do

        before do
          @original = Hamster.list(*values)
        end

        describe "and a predicate that returns true for values <= 2" do

          before do
            @result = @original.span { |item| item <= 2 }
            @prefix = @result.first
            @remainder = @result.last
          end

          it "preserves the original" do
            @original.should == Hamster.list(*values)
          end

          it "returns the prefix as #{expected_prefix.inspect}" do
            @prefix.should == Hamster.list(*expected_prefix)
          end

          it "returns the remainder as #{expected_remainder.inspect}" do
            @remainder.should == Hamster.list(*expected_remainder)
          end

        end

        describe "without a predicate" do

          before do
            @result = @original.span
            @prefix = @result.first
            @remainder = @result.last
          end

          it "returns a tuple" do
            @result.is_a?(Hamster::Tuple).should == true
          end

          it "returns self as the prefix" do
            @prefix.should equal(@original)
          end

          it "returns an empty list as the remainder" do
            @remainder.should be_empty
          end

        end

      end

    end

  end

end

Lazy spec task creation

2010-01-21T00:00:00-08:00

I converted a Ruby project over to use Bundler for gem dependency management today. For the most part it worked flawlessly except, that is, when the CI build ran for the first time after the conversion:

LoadError: no such file to load -- vendor/gems/environment

Stacktrace:
tasks/spec.rb:8:in `require'
tasks/spec.rb:8:in `<top (required)>'
...
Rake aborted!

The short story: The spec task definition needed the RSpec gem to be loaded but it wasn’t until after all tasks had been defined.

Now, I could single out the spec task definition and ensure it was loaded last but that would mean adding a bunch of code to my otherwise trivial Rakefile. The other option was to somehow defer the creation of the spec task until actually needed. After a bit of searching I couldn’t find anything particularly useful so I rolled my own:

namespace :spec do

  desc "Run specifications"
  task :run => :define do
    Rake::Task[:_run].invoke
  end

  task :define do

    require 'spec/rake/spectask'

    Spec::Rake::SpecTask.new(:_run) do |t|
      t.spec_opts << "--options" << "spec/spec.opts" if File.exists?("spec/spec.opts")
    end

  end

end

The spec:run task depends on the spec:define task to create a “hidden” spec:_run task to actually do the work.

The nice thing about this all the ickiness is hidden – remove the tasks/spec.rb file and nothing else really cares – and means I can treat the spec task like any other when creating my task dependencies.

As always, YMMV.

Why Object-Oriented Languages Need Tail Calls

2009-12-24T00:00:00-08:00

Update 2009/12/30: MacRuby supports a limited form of TCO as well. I received similar results as for YARV (see below) the differences being you’re not limited to the call being the last statement in the method and there’s a bug where you receive a segmentation fault rather than a stack overflow.

Update 2009/12/27: According to this redmine ticket, YARV has some limited TCO support which is disabled by default. I performed the necessary incantations to enable it, only to discover the true meaning of “limited”: optimise calls to the same method in the same instance iff the call is the last statement in the method.

Disclaimer: I unashamedly stole the title after reading another article on the same topic.

Some of you may know of a little project I’ve been working on in my, albeit very limited, spare time. Hamster started out as an implementation of Hash Array Mapped Trees (HAMT) for Ruby and has since expanded to include implementations of other Persistent Data Structures such as Sets, Lists, Stacks, etc.

For those that aren’t up with HAMTs or persistent data structures in general, they have a really neat property: very efficient copy-on-write operations. This allows us to create immutable data-structures that only need copying when something changes, making them a very effective when writing multi-threaded code.

Hamster also contains an implementation of Cons Lists with all the usual methods you’d expect from a Ruby collection such as map, select, reject, etc. thrown in for good measure.

One of the things I really wanted to investigate was laziness. So, for example, when evaluating:

Hamster.interval(1, 1000000).filter(&:odd?).take(10)

Rather than generate a list with a million values, evaluate them all against the filter, and then select the first ten, Hamster lazily generates the list, the evaluation of filter, and even take. In fact, as it stands, the example code wont’ actually do anything; you would need to call head to kick-start anything happening at all. This behaviour extends, to the extent possible, to all other collection methods.

Hamster also supports infinite lists. For example, the following code produces an infinite list of integers:

def integers
  value = 0
  Hamster.stream { value += 1 }
end

Now we can easily generate a list of odd numbers:

integers.filter(&:odd?)

Again, rather than generate every possible integer and filter those into odd numbers, the list is generated as necessary.

OK, so enough with the apparent shameless self-promotion. Let’s get to the point.

My first implementation of lists used recursion for collection methods. The code was succinct, and, IMHO elegant. It conveyed the essence of what I was trying to achieve. It was easier to understand and thus, I would surmise, easier to maintain. The problem was that for any reasonably large list, stack overflows were common place. The lack of Tail-Call-Optimisation (TCO) meant that the recursive code would eventually blow whatever arbitrary stack limits were in place. The solution: convert the recursive code to an equivalent iterative form.

Once all methods had been re-implemented using iteration, the code ran just fine on large lists; no more stack overflow errors. The downside was, the code had almost doubled in size–1~2 lines of code became 2~4 or in some cases even more. The code was now harder to read and far less intention revealing. In short, the lack of Tail-Call-Optimisation lead to less maintainable and I’d hazard a guess, more error prone code.

The story however, doesn’t end there. Take another (albeit contrived) example that partitions integers into odds and evens:

partitions = integers.partition(&:odd?)
odds = partitions.car
evens = partitions.cadr

You would expect odds to contain [1, 3, 5, 7, 9, ...], and evens to contain [2, 4, 6, 8, 10, ...]. But the way I initially implemented the code it didn’t. Here’s an example to show what happened:

odds.take(5)    # => [1, 3, 5, 7, 9]
evens.take(5)   # => [2, 12, 14, 16, 18]

Confused? So was I until it dawned on me that I had broken a fundamental principle: immutability. The underlying block that generates the list of integers has state! Enumerating the odd values first produces the expected results but once we get around to enumerating the even values, the state of the block is such that it no longer starts at 1–reversing the order of enumeration produces a corresponding reversal of the error. Pure functional Languages such as Haskell have mechanisms for dealing with this but in Ruby, the only construct I really have available to me is explicit caching of generated values.

Once I had cached the values all was well, or so I thought. I started to write some examples that used files as lists:

File.open("my_100_mb_file.txt") do |io|
  io.to_list.map(&:chomp).map(&:downcase).each do |line|
    puts line
  end
end

Running the code above took forever to run, much slower than the non-list equivalent. I expected a little slow down sure, but nothing like that which I was seeing.

At first I suspected garbage collection–perhaps the virtual machine was being crushed by the sheer number of discarded objects; I could find no evidence for this. Next, I suspected synchronisation–anything with state needs synchronisation. Again, I found no evidence for this either. A bit more fiddling and a few dozen print statements later–Ruby has no real profiling tools that I’m aware of, something that frustrates me no end at times–I realised what the problem was.

When I failed to find any evidence of garbage collection as the culprit, it had seemed a bit odd but I wasn’t sure why I felt that way and thus moved on. Had I stopped and thought about it for a while I may have realised that in fact that was exactly the problem: there was NO evidence of garbage collection at all. How could that be? Processing hundreds of thousands of lines in a 100MB text file using a linked list was sure to generate lots of garbage. Once a line had been processed, the corresponding list element should no longer have been referenced and thus made available for garbage collection, unless… unless for some mysterious reason each element was still being referenced.

My caching implementation worked like this: As each value is generated, it’s stored in an element and linked to from the previous element: [A] -> [B] -> [C]. At face value this works well–if you never hold a references to “A” or “B”, they will become available for garbage collection. So what could possibly have been going wrong? Each line was being processed and then discarded. Surely, that meant each corresponding element should have become available for garbage collection?

Now recall that I had converted the recursive code to an iterative equivalent. This had now come back to bite me, hard!–though to be fair the recursive code would have suffered in a similar and perhaps more obvious way. The call to map runs in the context of the very first line which, because of the caching, directly and indirectly references every other line that is processed! The lack of Tail-Call-Optimisation in Ruby means that whether I use recursion or iteration, if I process all elements from the head of a stream, the garbage collector can never reclaim anything because the head element is always referenced until the end of the process!

Some of my colleagues have suggested that I just get over it and use a “real” language like Clojure. Whilst I understand the sentiment, the point of Hamster is not necessarily to implement a functional language in Ruby. Rather, it is to see what can be done in object-oriented languages and, in this case, Ruby.

Hamster has allowed me to demonstrate that functional language idioms can, for the most part, translate quite well into object-oriented equivalents. However, the lack of Tail-Call-Optimisation severely limits what is possible.

Responsibility Traps

2009-10-31T00:00:00-07:00

I recently watched a wonderfully insightful presentation by Eric Evans on Responsibility Traps. In it he describes a number of traps into which otherwise well meaning and capable developers fall. Among others these include:

Building a platform to make other (lesser) programmers more productive
Cleaning up other people’s mess; being a janitor
Making hackers (pejorative) look even better

He concludes that the responsible developer often focuses her energy on solving the wrong problem, incorrectly believing it to be in the best interests of the project/company/etc. as a whole.

Since then I have subconsciously been on the lookout for other areas of life where this anti-pattern arises and to my delight (or perhaps horror), I see it everywhere.

I thought about the number of relationships that I had dragged on for much longer than was probably good for either party, where, as a result of my desire to avoid hurting anyone, both parties suffered needlessly.

I spoke with a colleague who told me how, in a recent attempt to both fulfil his unspoken promise to help a project out of trouble and preserve the reputation of his sponsors, he continued to engage with the client well beyond his own reasonable belief of success. A near stress-related break down followed shortly thereafter leading to strained relationships all ‘round.

I can’t count the number of times I’ve tried to “rescue” a project I thought was doomed to failure, implement an unnecessarily complex story, “protect” a friend/relative/colleague/client from some bad news. In almost all cases the outcome was as bad as, if not worse than, it might have been had I confronted the reality at the outset.

Why do I fall into these traps when I believe I’m primarily motivated by of a sense of duty and responsibility? Partly I think it’s about taking a path of least resistance – It seems easier to try and press ahead to “fix” the symptom than to address the underlying cause; partly it’s about self-esteem – I’m a failure unless I can solve the problem; there’s an aspect of self-importance – few others can see what I see, so it’s up to me to do something about it. And then there’s just a plain old misguided sense of responsibility.

Many years ago, someone handed me a copy of a Garfield cartoon – ironically it was probably a technically illegal copy. The cartoon showed Garfield in bed on a Monday morning with Jon telling him to “Get out of bed Garfield!” Garfield wonders to himself “What’s my motivation?” Taken literally that doesn’t really say much but the message it has left with me is to be honest with myself about what really drives my decisions.

As a professional software developer, I believe I have a duty of care to act in the best interests of my employer even if that means delivering the news nobody wants to hear– “I think we’d be better off killing this project than sinking anymore money into it.” – and that doing so is ultimately better for me as well.

More often than not, the way a message is received has far more to do with the way in which it is delivered than with the substance of the message itself. Telling someone they’re screwed unless they do what you say probably won’t get you anywhere; suggesting humbly that they are spending double what they could be paying if all they are after is someone who’ll unquestioningly implement whatever whacky ideas they present, has historically worked out much better for me.

I’ve come to the conclusion that, more important than acting responsibly is to act ethically – fairly and honestly. If you act ethically you will necessarily end up acting responsibly but not necessarily the other way ‘round. Acting ethically takes courage, determination and optimism and that’s bloody hard work™. I encourage you to be brave, take a risk, act ethically and do the right thing.

Plugins: Grab 'em while they're stale

2009-09-19T00:00:00-07:00

I don’t like leaving unused code lying around, unused applications installed, unused clothes in the wardrobe, etc. As a consequence I’m often referred to as ‘Mr. Detritus’.

As you probably know, I’ve created a number of Ruby on Rails plugins over the years. Most of them when I first started out with Rails and for that matter, Ruby. Most of them had poor (if any) test coverage and the code looked generally like a dog’s breakfast but they satisfied a need – scratched an itch if you like – I had at the time.

Time marches on and although I will continue to use Ruby as a language, I no longer have any desire to use Rails. Some of my plugins ended up in Rails core, I’ve continued to use others on recent projects but most of them have been left to rot – some of them I wouldn’t use even if I did another Rails project having long since considered them failed experiments.

And so it is that I will very shortly (within the next month) delete most of the plugins from my GitHub account. If you wish to continue using them, feel free to fork and keep a copy for yourself. Republish them under your own name if you wish for they will no doubt be better cared for by you than me.

Update: By popular demand, a Once off never to be repeated copy of the UNSUPPORTED Rails plugins.

Random thoughts on our current Agile process

2009-08-31T00:00:00-07:00

It’s late and I don’t seem to be able to sleep so for something to do I thought I’d jot down (ok copy from an email I sent out earlier in the week) some thoughts about our development process as it has evolved over the last few months or so.

As always, I can only speak from personal experience (one data point doesn’t really count for much) so here’s my totally subjective perspective, YMMV:

Full-time pairing when possible/practical
No more than one card per pair in dev at any time
No more than one card per pair in ready-for-dev at any time
Nothing to be blocked; If it’s blocked we work to remove the blockage
New cards are demand pulled into read-for-dev as cards are moved from dev to ready-for-test
We have big picture story card sessions as required
We have planning meetings at the start of each iteration where we discuss what’s “planned”
We give everything t-shirt sizes (S, M, L)
No technical stories; everything must be done because it delivers business value
We don’t measure velocity
We aggressively split cards
I repeat, we aggressively split cards
Daily stand-ups
Parking lot for post-stand up discussions
2-week iterations with retro followed by a kick-off to discuss the stories
We try to focus on doing things as simply as possible
We try to focus on building things correctly rather than as fast as possible
We fight to have a REAL user available to better understand their needs
We rally against the usual cries of “but I know we’ll need it”; We trust that by building things simply we can always add on the extra functionality later
Trying to deliver everything to everyone leads to delivering nothing at all to anyone
T-shirt sizes help the business prioritise not estimate delivery dates; Business value is a function of, among other things, time/cost to build
Rolling technical stories into stories that deliver business value force us to change course slowly and justify changes
We usually have parallel implementations of some things as a consequence
Minimal changes are allowed on “old” implementations; anything substantial requires a migration to the “new” implementation
If we can deliver some business value early by splitting the cards we do so as soon as possible; Eg. business can view existing data in the new form but editing is a new card because they can still use the old mechanism for that.
It’s critical that the whole team is taken on the “journey” so they understand why things are being built. Doing so brings the team into alignment and also allows the team to make informed decisions such as re-structuring work to enable splitting cards for aggressively.
Bringing the team along for the journey can be painful, never gets easier, and is always worth the effort.
Just-in-time stories really does enable the business to leave the decision as to what’s important to the last possible moment
Implementing the smallest amount of code possible for each story is critical to enabling just-in-time development; less code == greater flexibility; E.g. don’t use a database when the data comes from a spreadsheet and presently only ever changes in a spreadsheet.
We do as much forward thinking as possible/practical; We think of as many likely scenarios as we can and keep reducing the scope of the implementation so as not to preclude implementing them later on
Almost nothing ends up looking as we thought it would when first envisaged.
More important than writing the code is working out how to structure the implementation so that we get the job done without precluding possible future work; sometimes this means at least thinking through a strategy for migrating from one implementation to another later on if necessary.
It’s amazing how splitting stories reveals just how little the business value certain aspects of stories
We almost never get through everything that was “planned”
We almost always end up doing stories that weren’t “planned”
We are as close as we can get (due to the bureaucratic nature of the client’s operations group) to on-demand deployment into production. Ideally we’d like it to be automated but that’s just not going to happen anytime soon
We’re motivated by getting things done; call that velocity if you will but we really haven’t found a need to measure velocity. Delivering a constant stream of small but valuable stuff into production every week is VERY motivating.
We value delivering something over delivering nothing
We actively plan for change Caveats:
We have a highly competent team
We have a fixed budget
We have internal and external users
We have UI and data-only users
We have an existing implementation we are evolving away from It’s far from perfect and is constantly evolving but as Travis observed, it kinda represents a snapshot of what being Agile means to me right now.

We're Recruiting

2009-08-01T00:00:00-07:00

If you haven’t heard already, Cogent Consulting are recruiting.

Cogent Consulting prides itself on the depth of its experience with agile software development, and its ability to leverage this experience to benefit Cogent clients. We are an open-book company, with comprehensive employee participation in decision-making.

What do we do? We’re a three part story. We go out on site as consultants to help our clients get better at producing good software, by both coaching them in agile techniques and working as integral part of their development teams. We produce high-quality websites for clients from our own premises. Finally, we build our own (mostly web-based) products, using the range of great talents that make up our team.

Right now we’re looking for people who can perform hands-on web application development both in our offices and on client sites, predominantly in Ruby on Rails.

We’re also looking for people who can provide hands-on support to clients undertaking agile transformations at both the small and large scale, as well as help out with internal product development

In either case You’ll need to show us that you:

understand the principles of agile software development and have experience working on agile projects
are collaborative, but willing to be a benevolent dictator when required
can represent us on a client site in a way that makes us proud
have a passion for software development and you continue your professional development outside of work hours
have experience with Ruby on Rails, or you are able to learn it very quickly. Extra points if you’re a Smalltalk or Haskell expert
understand that “done” means the software is in production
have a demonstrated track record of successful delivery
have a passion for software development and you continue your professional development outside of work hours
think that the time after five o’clock belongs to your family (yes, that’s contradictory - we want to know how you deal with that contradiction)
understand that software and process should be opinionated, but not bigoted
can read, you choose to read, and you understand what you read
are intellectually omnivorous
consider communication (written and verbal) to be amongst your strongest skills

In return, we’ll provide you with a collegial environment that rewards inquisitiveness rather than being an ongoing inquisition. We’ll treat you as part of the Cogent family, and give you a share of the profit and/or the products that we develop. We’ll provide an environment where you can work with your peers, be challenged, and be the best that you can be.

If this sounds like your thing, you can visit our website for more information about Cogent, or email us directly: info@cogentconsulting.com.au

Less delicious, yet more satisfying

2009-07-14T00:00:00-07:00

These days, I spread my research and reading between Instapaper and Evernote. IMHO, delicious is essentially a big old shed with crap in it and no way to actually use any of it other than marvel at how much stuff I’ve collected.

On the other hand, both Instapaper and Evernote add value to the stuff I’ve collected: Instapaper allows me to read blogs and websites on my phone, and Evernote allows me to collect and organise information according to project, tags, etc.

As Travis pointed out, this makes it difficult (nay impossible) for others to see what I’m reading and is largely the reason I send out almost daily emails to colleagues on stuff I think is more generally interesting.

This morning I noticed that Instapaper helpfully provide a read-only feed of my list. So, for anyone interested, here is the link: http://www.instapaper.com/rss/175381/1rDOvQp1xwBTeMIoml2TuzPjlmM

Update:If you’d rather not wade through everything I read to find the good bits, I’ve started “starring” items I found insightful and/or think are of more general interest. Here’s the feed: http://www.instapaper.com/starred/rss/175381/ttNEuQvOmmM5sX94f0HCO7ns

Problem Solving

2009-06-25T00:00:00-07:00

It occurred to me recently that I have this notion of programming as a process that involves breaking a problem down into a sets of smaller and smaller problems until I have something I know how to solve. (I mentioned this to Steve yesterday which reminded him of a joke about an engineer and a mathematician.)

I have previously just assumed that I therefore follow this process when I’m actually problem solving however, on reflection, I’n not so sure. More specifically, I’m either not doing it at all or, at the very least, I’m doing it intuitively.

I wonder how many people do (or have done) this as an explicit part of their own problem solving and if so, what effects they’ve noticed as a consequence.

Collaboration

2009-06-25T00:00:00-07:00

I’ve heard the phrase “Excuse my poor code” (or words to that effect) a number of times recently. I’ve said it to myself about my own code (if not to others), I’ve heard two of my work colleagues say it, and I’ve had a customer’s sole developer apologise ad nauseum about his code.

I’m not here to make any determination as to the veracity of these claims but the thing that I find interesting is that in all cases, the majority of the code in question was written by a single person.

Now this may be difficult for some to believe but I inherently don’t trust my own opinion. I do tend to confidently put ideas forward as bold assertions to be shot down when I’m wrong. The confidence comes from knowing I’ll defend my ideas to the hilt and not take it personally when my argument is proven flawed.

I hate working alone because I don’t trust decisions that weren’t arrived at through furious debate. I don’t like developers working alone no matter how good they are (or think they are) because I don’t trust they can be objective enough by themselves. In fact I don’t care if they’re working in pairs of developers or not, I just want their decisions to be scrutinised by smart people as early as possible.

The idea that the majority of a developer’s work involves going off to think on their own is totally nonsensical to me. It feels like a lot of hocus pocus going on behind closed doors and then poof a few days later something is divined. Not only is the process opaque, but the possibility for smart people to scrutinise is left as late as possible. Sure, I like to sit and ponder without the hullabaloo of every man and his dog trying to give me their 2c worth as much as the next but the lack of transparency and scrutiny in this as a process is something I find very difficult to accept.

Smart people working alone is even worse. They’re often implicitly promoted to the position of grand wizard, the seer and knower of all things. They go away, think about a problem and come back with much fanfare (trumpets playing, drums beating) to bestow upon the people their creation who will wonder in amazement at the design, so simple and yet, so complex, that only they can truly grasp the significance of what they have achieved.

When I write code on my own, I feel personally, individually, responsible. When I write code on my own, the pressures I feel are, mostly, self imposed. They make me lie awake at night. They make me code into the wee hours of the morning, on the train home. I worry that I’ve missed something. I start to believe my own hype. That I’m good enough to do this on my own. That because I did most of it on my own, I should fix it on my own lest someone else realise how crap my code is. In a vacuum my ideas have no predators. As importantly (if not more so), my priorities have no predators. I race towards a goal without stopping to re-think that goal because I’m stuck in a vicious cycle of self doubt and self confidence. Moreover, when I work alone, the production of code becomes the focus and not, as I believe it should be, thinking!

When we write code together we have shared code ownership. I feel like someone has my back. That even if we got the decision “wrong”, we decided together and if two smart people can’t get it right then maybe it’s good enough, maybe there are diminishing returns for adding more people to solving the problem. When I go home at night, sure I might tinker but I know there’s not much point to spending a lot of effort because I’ll just be re-doing it tomorrow anyway. And if there is something we missed, someone has my back if need be.

I speak as one who has been, done and experienced others doing all of the above and decided once and for all that it’s just not worth working alone. Having more than one person working on something WILL cost more but I assert that the result will be better and will be achieved more quickly and with less churn. Anyone who thinks they’re smart enough to do otherwise needs to learn to keep it in their pants.

I have this mental model of software development that is akin to the way I’ve heard Abraham Lincoln’s cabinet described. Apparently he got a whole bunch of really smart people together, even those from opposite sides of politics, and let them argue it out. His (Lincoln’s) role was to decide when to stop the debate and which idea(s) to act on.

I think that’s what collaborative development should be.

YMMV :)

No, Sleep, Till Bedtime

2009-06-14T00:00:00-07:00

Or at least until all those Twitter client developers have fixed their Twitpocalypse bugs. In case you didn’t know, a few days ago the ID range used for Twitter’s messages exceeded 2^31 (approximately 2 billion) causing any apps that stored them as 32-bit integers to think they were really small negative numbers.

It’s usual – and I say usual because I don’t always adhere to it – policy for storing external identifiers is to treat them as text, even when I know they are numbers. Why? Essentially because I consider it a coincidence that they’re numbers. That identifier, number though it may be, has no special significance to me over and above being an opaque handle to some entity in another system. As such, I like to treat them as text.

Discussing this with a good friend and colleague of mine, the question of column width came up. Ie. if you’re going to make it text, how long should the column be? If you’re lucky enough to be using a database such as PostgreSQL, then the answer is: it doesn’t matter – there’s no performance benefit to artificially limiting the size of the column. For other databases, the common practice is to use something like VARCHAR(255). Think about it, even if it is a number that’s 10^255!

Twitter claims that its API is RESTful. And if to you, REST means nice, predictable URLs with some semantic path possibly followed by a numeric id and returning numeric ids in search results, then yes, it’s RESTful. Want to see the most recent messages for a user? There’s a simple HTTP request you can make to a nice, semantic (if you speak English) URL that returns a list of them and their identifiers. And, as expected, our Twitter clients have been dutifully squirrelling these ids away in integer fields (probably because that’s the default) and all has well until 2 days ago.

Now, without going into too much of an ideological rant, I happen subscribe to the principle that RESTful URLs should be opaque. That is, a URL is a URL is a URL. No slicing, no dicing, no assembling, no joining. If I have a URL to a resource then that’s what I use. Period. End of story. (You can find plenty of discussion on this by Roy Fielding using Google.)

So, back to our column widths. Assuming we have a text field in our database large enough to accommodate a URL, we could go one step further. Rather than treat the identifier as text and storing that, why not go the whole hog and store the URL instead?

As far as I can tell, the only reason is that Twitter’s API, RESTful though they may claim, sends back numeric identifiers rather than URLs which in turn leads developers to incorrectly assume that they should be storing them as numbers.

On their own, identifiers are meaningless and in fact, useless. To utilise an identifier requires us to know the system in which it is stored and the collection in which it belongs. If instead each piece of information was identified by a URL we get all that context for free and the power to share information grows phenomenally.

To me, the beauty and power of the internet is the ability to link together disparate systems in ways no one had previously imagined. More specifically, in ways the publishers of the information never considered.

Opaque URLs combined with idiomatic use of HTTP verbs can help reduce the coupling between producers and consumers by giving back control to producers in how and where they store information and at the same time increasing the freedom for others to share and use that information.

(That last paragraph reads like an Amnesty International commercial!)

Shameless Self Promotion

2009-05-15T00:00:00-07:00

So the past couple of months, I’ve finally had the luxury of starting to realise my (and Cogent’s) dream of doing product development.

We just recently launched what we hope is a very simple, easy to use and somewhat opinionated web application for Getting Things Done™ (GTD). It’s a crowded market to be sure but we really believe we understand GTD well enough to deliver a system that is more than just a to-do list with GTD inspired keywords.

Runway is still in the early stages of feature development. For those that know anything about GTD, you’ll be happy to hear that we’re working on delivering Projects, Artifacts, Agendas and of course an Inbox, to name but a few, in the very near future.

What you see now is, and will always be, free. at some point we’ll be adding pay-for features but we’ll also be doing the right thing by all our early adopters. So, if you have 5 or so minutes, we’d love for you to sign up, have a play, and of course, tell us what you think.

Web standards and all I got was this lousy website

2009-04-17T00:00:00-07:00

Over the Easter long weekend, I had a great break from work and a great opportunity to think about and reflect on my career, my job, and my profession as a whole. It’s safe to say I become a bit disheartened and disillusioned. The one striking conclusion I kept arriving at is that we are so technology focused that we spend too much time, money and effort building things that the customer is “happy” with but not blown away by. That we artificially constrain the end user experience based on our notions of “correctness”. In particular, web application development is largely a bunch of dick-pulling technical masturbation, forever re-inventing the wheel at a ridiculously low level of abstraction shoving our technological solutions down user’s throats in the name of “software engineering”. What’s worse is that I’ve been complicit. Not only by buying the hype but often by trying to do “the right thing” even when I felt as though I was bashing my head against a brick wall.

Rewind the clock to somewhere between 1996 and 1999. During those years I, along with a good friend and colleague built a desktop application that was delivered to thousands of users across Australia using nothing more than good old-fashioned client-server SQL written in, of all things, PowerBuilder – kinda like VisualBasic. More than 10 years ago, the user experience was compelling and sophisticated, it performed exceptionally well over 2400 baud dialup modems, and we built the initial release with only 2 people over 3 months from scratch. As shameful as it is, especially coming from one so vocal about automated testing as I, we had nothing but manual testing but we also had few bugs and when users did find a problem, we fixed and redeployed within 24 hours – mostly because we didn’t want to interrupt users as they worked and so waited until after hours. Over the next 12 months, we were able to adapt to the user’s needs immediately. Rarely did we add the features as request but we always managed to produce a solution they actually needed. Fast forward a decade and I feel like I suck because I honestly don’t believe I could do the same thing again today. In fact, I challenge any of us to build the same user experience with our existing technology stack.

To those that know me well, I will no doubt sound like a broken record but I can’t help feel we’ve been trying to coerce HTML & CSS into something they just aren’t and doing so for a decade now.

Think about it, HTML: HyperText Markup Language. Does that sound like it has anything to do with layout and design? In fact do you know any designers, even those that call themselves web designers, that do any of their design work in HTML/CSS? No – well none that I’ve ever heard of. The closest I can think of is a colleague who does his wireframes in OmniGraffle and then generates HTML/CSS. Why? I put it to you it’s because we don’t think in HTML/CSS. You CAN’T effectively think in HTML/CSS and if a guy who’s expertise lies in designing user interfaces can’t think in terms of HTML/CSS why the hell do we think we should?

HTML was designed for linking documents with a modicum of layout and has served that purpose admirably. As a result, the web browser largely won the battle for desktop supremacy and almost everyone has a web browser and regularly uses a number of web sites. Similarly, pretty much everyone has a computer running an 80x86 based CPU and run dozens of applications built specifically for it. HTML/CSS are the machine language of the web.

For those of us lucky enough to have done any assembler programming, we’ve also been lucky enough not to have had to do any for a very long time. Instead, we chose to move away from assembler to other languages. C, C++, Java, Smalltalk, Python, Perl, Ruby, literally dozens of other programming languages that have systematically improved the level of abstraction. Many of these languages now run on top of the JVM, LLVM, CLR, etc. themselves abstractions on top of the underlying CPU.

Did we move because the runtime was faster? Hardly. In fact in almost all cases outrageous claims were made early on that poor performance would be the undoing of these languages and in almost all cases these claims ultimately proved unfounded. No, we moved to these languages because we hoped they would give us a better level of abstraction. That we could code more closely to the way we think. That we would one day realise the dream of literally thinking in code.

Even within languages we constantly strive to improve the level of abstraction. In many cases we’ve created Domain-Specific-Languages in order that we are better able to think IN the language most appropriate to the task at hand rather than needing to perform some contorted mapping process. This is the reason the Ruby community has slowly moved from Test::Unit to RSpec/Shoulda: Test::Unit does the job just fine but it’s verbose and “too close to the metal”. Just like assembler. When I’m the most productive I’m literally thinking in code.

We’ve largely sorted the back-end problems: Database access layers, routing, data format conversion, validation, you name it it’s all been largely worked out in whatever framework and language combination you can imagine. The same cannot be said of the front-end WHERE IT ACTUALLY MATTERS.

Granted, HTML/CSS has undergone change but to what extent and to what end? We have JSP, ASP, ERB, HAML, SASS, Liquid, blueprint, jQuery, Prototype, MooTools, Dojo, YUI, etc. but none of them appreciably raises the level of abstraction. Most advances in the world of HTML/CSS are lipstick. They’re all constrained by the fallacy that HTML/CSS is the holy grail of web design. No, the whole problem with web development is that we haven’t abstracted away the underlying technology, instead we’ve been conned by a bunch of HTML/CSS gurus and boffins who think that designing the perfect machine code is all the world needs. There is nothing more primitive than HTML+CSS when it comes to the web.

HTML & CSS try to be all things to all people and by doing so, much like J2EE, we ended up with a set of primitive tools that are repetitive, verbose, hard to test, maintain and refactor and ultimately provide a user experience that can best be described as a tarted up, 24-bit 3270 terminal. Don’t believe me? Point me at a website where the user experience feels liquid and natural. Where it literally gets out of your way so that you never even realise you’re using it? For the most part you can’t. The poster children of the Rails world provide at best a rudimentary user experience. I suspect people use them because there is no alternative, not because it’s actually a great UX. Why? IMHO because the technology choices are just plain awful. If you can find a website with a rich user experience that just melts away, you’ll likely find a bunch of developers who either had nervous breakdowns or spent many years building some superduper framework, or both!

To be fair I’m no doubt coming across as though HTML/CSS is to blame for all the world’s problems. Not at all. We suffer from similar problems across the board in software development. It just so happens that I’ve been in the world of web development for a long time now and feeling the effects.

I’m not advocating the use of any particular technology – that would kinda defeat the purpose of my argument. What I am saying is that I believe we’re stuck in a mindset that only allows us to think inside the incredibly narrow bounds of something we’re used to, IMHO, only because it’s all we’re used to.

Rather than embracing the “web paradigm” how about we embrace the user and their experience and decide what technology would best enable us to deliver that.

A Title Case Gem for Ruby

2009-02-02T00:00:00-08:00

A project I’m working on called for some “smart” capitalisation of page titles. Essentially I wanted to take a URL slug) and generate a page title.

Rails comes with a built-in String#titleize method that capitalises every word but that looked a little odd when the title was something like: “My Hovercraft Is Full Of Eels”. So I went on a hunt for something “smarter”.

After a little search I stumbled upon Marshall Elfstrand’s JavaScript, Ruby, and Objective-C ports of John Gruber’s “Title Case” algorithm and decided to turn it into a Gem that adds String#titleize and String#titleize! (aliased as #titlecase, and #titlecase! respectively). When used in a Rails environment, this effectively replaces the Rails versions.

Now my page titles look a little more human-like: “My Hovercraft is Full of Eels”.

Plugins move

2009-01-18T00:00:00-08:00

Following hot on the heels of my blog move, I’ve finally moved all my rails plugins off the venerable RubyForge and onto GitHub.

Since I started working at CogentConsulting – no we’re not “The Company of ex-ThoughtWorkers” unless you count all 3 of us as somehow statistically significant – I’ve had less and less time and less and less inclination to spend any appreciable effort on RedHill related stuff to the point where the company really exists just to support and market Simian.

As a consequence, I’ve also dropped the RedhillOnRails moniker in favour of publishing the plugins under my personal account.

Blog move

2009-01-17T00:00:00-08:00

If you’re reading this then the move of my blog was successful and thank-you for putting up with a screwy RSS feed during the transition. No doubt you received double or possibly even triple posts.

Why the move? Well, even though GeekISP have been a fantastic hosting provider over the years and MovableType has been pretty reliable as a blogging platform, in my never ending quest to Do Less Stuff, I figured it was time to move the pain somewhere else.

From a technical perspective, the move was fairly easy though not without some pain. There is no direct way to import from MT to Blogger however I did find a tool that helped convert the MT export file into something Blogger could import.

I also wrote a quick script to replace all internal references with new links as well as generating a new .htaccess file for any links from the outside world. This step was pretty easy although it took some trial and error to work out what how Blogger converts titles into URLs – as near as I can tell it truncates to a maximum of 40 characters with a bias towards word boundaries. The duplicate posts appearing in the RSS feed were as a direct result of me re-creating the entire blog several times fixing little things here and there.

And so it is that my blog comes to be here on Blogger. The next step is to move all my domain hosting to Google Sites but that’s for another day. Hopefully this will be the last move for some time and, with someone else maintaining my blogging software, hopefully less stuffing around on my part.

Rails, meet Drupal.

2009-01-12T00:00:00-08:00

If you’ve been considering integrating (or replacing) your Drupal application with a Rails application, then Drupal Fu may come in handy.

It’s pretty rough-and-ready – I essentially just ripped the code out of an existing application and cobbled it together – with, as yet, no plugin infrastructure, Rakefile, or anything else that might give you a degree of confidence in the quality of the code :)

That said, the code has been working in a production application for a while and we figured it might help out some others going through the same pain.