home / blog

Responsive 2D table

For WeatherSupermarket I need to layout a grid forecasts for five days for up to six providers, Met Office, AccuWeather etc. I have the providers vertically, each with the location, and a dropdown box to change the location. The days are laid out horizontally. This has always worked fine for Desktop browsers with plenty of pixel real estate, but not for mobiles where there is limited width.

Mon 17th Tue 18th > more days…
Met Office Forecast Forecast Forecast
Weather Channel Forecast Forecast Forecast
AccuWeather Forecast Forecast Forecast
BBC Forecast Forecast Forecast
MetCheck Forecast Forecast Forecast
The Weather Channel Forecast Forecast Forecast

Possible solutions

I looked at making the table header fixed on the left and scrolling horizontally, however this is difficult if you have both vertical and horizontal headers which need to remain on display at all times.

I also tried using floating divs, however this did not work as the forecast cells are all different heights which isn’t known in advance. e.g. Met Office forecasts have a lot of detail, but BBC are one liners.

Solution

I kept the markup how it was, but introduced some JavaScript to use the window width to approximate the number of days that can be comfortably shown at one time. To allow the user to see other days, there are two large buttons to move left and right – these are enabled only when there are more columns available in that direction.

For dynamic updates – e.g. resizing the window, or flipping a mobile/tablet between portrait and landscape I put a change listener of the window object.

Example. Portrait on a mobile

Portrait

Example. Landscape on a mobile

Landscape

Example. Small Desktop (netbook)

Desktop

Code

define([ 'jquery', 'underscore' ], function() {

  "use strict";
  var $ = require('jquery');
  var _ = require('underscore');

  var TableResize = function(options) {
    this.table = $(options.table);
    this.nextButton = $(options.nextButton);
    this.previousButton = $(options.previousButton);
    $(this.nextButton).click(_.bind(this.next, this));
    $(this.previousButton).click(_.bind(this.previous, this));

    this.columnWidth = 200;
    this.headerWidth = 100;
    this.resize();
    $(window).resize(_.bind(this.resize, this));
  };
  TableResize.prototype.next = function() {
    if (this.isNext()) {
      this.visible++;
      this.quickUpdate();
      this.buttonsUpdate();
    }
  };
  TableResize.prototype.previous = function() {
    if (this.isPrevious()) {
      this.visible--;
      this.quickUpdate();
      this.buttonsUpdate();
    }
  };
  TableResize.prototype.setPointer = function(pointer) {
    this.visible = pointer;
    this.quickUpdate();
    this.buttonsUpdate();
  };
  TableResize.prototype.isNext = function() {
    return this.visible + this.visibleCols < this.columnCount - 1;
  };
  TableResize.prototype.isPrevious = function() {
    return this.visible > 0;
  };
  TableResize.prototype.buttonsUpdate = function() {
     this.nextButton[0].disabled = !this.isNext();
     this.previousButton[0].disabled = !this.isPrevious();

  };
  TableResize.prototype.update = function() {
    this.resize();
  };
  TableResize.prototype.resize = function() {
    this.columnCount = this.table.find('tr').eq(0).find('td,th').length;
    this.visibleCols = this.getVisibleColumns();
    this.visible = 0;
    this.quickUpdate();
    this.buttonsUpdate();
  };
  TableResize.prototype.quickUpdate = function() {
    var i;
    for (i = 0; i < this.columnCount; i++) {
      this.setColumnVisible(i, i >= this.visible && i < this.visible + this.visibleCols);
    }
  };

  TableResize.prototype.getVisibleColumns = function() {
    var visibleCols = ($(window).width() - this.headerWidth) / this.columnWidth;
    visibleCols = Math.floor(visibleCols);
    if (visibleCols < 1) {
      visibleCols = 1;
    }
    return visibleCols;
  };

  TableResize.prototype.getColumn = function(colIndex) {
    var cells = $([]);
    this.table.find("tr").each(function(rowIndex, row) {
      var cell = $(row).find('td,th').eq(colIndex + 1);
      cells = cells.add(cell);
    });
    return cells;
  };
  TableResize.prototype.setColumnVisible = function(colIndex, visible) {
    this.getColumn(colIndex).toggle(visible);
  };

  return TableResize;
});
Posted in geek | Tagged , | Leave a comment

CSS spriting using RMagick

I wanted to give CSS spriting a go – i.e. one large image montage and then using background-position to ‘see-through’ to the correct image.

The best off the shelf solution looks like SmartSprites but this requires you annotate your CSS with configuration…

So I wondered if I could knock something cheap and cheerful using RMagick – ImageMagick for Ruby.

Convention over configuration.

For my approach to work I made the following assumptions.

  • File name same as class-name, e.g. next.png and div.next
  • All images in CSS are class-names, not IDs.
  • Image groups in directories, e.g. symbols/cloud.png, symbols/rain.png are classes symbols-cloud and symbols-rain respectively.
  • All files in img/ at top level or in group directory at top level

Finding the images

Do a directory trawl of img/ looking for .png, exclude sprite.png itself.

images = Dir.glob("img/**/*.png").sort.select {|it| !(it.match("favicon|sprite"))}

Combining the images

I’m sure there are loads of complicated optimal layout algorithms, but I just went for one long row.

# image that is much larger, we'll crop later
cat = Image.new(4000, 1000) do 
  self.background_color= "transparent"
end
cat.alpha(Magick::ActivateAlphaChannel)
max_height = 0
x = 0
images.each do |f|
  img = Magick::Image::read(f)[0]
  img.alpha(Magick::ActivateAlphaChannel)
  cat.composite!(img, x, 0, Magick::OverCompositeOp)
  x = x + img.columns
  max_height = [max_height, height].max
end
cat.crop!(0, 0, x, max_height)
list = ImageList.new
list.push cat
list.write("png:img/sprite.png")

Creating the CSS

I added the following to the loop to build up the CSS as the image was built – quick and dirty

css_output = ''
css_classes = Array.new
images.each do |f|
  ...
  css_output = css_output + ".#{css(f)} {\n"
  css_output = css_output + "  background-position: -#{x}px 0;\n"

  css_output = css_output + "}\n"

  css_classes.push("." + css(f))
  x = x + width
  max_height = [max_height, height].max
end
css_output = css_output + "#{css_classes.join(',')} {\n background-image: url(\"/img/sprite.png\");\n}\n"

File.write('sprite.css', css_output)

Example CSS output showing just a few images…

Note this only the spriting dynamic part. I had existing handcrafted CSS for background-attachment, width, height etc. This is then imported into my main.css using @import and then inlined/optimized later in the build process.

.error-cloud {
  background-position: -0px 0;
}
.next {
  background-position: -80px 0;
}
.previous {
  background-position: -124px 0;
}
.provider-accuweather {
  background-position: -168px 0;
}
.provider-bbc {
  background-position: -250px 0;
}
.error-cloud,.next,.previous,.provider-accuweather,.provider-bbc {
 background-image: url("/img/sprite.png");
}

Posted in geek | Tagged , , | Leave a comment

Assorted IE8 polyfills: forEach, trim, placeholders

I ended up using all of the following during recent development of weathersupermarket 2.0 and another site.

Array.prototype.forEach() implemented using a for loop and fn.apply

  if (typeof Array.prototype.forEach !== 'function') {
    Array.prototype.forEach = function(callback, context) {
      for (var i = 0; i < this.length; i++) {
        callback.apply(context, [ this[i], i, this ]);
      }
    };
  }

String.prototype.trim() implemented using a regex. Just replace leading a trailing spaces with the empty string.

  if (typeof String.prototype.trim !== 'function') {
    String.prototype.trim = function() {
      return this.replace(/^\s+|\s+$/g, '');
    };
  }

The input/textarea placeholder attribute implemented using jQuery and data- attributes. Custom data attributes are only standardized in HTML5 but pretty much every browser ever has supported them via DOM setAttribute() which jQuery attr() delegates onto.

  if (!Modernizr.input.placeholder) {
    var addPlaceholder = function() {
      var input = $(this);
      if (input.val() == '' || input.val() === input.attr('placeholder')) {
        input.addClass('placeholder');
        setValue(input, input.attr('placeholder'));
      }
    };
    $('input[placeholder],textarea[placeholder]').focus(function() {
      var input = $(this);
      if (input.val() === input.attr('placeholder')) {
        setValue(input, '');
        input.removeClass('placeholder');
      }
    }).blur(addPlaceholder);
    $('input[placeholder],textarea[placeholder]').each(function(it) {
      addPlaceholder.apply(this);
    });

  }

JSON support for IE8. I actually found this was only necessary when I needed to do raw JSON.stringify() calls etc. jQuery $.ajax is coded to use a regex/eval solution if the JSON native object does not exist. So I only included the Douglas Crockford json2.js on one of the sites in question.

Posted in geek | Tagged , , | Leave a comment

Rails, active record, concurrency and scaling

Coming from a Java background I’m confused by ruby’s poor support for concurrency

The problem…

I’m rewriting weathersupermarket, a screen scraper for weather forecasts using Rails. This is the basic logic of server forecast service.

user requests forecast for location 123
get location 123 from database
if last forecast retrieved more than N seconds ago
  retrieve new forecast
  save forecast
  update last retrieved time
then
  return results from database
end

This works fine, but when it comes to handling requests from different users concurrently it sucks. Why. By default rails runs in a single threaded mode, serving one request than the next.

More threads can help?

I realise that throwing more threads at a problem doesn’t help unless you’ve got cores to service those threads. However in my case a lot of the blocking work is IO bound, i.e. requesting data from remote sites. For example, two users request data for London and Manchester, in theory the downloading of the forecasts can be done in parallel…

Enabling multithreading

Usefully, Rails 4.0 enables config.threadsafe! by default, so no work required. However the default development server WEBrick does not support it. However the lightweight server thin does. By default it supports 1024 concurrent connections – my app will break long before that!

thin --threaded --max-conns 30 start
# do some requests in parallel
wget http://localhost:3000/app/forecast/100 &
wget http://localhost:3000/app/forecast/101 &
wget http://localhost:3000/app/forecast/102 &
wget http://localhost:3000/app/forecast/103 &

Next problem

Enabling multi-threading creates a unique problem for my app. Some requests can take 5-10 seconds to complete. If there were to be another request for the same data during that period then the work would be done again, in parallel, this is not ideal.

My solution is to introduce a proxy around my service class with the following logic.

if this request being processed already
  wait for other thread to complete
  return results from other thread
else
  add request to hash map with current thread
  do the work
  save the result
end

This is the implementation with the necessary Mutex to protect the internal state.

class RequestProxy
  def initialize(target)
    @queue = Hash.new
    @queue_lock = Mutex.new
    @target = target
  end

  def get_current_request(key)
    @queue[key]
  end

  def wait_for_request(request)
    if Thread.current.object_id != request[:thread]
      request[:thread].join
    end
    self.remove_request(request)
  end

  def remove_request(request)
    @queue.delete request[:key]
  end

  def save_request(key)
    new_request = {key: key,thread: Thread.current }
    @queue[key] = new_request
    new_request
  end

  def request(key)
    current_request = nil
    new_request = nil
    @queue_lock.synchronize {
      current_request = self.get_current_request(key)
      if current_request == nil
        new_request = self.save_request(key)
      end
    }
    if current_request != nil
      self.wait_for_request(current_request)
      result = current_request[:result]
    else
      begin
        result = @target.request(key)
        new_request[:result] = result
      rescue Exception => e
        new_request[:result] = nil
        raise e
      ensure
        @queue_lock.synchronize {
          self.remove_request(new_request)
        }
      end
    end

    result
  end

end

Next problem

So everything is great now. Well, not really… I’m using ActiveRecord

…and then problem is actually slightly more complicated than I explained above… As an optimization to simplify and improve client speed the service interface supports requests for up to 6 locations in parallel, the worst case. This happens when a search for say ‘London’ returns results from all 6 forecast providers.

So with ActiveRecord a database connection is required for every thread that accesses the database. Requests from the threads coming from the controllers seem fine as the same ones are used over and over again, however when I started accessing the database from newly spawned threads I got Timeout errors. It seems that connections are held until released or the reaper gets them. You can help ActiveRecord out by telling it when you’re done with access on the current thread using release_connection

ActiveRecord::Base.connection_pool.release_connection
Posted in geek | Tagged , | Leave a comment

“unknown compression method” with open-uri

In my local build of ruby 2.1.1 I see “unknown compression method” when calling open on certain URLs…

net/http/response.rb:357:in finish': unknown compression method (Zlib::DataError)

It looks like open-uri adds the header saying it can accept gzip and deflate (Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3) but if deflate is sent it blows up.

# broken - defaults to "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3" but cannot deal with deflate
open("http://www.wunderground.com//q/zmw:00000.1.03334").read

# fixed, only accept gzip
open("http://www.wunderground.com//q/zmw:00000.1.03334", "Accept-Encoding" => "gzip").read
Posted in geek | Leave a comment

Simulating a server with netcat

Note use of \r\n. and echo -n, i.e. no automatic new line and -e escape chars

echo -ne "HTTP/1.1 200 OK\r\nContent-Encoding: gzip$(echo hello-world | gzip)\n" | netcat -l 4000
Posted in geek | Leave a comment

jQuery building array selector

I’ve just been working on code to dynamically show/hide columns in a horizontal table to fit the width of the browser, i.e. show 1 column on 320px mobile, but all on desktop etc.

To support this I need to animate the Nth cell from each row concurrently.

Firstly create an empty selector $([]). Apparently $() used to work but functionality has changed between jQuery versions.

Secondly, append jQuery objects to the empty selector using add(). e.g. selector = selector.add(foo). Note add() returns a new selector. The push() method does exist, but this is marked for internal use only.

var cells = $([]);
this.table.find("tr").each(function(rowIndex, row) {
  cells = cells.add($(row).find('th,td').eq(colIndex));
});

cells.animate({
  width : '100px'
});
Posted in geek | Leave a comment

HTTP 503 from IE6 on dreamhost

It seems that dreamhost somehow returns HTTP 503 if the user agent is exactly “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”. This is from the modern.ie IE6 windows XP build.

wget --user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://hellomynameis.org.uk/"
--2014-04-09 11:31:18-- http://hellomynameis.org.uk/
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2014-04-09 11:31:19 ERROR 503: Service Temporarily Unavailable.

If the user agent is anything else it works fine … e.g. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) foo bar

wget --user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) foo bar" "http://hellomynameis.org.uk/"
--2014-04-09 11:33:46-- http://hellomynameis.org.uk/
HTTP request sent, awaiting response... 200 OK

I debugged the headers sent by IE6 using netcat…

nc -l 8080
GET / HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: 10.5.1.69:8080
Connection: Keep-Alive

Latest on this…

Dreamhost ‘Extra Web Security’ option in the dreamhost panel causes this issue, this is just a switch for mod_security. It explicitly blocks IE6, but only the SV1 UA variant. I tried a whole load of user agents, including those all the way back to IE2 on Windows 3.1. Only the SP1+ WinXP IE6 variant is blocked.

I contacted supported, their response is IE6 is “not supported” so not going to do anything about it. I replied explaining the issue again but no luck.

if (customerQuestion.contains("IE6")) {
   response.add("IE6 is not supported");
   disregardAllContext();
   stopThinking();
   stopReading();
   enableCommonSense(false);
}
Posted in geek | Tagged | Leave a comment

AJAX contact form with Ruby CGI backend

In 2013 I vowed never to write a line of PHP again, it’s dirty and bad for your CV. When it came to implementing a basic AJAX form for a small campaign website I decided to use my new friend Ruby, but wasn’t as straightforward as I had hoped…

The site is otherwise static so using rails seemed like a bit of overkill, along with the deployment hell it causes with dreamhost. Yes the performance of CGI isn’t great compared to PHP with fast CGI mode, but we’re only talking small traffic volumes.

The ruby CGI script reads the POST data from standard input, sends the necessary headers. The whole thing is wrapped with an exception handler to avoid returning HTTP 500 errors that would otherwise occur with a non-zero exit status.The script either returns JSON {success:true} or {success:false, message:’…’}

With dreamhost it is important to tell the mail gem to use sendmail rather than SMTP otherwise you’ll get into all sorts of authentication issues.

Also with shared hosting you need to setup a custom gem directory e.g ~/.gems, gem install mail, and set the GEM_HOME variable before the call to require ‘mail’

#!/usr/bin/ruby

ENV['GEM_HOME'] = "/home/bob/.gems"

require 'mail'
require 'cgi'
require 'json'

# Must get stdin before CGI object created
stdin = STDIN.read
cgi = CGI.new
puts cgi.header("type" => "application/json")

begin
  data = JSON.parse(stdin)
  mail = Mail.new do
    from     'user@example.org'
    to       'user@example.org'
    subject  'Feedback from example.org'
    body     "Name:#{data['name']}\nComments:#{data['comments']}" 
  end
  mail.delivery_method :sendmail

  mail.deliver

  puts ({success: true}).to_json

rescue Exception => e
  puts ({success: false,message: e.message}).to_json
end

I used .htaccess to hide /app/contact.rb behind /app/contact

Options +ExecCGI
AddHandler cgi-script .rb
<ifModule mod_rewrite.c>
    RewriteEngine On
    RewriteRule ^contact$ contact.rb [L]
</ifModule>

The JavaScript reads the values from two input fields, converts to JSON and posts to /app/contact. The result from the HTTP POST is also JSON so that response is handled and relevant feedback given to the user.

var result = $('#contact .result');
var button = $("#contact button.send");

var success = function() {
  result.removeClass('failure');
  result.addClass('success');
  result.text("Your feedback has been sent. Thank you");
};

var failure = function() {
  result.removeClass('success');
  result.addClass('failure');
  result.text("There was a problem sending your feedback. Please try again later");
};
button.click(function() {
  var data = {
    name : $('#contact .name').val(),
    comments : $('#contact .comments').val()
  };
  button.attr('disabled', 'disabled');
  $.ajax({
    url : 'app/contact',
    type : 'POST',
    data : JSON.stringify(data),
    dataType : 'json',
  }).done(function(result) {
    if (result && result.success) {
      success();
    } else {
      failure();
    }
  }).fail(function(error) {
    failure();
  }).always(function() {
    button.removeAttr('disabled');
  });
});
Posted in geek | Tagged , | Leave a comment

Thunderbird: “The server may have gone down or there may be a network problem” [SOLVED]

Bizare issue on OS X 10.6 effecting both Mail and Thunderbird. Mail simply fails to download inbox, silently without error – helpful. Thunderbird reports “server email@example.org has disconnected. The server may have gone down or there may be a network problem”

Cause avast email virus scanner. Solution. Disable email virus scanner.

Posted in geek | Leave a comment