home / blog

Mythtv transcode for Android

  • After a number of issues with avconv I switched to ffmpeg. Out of memory (OOM) errors. Non-monotonic timestamp issues, and audio/video sync problems
  • Sometimes mythtranscode can fail. If it does, the recommended way to fix it is to stream copy the .mpg using ffmpeg, replace the original file and try again.
  • Android only supports H264 baseline profiles.
  • 480p works quite nicely on a Samsung S3 screen
  • Built-in mythtv editor for cut-points easy and quick to use – in conjunction with “honorcutlist” option.

Custom job command to invoke

/home/mythtv/bin/transcode.sh "%DIR%/%FILE%" "%TITLE%" "%SUBTITLE%" "%STARTTIMEUTC%" %CHANID%

Script

#!/bin/bash

file="$1"
title=$(echo $2 |  tr -c "[^A-Za-z0-9]" "." )
subtitle=$(echo $3 |  tr -c "[^A-Za-z0-9]" "." )
date=$(echo $4 | perl -pe 's/(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})/\1.\2.\3-\4.\5/')
nice_title="$title-$subtitle-$date"

DEST="$(dirname $file)/phone"
mkdir -p $DEST

WORKING="$(dirname $file)/working"
mkdir -p $WORKING
cd $WORKING

if [ -f "$DEST/$nice_title.mp4" ]; then
  echo "$DEST/$nice_title.mp4 already exists. Exiting"
  exit
fi

mythtranscode --mpeg2 --chanid $5 --starttime $4 --honorcutlist
if [ ! "$?" == "0" ]; then
 echo "Transcode failed. Attempting to fix with stream copy"
 $HOME/bin/ffmpeg -i $1 -acodec copy -vcodec copy $1.fixed.mpg
 mv $1 $1.backup
 mv $1.fixed.mpg $1
 mythtranscode --mpeg2 --chanid $5 --starttime $4 --honorcutlist
fi;
 
$HOME/bin/ffmpeg -y -i $1.tmp \
 -c:v libx264 -preset slower -profile:v baseline \
 -strict experimental -c:a libfdk_aac -b:a 96k \
 -vf yadif,scale="trunc(oh*a/2)*2:480" -crf 23 -r 25 \
 "$nice_title.mp4"

mv -f "$nice_title.mp4" $DEST
Posted in geek | Leave a comment

non monotonically increasing dts error with mythtranscode [solved]

I found the error when processing the output from mythtranscode with avconv.

invalid non monotonically increasing dts to muxer in stream 1

I fixed it with a stream copy using mencoder

mencoder "input.mpg" -oac copy -ovc copy -o output.mp4

The same technique with avconv did not work, it crashed with the same non monotonic error.

avconv -y -i "input.mpg" -c:v copy -c:a copy output.mp4
Posted in geek | Leave a comment

Responsive 2D table

For WeatherSupermarket I need to layout a grid forecasts for five days for up to six providers, Met Office, AccuWeather etc. I have the providers vertically, each with the location, and a dropdown box to change the location. The days are laid out horizontally. This has always worked fine for Desktop browsers with plenty of pixel real estate, but not for mobiles where there is limited width.

Mon 17th Tue 18th > more days…
Met Office Forecast Forecast Forecast
Weather Channel Forecast Forecast Forecast
AccuWeather Forecast Forecast Forecast
BBC Forecast Forecast Forecast
MetCheck Forecast Forecast Forecast
The Weather Channel Forecast Forecast Forecast

Possible solutions

I looked at making the table header fixed on the left and scrolling horizontally, however this is difficult if you have both vertical and horizontal headers which need to remain on display at all times.

I also tried using floating divs, however this did not work as the forecast cells are all different heights which isn’t known in advance. e.g. Met Office forecasts have a lot of detail, but BBC are one liners.

Solution

I kept the markup how it was, but introduced some JavaScript to use the window width to approximate the number of days that can be comfortably shown at one time. To allow the user to see other days, there are two large buttons to move left and right – these are enabled only when there are more columns available in that direction.

For dynamic updates – e.g. resizing the window, or flipping a mobile/tablet between portrait and landscape I put a change listener of the window object.

Example. Portrait on a mobile

Portrait

Example. Landscape on a mobile

Landscape

Example. Small Desktop (netbook)

Desktop

Code

define([ 'jquery', 'underscore' ], function() {

  "use strict";
  var $ = require('jquery');
  var _ = require('underscore');

  var TableResize = function(options) {
    this.table = $(options.table);
    this.nextButton = $(options.nextButton);
    this.previousButton = $(options.previousButton);
    $(this.nextButton).click(_.bind(this.next, this));
    $(this.previousButton).click(_.bind(this.previous, this));

    this.columnWidth = 200;
    this.headerWidth = 100;
    this.resize();
    $(window).resize(_.bind(this.resize, this));
  };
  TableResize.prototype.next = function() {
    if (this.isNext()) {
      this.visible++;
      this.quickUpdate();
      this.buttonsUpdate();
    }
  };
  TableResize.prototype.previous = function() {
    if (this.isPrevious()) {
      this.visible--;
      this.quickUpdate();
      this.buttonsUpdate();
    }
  };
  TableResize.prototype.setPointer = function(pointer) {
    this.visible = pointer;
    this.quickUpdate();
    this.buttonsUpdate();
  };
  TableResize.prototype.isNext = function() {
    return this.visible + this.visibleCols < this.columnCount - 1;
  };
  TableResize.prototype.isPrevious = function() {
    return this.visible > 0;
  };
  TableResize.prototype.buttonsUpdate = function() {
     this.nextButton[0].disabled = !this.isNext();
     this.previousButton[0].disabled = !this.isPrevious();

  };
  TableResize.prototype.update = function() {
    this.resize();
  };
  TableResize.prototype.resize = function() {
    this.columnCount = this.table.find('tr').eq(0).find('td,th').length;
    this.visibleCols = this.getVisibleColumns();
    this.visible = 0;
    this.quickUpdate();
    this.buttonsUpdate();
  };
  TableResize.prototype.quickUpdate = function() {
    var i;
    for (i = 0; i < this.columnCount; i++) {
      this.setColumnVisible(i, i >= this.visible && i < this.visible + this.visibleCols);
    }
  };

  TableResize.prototype.getVisibleColumns = function() {
    var visibleCols = ($(window).width() - this.headerWidth) / this.columnWidth;
    visibleCols = Math.floor(visibleCols);
    if (visibleCols < 1) {
      visibleCols = 1;
    }
    return visibleCols;
  };

  TableResize.prototype.getColumn = function(colIndex) {
    var cells = $([]);
    this.table.find("tr").each(function(rowIndex, row) {
      var cell = $(row).find('td,th').eq(colIndex + 1);
      cells = cells.add(cell);
    });
    return cells;
  };
  TableResize.prototype.setColumnVisible = function(colIndex, visible) {
    this.getColumn(colIndex).toggle(visible);
  };

  return TableResize;
});
Posted in geek | Tagged , | Leave a comment

CSS spriting using RMagick

I wanted to give CSS spriting a go – i.e. one large image montage and then using background-position to ‘see-through’ to the correct image.

The best off the shelf solution looks like SmartSprites but this requires you annotate your CSS with configuration…

So I wondered if I could knock something cheap and cheerful using RMagick – ImageMagick for Ruby.

Convention over configuration.

For my approach to work I made the following assumptions.

  • File name same as class-name, e.g. next.png and div.next
  • All images in CSS are class-names, not IDs.
  • Image groups in directories, e.g. symbols/cloud.png, symbols/rain.png are classes symbols-cloud and symbols-rain respectively.
  • All files in img/ at top level or in group directory at top level

Finding the images

Do a directory trawl of img/ looking for .png, exclude sprite.png itself.

images = Dir.glob("img/**/*.png").sort.select {|it| !(it.match("favicon|sprite"))}

Combining the images

I’m sure there are loads of complicated optimal layout algorithms, but I just went for one long row.

# image that is much larger, we'll crop later
cat = Image.new(4000, 1000) do 
  self.background_color= "transparent"
end
cat.alpha(Magick::ActivateAlphaChannel)
max_height = 0
x = 0
images.each do |f|
  img = Magick::Image::read(f)[0]
  img.alpha(Magick::ActivateAlphaChannel)
  cat.composite!(img, x, 0, Magick::OverCompositeOp)
  x = x + img.columns
  max_height = [max_height, height].max
end
cat.crop!(0, 0, x, max_height)
list = ImageList.new
list.push cat
list.write("png:img/sprite.png")

Creating the CSS

I added the following to the loop to build up the CSS as the image was built – quick and dirty

css_output = ''
css_classes = Array.new
images.each do |f|
  ...
  css_output = css_output + ".#{css(f)} {\n"
  css_output = css_output + "  background-position: -#{x}px 0;\n"

  css_output = css_output + "}\n"

  css_classes.push("." + css(f))
  x = x + width
  max_height = [max_height, height].max
end
css_output = css_output + "#{css_classes.join(',')} {\n background-image: url(\"/img/sprite.png\");\n}\n"

File.write('sprite.css', css_output)

Example CSS output showing just a few images…

Note this only the spriting dynamic part. I had existing handcrafted CSS for background-attachment, width, height etc. This is then imported into my main.css using @import and then inlined/optimized later in the build process.

.error-cloud {
  background-position: -0px 0;
}
.next {
  background-position: -80px 0;
}
.previous {
  background-position: -124px 0;
}
.provider-accuweather {
  background-position: -168px 0;
}
.provider-bbc {
  background-position: -250px 0;
}
.error-cloud,.next,.previous,.provider-accuweather,.provider-bbc {
 background-image: url("/img/sprite.png");
}

Posted in geek | Tagged , , | Leave a comment

Assorted IE8 polyfills: forEach, trim, placeholders

I ended up using all of the following during recent development of weathersupermarket 2.0 and another site.

Array.prototype.forEach() implemented using a for loop and fn.apply

  if (typeof Array.prototype.forEach !== 'function') {
    Array.prototype.forEach = function(callback, context) {
      for (var i = 0; i < this.length; i++) {
        callback.apply(context, [ this[i], i, this ]);
      }
    };
  }

String.prototype.trim() implemented using a regex. Just replace leading a trailing spaces with the empty string.

  if (typeof String.prototype.trim !== 'function') {
    String.prototype.trim = function() {
      return this.replace(/^\s+|\s+$/g, '');
    };
  }

The input/textarea placeholder attribute implemented using jQuery and data- attributes. Custom data attributes are only standardized in HTML5 but pretty much every browser ever has supported them via DOM setAttribute() which jQuery attr() delegates onto.

  if (!Modernizr.input.placeholder) {
    var addPlaceholder = function() {
      var input = $(this);
      if (input.val() == '' || input.val() === input.attr('placeholder')) {
        input.addClass('placeholder');
        setValue(input, input.attr('placeholder'));
      }
    };
    $('input[placeholder],textarea[placeholder]').focus(function() {
      var input = $(this);
      if (input.val() === input.attr('placeholder')) {
        setValue(input, '');
        input.removeClass('placeholder');
      }
    }).blur(addPlaceholder);
    $('input[placeholder],textarea[placeholder]').each(function(it) {
      addPlaceholder.apply(this);
    });

  }

JSON support for IE8. I actually found this was only necessary when I needed to do raw JSON.stringify() calls etc. jQuery $.ajax is coded to use a regex/eval solution if the JSON native object does not exist. So I only included the Douglas Crockford json2.js on one of the sites in question.

Posted in geek | Tagged , , | Leave a comment

Rails, active record, concurrency and scaling

Coming from a Java background I’m confused by ruby’s poor support for concurrency

The problem…

I’m rewriting weathersupermarket, a screen scraper for weather forecasts using Rails. This is the basic logic of server forecast service.

user requests forecast for location 123
get location 123 from database
if last forecast retrieved more than N seconds ago
  retrieve new forecast
  save forecast
  update last retrieved time
then
  return results from database
end

This works fine, but when it comes to handling requests from different users concurrently it sucks. Why. By default rails runs in a single threaded mode, serving one request than the next.

More threads can help?

I realise that throwing more threads at a problem doesn’t help unless you’ve got cores to service those threads. However in my case a lot of the blocking work is IO bound, i.e. requesting data from remote sites. For example, two users request data for London and Manchester, in theory the downloading of the forecasts can be done in parallel…

Enabling multithreading

Usefully, Rails 4.0 enables config.threadsafe! by default, so no work required. However the default development server WEBrick does not support it. However the lightweight server thin does. By default it supports 1024 concurrent connections – my app will break long before that!

thin --threaded --max-conns 30 start
# do some requests in parallel
wget http://localhost:3000/app/forecast/100 &
wget http://localhost:3000/app/forecast/101 &
wget http://localhost:3000/app/forecast/102 &
wget http://localhost:3000/app/forecast/103 &

Next problem

Enabling multi-threading creates a unique problem for my app. Some requests can take 5-10 seconds to complete. If there were to be another request for the same data during that period then the work would be done again, in parallel, this is not ideal.

My solution is to introduce a proxy around my service class with the following logic.

if this request being processed already
  wait for other thread to complete
  return results from other thread
else
  add request to hash map with current thread
  do the work
  save the result
end

This is the implementation with the necessary Mutex to protect the internal state.

class RequestProxy
  def initialize(target)
    @queue = Hash.new
    @queue_lock = Mutex.new
    @target = target
  end

  def get_current_request(key)
    @queue[key]
  end

  def wait_for_request(request)
    if Thread.current.object_id != request[:thread]
      request[:thread].join
    end
    self.remove_request(request)
  end

  def remove_request(request)
    @queue.delete request[:key]
  end

  def save_request(key)
    new_request = {key: key,thread: Thread.current }
    @queue[key] = new_request
    new_request
  end

  def request(key)
    current_request = nil
    new_request = nil
    @queue_lock.synchronize {
      current_request = self.get_current_request(key)
      if current_request == nil
        new_request = self.save_request(key)
      end
    }
    if current_request != nil
      self.wait_for_request(current_request)
      result = current_request[:result]
    else
      begin
        result = @target.request(key)
        new_request[:result] = result
      rescue Exception => e
        new_request[:result] = nil
        raise e
      ensure
        @queue_lock.synchronize {
          self.remove_request(new_request)
        }
      end
    end

    result
  end

end

Next problem

So everything is great now. Well, not really… I’m using ActiveRecord

…and then problem is actually slightly more complicated than I explained above… As an optimization to simplify and improve client speed the service interface supports requests for up to 6 locations in parallel, the worst case. This happens when a search for say ‘London’ returns results from all 6 forecast providers.

So with ActiveRecord a database connection is required for every thread that accesses the database. Requests from the threads coming from the controllers seem fine as the same ones are used over and over again, however when I started accessing the database from newly spawned threads I got Timeout errors. It seems that connections are held until released or the reaper gets them. You can help ActiveRecord out by telling it when you’re done with access on the current thread using release_connection

ActiveRecord::Base.connection_pool.release_connection
Posted in geek | Tagged , | Leave a comment

“unknown compression method” with open-uri

In my local build of ruby 2.1.1 I see “unknown compression method” when calling open on certain URLs…

net/http/response.rb:357:in finish': unknown compression method (Zlib::DataError)

It looks like open-uri adds the header saying it can accept gzip and deflate (Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3) but if deflate is sent it blows up.

# broken - defaults to "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3" but cannot deal with deflate
open("http://www.wunderground.com//q/zmw:00000.1.03334").read

# fixed, only accept gzip
open("http://www.wunderground.com//q/zmw:00000.1.03334", "Accept-Encoding" => "gzip").read
Posted in geek | Leave a comment

Simulating a server with netcat

Note use of \r\n. and echo -n, i.e. no automatic new line and -e escape chars

echo -ne "HTTP/1.1 200 OK\r\nContent-Encoding: gzip$(echo hello-world | gzip)\n" | netcat -l 4000
Posted in geek | Leave a comment

jQuery building array selector

I’ve just been working on code to dynamically show/hide columns in a horizontal table to fit the width of the browser, i.e. show 1 column on 320px mobile, but all on desktop etc.

To support this I need to animate the Nth cell from each row concurrently.

Firstly create an empty selector $([]). Apparently $() used to work but functionality has changed between jQuery versions.

Secondly, append jQuery objects to the empty selector using add(). e.g. selector = selector.add(foo). Note add() returns a new selector. The push() method does exist, but this is marked for internal use only.

var cells = $([]);
this.table.find("tr").each(function(rowIndex, row) {
  cells = cells.add($(row).find('th,td').eq(colIndex));
});

cells.animate({
  width : '100px'
});
Posted in geek | Leave a comment

HTTP 503 from IE6 on dreamhost

It seems that dreamhost somehow returns HTTP 503 if the user agent is exactly “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”. This is from the modern.ie IE6 windows XP build.

wget --user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://hellomynameis.org.uk/"
--2014-04-09 11:31:18-- http://hellomynameis.org.uk/
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2014-04-09 11:31:19 ERROR 503: Service Temporarily Unavailable.

If the user agent is anything else it works fine … e.g. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) foo bar

wget --user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) foo bar" "http://hellomynameis.org.uk/"
--2014-04-09 11:33:46-- http://hellomynameis.org.uk/
HTTP request sent, awaiting response... 200 OK

I debugged the headers sent by IE6 using netcat…

nc -l 8080
GET / HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: 10.5.1.69:8080
Connection: Keep-Alive

Latest on this…

Dreamhost ‘Extra Web Security’ option in the dreamhost panel causes this issue, this is just a switch for mod_security. It explicitly blocks IE6, but only the SV1 UA variant. I tried a whole load of user agents, including those all the way back to IE2 on Windows 3.1. Only the SP1+ WinXP IE6 variant is blocked.

I contacted supported, their response is IE6 is “not supported” so not going to do anything about it. I replied explaining the issue again but no luck.

if (customerQuestion.contains("IE6")) {
   response.add("IE6 is not supported");
   disregardAllContext();
   stopThinking();
   stopReading();
   enableCommonSense(false);
}
Posted in geek | Tagged | Leave a comment