Bluish Coder

Random and Binary IO using Iteratees

I wrote previously about interesting developments related to left folds and i/o. Oleg Kiselyov has come up with more Iteratee based goodness involving binary i/o and random access.

Iteratees presuppose sequential processing. A general-purpose input method must also support random IO: processing a seek-able input stream from an arbitrary position, jumping back and forth through the stream. We demonstrate random IO with iteratees, as well as reading non-textual files and converting raw bytes into multi-byte quantities such as integers, rationals, and TIFF dictionaries. Positioning of the input stream is evocative of delimited continuations. ... We show a representative application of the library: reading a sample TIFF file, printing selected values from the TIFF dictionary, verifying the values of selected pixels and computing the histogram of pixel values. The pixel verification procedure stops reading the pixel matrix as soon as all specified pixel values are verified. The histogram accumulation does read the entire matrix, but incrementally. Neither pixel matrix processing procedure loads the whole matrix in memory. In fact, we never read and retain more than the IO-buffer-full of raw data.

Firefogg - an extension to encode and upload ogg videos

I got an email today about a great new extension that is being worked on to help with sites supporting Ogg Theora videos. Firefogg is a Firefox extension to allow selecting a video, encoding it into Ogg format, and uploading it to a server. Sites supporting this extension can receive videos transcoded to Ogg with all the work to do the transcoding done by the client.

The site has details on the client side JavaScript API as well as source for some server side examples using PHP and Django. The extension itself is open source. I plan to add support for this to tinyvid.tv to enable video uploads in Theora format.

Note that the extension requires a nightly build of Firefox 3.1.

SRT Subtitles with HTML5 video

j^, the author of many cool open video related projects (including the Theora transcoder ffmpeg2theora) has written a jQuery based library to extract and show subtitles while a <video> is being played.

The jquery.srt page has details and a demo. Embedding code looks like:

<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="jquery.srt.js"></script>

<video src="http://example.com/video.ogv" id="video" controls>
<div class="srt"
     data-video="video"
     data-srt="http://example.com/video.srt" />

SRT is a common subtitle format and with this solution you can easily add subtitles to your HTML 5 videos.

Changes to git mirror of Firefox repository

I've made changes to the way I manage the git repository holding the import of the Firefox source. Since the bulk of the HTML 5 <video> implementation is now in the main Firefox source I no longer maintain the video branch. I'll be removing that and the Firefox mirror soon.

The main Mozilla repositories have been split into two now. They are mozilla-central, which holds the current trunk code and represents Firefox 3.2 (or whatever the next version will be called), and mozilla-1.9.1, which is what will be Firefox 3.1. The details of that split are described in this post.

I now import these mercurial repositories directly into git using git's fast import tool. This gives me complete history and makes things like bisect and blame a bit easier to work with. The history of mozilla-1.9.1 being based on mozilla-central is also retained, as best as I could manage. These imported repositories can be browsed at http://gitweb.bluishcoder.co.nz. They can be retrieved via git:

git clone git://bluishcoder.co.nz/git/mozilla-central.git
git clone git://bluishcoder.co.nz/git/mozilla-1.9.1.git

The way I work with these mirrors is to create a git repository that has these added as remotes. This way I can switch between the two easily, cherry pick patches between them, and do logs and diffs between them. This is how I set that up:

$ mkdir firefox
$ cd firefox
$ git init
$ git remote add mozilla-central git://bluishcoder.co.nz/git/mozilla-central.git
$ git remote add mozilla-1.9.1 git://bluishcoder.co.nz/git/mozilla-1.9.1.git
$ git fetch mozilla-central
$ git fetch mozilla-1.9.1

When working on a bug I create a branch for that bug, based on mozilla-central/master:

$ git checkout -b bug123456 mozilla-central/master
...make changes...
$ git commit -a -m "my changes"
$ git diff -U8 mozilla-central/master >bug123456.patch
...attach bug123456.patch to bugzilla for review...

Once reviewed the patch is then applied to mercurial and pushed. If I want to apply the same patch to test on the mozilla-1.9.1 branch I just use 'git cherry-pick'. To make things like 'git status' a bit less clutted I have the following .gitignore:

$ cat .gitignore
obj-*
*~
.mozconfig
config.cache
config.log
*.pyc
CVS
.mozconfig*
.svn
cvsco.log
cvsco.log.old
NONE
configure
gitchangelog
gitchangelog.bak
.hg
.nss.checkout
*.swp
.gitignore

Installing Arch Linux on the Asus EEE PC

I have a first generation Asus EEE PC (the 701 model) that I haven't really used much, mainly because I never got around to installing a more full featured Linux distribution on it than the one comes installed by default. I originally wanted it as a small portable machine to hack Factor and other languages on. Tonight I had a go at installing Arch Linux on it.

There is a good guide to installing Arch Linux on the EEE which I mostly followed. The basics of getting Arch up and running were pretty easy.

First step is to download the USB key version of the Core install disk. I copied this to a USB key using dd:

dd if=archlinux-2008.06-core-i686.img of=/dev/sdb

If you're doing this yourself, replace '/dev/sdb' with the device for your plugged in and unmounted USB key.

With the USB key plugged into the EEE PC, reboot it and press ESC at the bios screen. This will give an option to select to boot from the key. Choose that and Arch Linux will boot. You'll need to have a wired internet connection for this first part of the installation. The standard kernel on the USB key includes drivers for the wired ethernet, but not for the wifi.

Follow the standard install process for Arch Linux. When configuring the disk partitioning I removed all existing partitions and created one primary partition. Don't create a swap partition, following the caveats from the install guide. For the same reason choose ext2 for the filesystem.

When selecting the packages to install, ensure you choose the atl2 drivers, dhcpcd and wireless_tools. The first is so you get networking via the wired ethernet when you boot into the newly installed Arch Linux, the second for network configuration and the third is to enable configuring of the wireless connection.

Once the install is done, reboot. Everything should boot fine into the new Arch Linux install. The network should be working. If not run 'dhcpcd' to get connected.

To get wireless working download and install toofishes eeepc kernel:

# wget http://code.toofishes.net/packages/eee/kernel-eee-2.6.27.6-3-i686.pkg.tar.gz
# pacman -U kernel-eee-2.6.27.6-3-i686.pkg.tar.gz

This kernel contains everything needed for the hardware on the EEE PC. It worked for me anyway. Once installed, enable it in grub by adding something like the following added to /boot/grub/menu.lst:

# Arch Linux eeepc
title Arch Linux eeepc
root (hd0,0)
kernel /boot/vmlinuzeee root=/dev/sda1 ro

Reboot and choose this option. To setup the wifi use:

# ifconfig wlan0 up
# iwconfig wlan0 essid your_ssid_here

If you use WPA authentication then you need to set up wpa_supplicant. I did this by backing up my /etc/wpa_supplicant.conf and creating a new one with:

# wpa_passphrase your_ssid_here your_passphrase_here >/etc/wpa_supplicant.conf

Edit /etc/wpa_supplicant.conf and add at the top:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=wheel

Now you can associate with your chosen access point using WPA:

wpa_supplicant -D wext -i wlan0 -c /etc/wpa_supplicant.conf &
...wait a few seconds for it to connect...
dhcpd wlan0

This information was obtained from the Arch Linux wpa_supplicant docs Occasionally wpa_supplicant gives some errors but they appear to be safely ignorable since it still works. You can check if you're associated with the access point using:

# ifconfig wlan0

Once connected, update Arch Linux to the latest version:

# pacman -Syu

There's more to do to get the camera and sound going, and to install X, but that's the basics to get things up and running - as far as I've gotten so far. Total disk usage at this point is 671M leaving most of my 4GB free. I'm sure that'll change once I add X into the equation.