Bluish Coder: haskell

Random and Binary IO using Iteratees

2008-12-08T02:31:00+13:00

I wrote previously about interesting developments related to left folds and i/o. Oleg Kiselyov has come up with more Iteratee based goodness involving binary i/o and random access.

Iteratees presuppose sequential processing. A general-purpose input method must also support random IO: processing a seek-able input stream from an arbitrary position, jumping back and forth through the stream. We demonstrate random IO with iteratees, as well as reading non-textual files and converting raw bytes into multi-byte quantities such as integers, rationals, and TIFF dictionaries. Positioning of the input stream is evocative of delimited continuations. ... We show a representative application of the library: reading a sample TIFF file, printing selected values from the TIFF dictionary, verifying the values of selected pixels and computing the histogram of pixel values. The pixel verification procedure stops reading the pixel matrix as soon as all specified pixel values are verified. The histogram accumulation does read the entire matrix, but incrementally. Neither pixel matrix processing procedure loads the whole matrix in memory. In fact, we never read and retain more than the IO-buffer-full of raw data.

Dynamic Compilation and Loading of Modules in Haskell

2008-11-25T02:09:00+13:00

The Haskell system GHC has libraries that provide the ability to compile Haskell code and dynamically load it into a running Haskell program. A library that provides this functionality is hs-plugins. Unfortunately hs-plugins doesn't work with the latest GHC release, 6.10.1.

There is an API to the internals of GHC that allows implementing runtime compilation and loading. The api is documented but there aren't very many usage examples that work with the latest GHC release. I stumbled across a blog post which no longer exists describing an older version of the API, but managed to retrieve it from archive.org which provided a lot of help since the API's are similar.

As a means to learn how to use the API I decided on a very simple use case of compiling and loading a very simple Haskell module containing a single function 'print' that outputs a string. The module looks like this:

module Test (Test.print) where

print :: String -> IO ()
print x = 
  putStrLn x

To compile and load this code dynamically we need a compilation target. This is created using the function guessTarget. It's passed a string that refers to the name of a module, or the name of a source file. By using a source file we can tell the Haskell system to compile and load it. The phase argument I set to Nothing as I have no idea what that is.

Once we have the target it needs to be added to the current compilation session. This is done via the addTarget function. When all the targets are added, a call to load will do the equivalent of a ghc --make to build the module if needed. So all the relevant target code looks like:

do
  target <- guessTarget "Test.hs" Nothing
  addTarget target
  load LoadAllTargets

Notice this is run within the 'do' syntax. This is because all these compiler functions must be called with a special Monad, called GhcMonad. The way this works is described later but just imagine that the GhcMonad is implicitly passed to all the Ghc functions we are calling automatically.

'load' returns a success flag that can be pattern matched on to determine if it succeeds or failed. If it succeeds we can start using the new module. To be able to access exported functions from the module we need to get a reference to the module using findModule and use setContext on it. 'setContext' takes two arrays containing modules. All modules in the first array will have their top level scope available. All modules listed in the second array will have only their exports available. In this example we want to access the 'print' function of 'Test', which is exported.

The function compileExpr is used to compile a string containing a Haskell expression. The result of this is an HValue which we can do things with. The expression that is compiled can access any of the bindings available in the context we set up via 'setContext'. So passing "Test.print" to 'compileExpr' will return the function 'print' from the 'Test' module that we dynamically built and loaded. Code similar to this:

do
  r <- load LoadAllTargets
  case r of
    Failed -> error "Compilation Failed"
    Succeeded -> do
      m <- findModule (mkModuleName "Test") Nothing
      setContext [] [m]
      value <- compileExpr "Test.print"

Unfortunately the result of 'compileExpr' is of no use to us as an HValue if we want the actual function type so we can call it. Haskell being strongly typed we have to 'cheat' by telling Haskell 'yes, I know you think this is an HValue, but I can assure you, it's really a function'. This is Haskell's super dangerous Unsafe.Coerce. It allows us to coerce from one type to another and is, as the name implies, completely unsafe. If the type you are coercing isn't really of the type you are coercing too then you'll very likely crash your program and/or corrupt data. It's the reinterpret_cast<> of the Haskell world. But for handling dynamic loading of code it seems exactly what we want. I'm not sure of any other way to convert the HValue type.

The type of 'print' in the 'Test' module is 'String -> IO ()'. 'unsafeCoerce' can be called to convert the 'HValue' to this:

value <- compileExpr "Test.print"
  do let value' =  (unsafeCoerce value) :: String -> IO ()
     return value'

The receiver of the result, value', can now call the function as it knows its type. The complete code for this example is:

import GHC
import GHC.Paths
import DynFlags
import Unsafe.Coerce

main :: IO ()
main =
    defaultErrorHandler defaultDynFlags $ do
      func <- runGhc (Just libdir) $ do
        dflags <- getSessionDynFlags
        setSessionDynFlags dflags
        target <- guessTarget "Test.hs" Nothing
        addTarget target
        r <- load LoadAllTargets
        case r of
          Failed -> error "Compilation failed"
          Succeeded -> do
            m <- findModule (mkModuleName "Test") Nothing
            setContext [] [m]
            value <- compileExpr ("Test.print")
            do let value' = (unsafeCoerce value) :: String -> IO ()
               return value'
      func "Hello"
      return ()

Build using:

$ ghc -package ghc --make Api.hs
[1 of 1] Compiling Main             ( Api.hs, Api.o )
Linking Api ...
$ ./Api
Hello
$

You can test that it is dynamically compiling and loading by modifying 'Test.hs' so the output is different. Change the 'putStrLn' to putStrLn $ x ++ " World!" and rerun 'Api' without recompiling it:

$ ./Api
Hello World!
$

I mentioned previously the implicit 'GhcMonad' that is passed to each GHC API function. This is provided by the runGhc word. It takes a pointer to the directory where GHC's library files reside. In this example I use the GHC.Paths (available from Hackage package to deal with this using 'libdir'. The second argument is the action to perform, the GhcMonad, and is built using the 'do' syntax. Haskell's 'do' syntax is syntactic sugar that does the magic of threading the GhcMonad state holding the targets, modules, contexts, etc through the various function calls. Find a favourite Monad tutorial/explanation if you want to know more.

'defaultErrorHandler' sets up the environment to handle compilation errors in a standard way. The 'DynFlags' related calls get and set standard flags that can be used to configure the compilation. You can set the equivalent of GHC arguments to control extra features that GHC can use, or use the interpreter (as used by GHCi) rather than the compiler.

Hopefully that helps explain some of what is going on and how to use the current GHC api. I'd appreciate comments on better ways of doing things, and other examples of the API.

The Left Fold Enumerator for i/o

2008-10-11T00:47:00+13:00

There have been some interesting papers and talks about approaches to handling i/o using left folds recently.

First was the galois tech talk about a safe and efficient i/o interface in Haskell using left fold enumerators. PDF slides are here. The example web server, Hyena, is available on github.

Oleg Kiselyov then gave a talk at DEFUN about using left folds for web servers. The slides and source for the talk are available at Oleg's site in the Haskell Iteratee I/O section.

We explain input processing with left-fold enumerator, using as an example HTTP request processing in Haskell. The approach is general and applies to processing data from various collections, from in-memory data structures to databases, files, sockets, etc.

Our approach differs in: permitting incremental processing; i/o interleaving comes by default; we shall see an example of i/o multiplexing with no need for threads and related locking and synchronization.

Unlike lazy IO, our approach is correct. There is not even hint of UnsafePerformIO. Unlike Handle-based IO, accessing a disposed resource like a closed handle is just impossible in our approach. Our approach has some other nice properties, permitting composing streams and stream processors.

One can use the same processor to handle several streams one after another. Or use two processors to process parts of the same source. One can combine processors vertically, which is very useful when one stream is embedded (chunk-encoded, escaped, UTF8-encoded) into another. Enumerators and iteratees, which generalize fold, have nice algebraic properties. But we won't talk about them.

Interesting weekend reading!

Minix and Haskell

2006-06-07T20:22:00+12:00

There's an interesting post on the Haskell group about Minix and the possibility of writing OS core components in Haskell for it.

Minix is an open source operating system that's been around for quite some time. I first tried it out sometime in the early 90's after downloading over some very slow dialup connection connected to a BBS.

From the post:

Andrew Tanenbaum said that it is important for a language like OCaml or Haskell to get more visibility among the OS developers; and Minix offers a very good way to do that. Minix3 is based on micro-kernel; all of the OS services (memory manager, file system, all the drivers, etc.) run as regular processes communicating through a well-defined protocol. It is irreleveant what language these services are written in, so long as they obey the protocol.

The poster goes on to suggest a way of slowly migrating the system routines to Haskell:

With Minix, we can replace one kernel service of the full-fledged, working OS with the one written in Haskell -- and immediately see how it all works. If it doesn't, the rest of OS still works and so we can unload the faulty service and load another implementation. Minix3 does indeed offer an easy road to the OS kernel for a language other than C. I said that there may be quite a bit of interest in Haskell community in such a project. I wonder if I'm right...

Linspire to use Haskell

2006-05-22T19:48:00+12:00

According to the Haskell Weekly News, the OS team at Linspire are standardising on on Haskell as their preferred language for core OS development. Linspire is a Linux distribution.

The announcement from the Linspire team is here.

Playing with HAppS

2006-05-10T05:55:00+12:00

I downloaded HAppS, the Haskell Application Server, I mentioned previously to try it out.

I already had GHC installed so I could build darcs but I needed a few additional packages to install HAppS. Most of these I obtained via apt-get but I had to manually install the following two:

fps
HaXml

Once they were installed compiling and building HAppS is done using the Cabal based setup file:

runhaskell Setup.hs configure
runhaskell Setup.hs build
sudo runhaskell Setup.hs install

Once that's out of the way I tried out the 'hello' example. This is located in the 'examples' subdirectory of 'HAppS' and needs to be built and run:

make
./hello

This starts up an application server listening on port 8000. Where to go from there in terms of building apps I'm not sure as there doesn't seem to be much 'how to' style documentation. Digging through the source and examples looks to be the next thing to try.

Unifying Events and Threads in Haskell

2006-05-03T08:16:00+12:00

Update: 3 May 2006: Source is now available

Lambda the Ultimate points to a great paper, A Language-based Approach to Unifying Events and Threads, by Peng Li and Steve Zdancewic.

This paper presents a language based technique to unify two seemingly opposite programming models for building massively concurrent network services: the event driven model and the multithreaded model. The result is a unified concurrency model providing both thread abstractions and event abstractions.

The paper is about the implementation of the unified concurrency model in Haskell. The threads are 'lightweight' threads and according to the paper scale to 10 million threads. The scheduler outperforms NPTL in I/O benchmarks.

Monads provide the thread abstraction by defining an imperative sub-language of Haskell with system calls and thread control primitives.

Higher-order functions provide the internal representation of threads in continuation passing style.

Lazy data structures provide the event abstraction, which is a lazy tree that represents the trace of system calls generated by threads.

Some interesting benchmark results too. One benchmark test they did involved launching 10 million threads that just yield within a loop. This was running on a machine with 1GB set aside for the GHC heap. The per thread cost was only 48 bytes!

Their multiprocessor benchmark, yielded a 3.65 times speedup over a non-MP version version on a 4CPU machine (The library is able to take advantage of SMP machines).

Other interesting things in the paper are a web server they built using this framework and an application level TCP stack written entirely in Haskell.

I hope the source for the things mention in this paper becomes available. In the meantime, you can read it here.

Source for Unify Available

2006-05-03T08:15:00+12:00

The source for the system presented in the paper 'A Language Based Approach to Unifying Events and Threads' is now available.

Download information is available at the Unify home page.

The source code package can be browsed online here. It contains the application-level thread library, a simple web server and a HTTP load generator. For more information, see the READ_ME file.

To compile and run the source code, you will need the Linux 2.6 kernel, the most recent development snapshot of GHC 6.5, and the Linux asynchronous I/O library (libaio).

(From Lambda the Ultimate).

Space Invaders Emulator in Haskell

2006-05-02T01:03:00+12:00

I been tinkering with Haskell lately to get familiar with a purely functional programming language. As a project to work on I'm trying to port the Space Invaders emulator I wrote for Factor to Haskell.

I'm pretty much a complete newbie to Haskell and have no idea on the best ways to implement this type of thing. I'm starting simple, seeing where the bottlenecks are, and will improve the code from there.

For representing the memory I'm using a standard Haskell Array type. Being a purely functional programming language this means that any updates to the array result in it being copied and the copy being returned. If this turns out to be too inefficient there are some variations on this array type, including UArray for unboxed storage of values.

My first cut at an 8080 CPU abstraction was:

data Cpu = 
 Cpu { a  :: Word8, 
       b  :: Word8, 
       c  :: Word8, 
       d  :: Word8, 
       e  :: Word8, 
       f  :: Word8, 
       h  :: Word8, 
       l  :: Word8, 
       pc :: Word16,
       sp :: Word16,
       doInterrupts :: Bool,
       mem :: Memory }

Functions that set register values in the Cpu type actually return a new Cpu with the old values copied across and the new value set. 'Memory' is the array type:

type Memory = Array Word16 Word8

This represents a mapping from 16 bit numbers (the memory address) to 8 bit values. To make the code easier to read I also created some data types to refer to the 8080 registers:

data Register16 = BC | DE | HL | SP | PC deriving (Show)
data Register8  = A | B | C | D | E | F | H | L deriving (Show)

To implement the instruction set I created an 'Instruction' datatype with constructors for each possible type of 8080 instruction. I got this idea from the OmegaGB Gameboy Emulator in Haskell. Much thanks to the author of that for making their developer log and design ideas available. As an example, part of my 'Instruction' type is:

data Instruction =
   MoveRegister   Register8 Register8 -- MOV r1,r2 
 | MoveFromMemory  Register8   -- MOV r,M 
 | MoveToMemory   Register8  -- MOV M,r 
 | MoveImmediate   Register8 Word8  -- MVI r,data 
 | MoveToMemoryImmediate  Word8   -- MVI M,data 
 | LoadRegisterPairImmediate Register16 Word16 -- LXI rp,data16
 | LoadAccumulatorDirect  Word16   -- LDA addr
 | StoreAccumulatorDirect Word16   -- STA addr
 | LoadHAndLDirect  Word16   -- LHLD addr
 | StoreHAndLDirect  Word16   -- SHLD addr
 | ...

My usual approach for decoding the machine code opcodes into these Instruction types would be to create a function that maps each opcode from 0 through to 255 to one of the instructions. Instead of doing this I went to the 8080 instruction set reference and created Haskell overloaded functions that work on the binary of the opcode and decode the registers,etc to operate on from that. So some of these functions look like:

opcodeToInstruction (Binary 0 1 d3 d2 d1 1 1 0) _ _ = 
 let r = register8Lookup (d3,d2,d1) in
   MoveFromMemory r

opcodeToInstruction (Binary 0 1 1 1 0 s3 s2 s1) _ _ = 
 let r = register8Lookup (s3,s2,s1) in
   MoveToMemory r

opcodeToInstruction (Binary 0 1 d3 d2 d1 s3 s2 s1) _ _ = 
 let r2 = register8Lookup (s3,s2,s1)
     r1 = register8Lookup (d3,d2,d1) in
   MoveRegister r1 r2

opcodeToInstruction (Binary 0 0 1 1 0 1 1 0) b2 _ = 
   MoveToMemoryImmediate b2

The 'register8Lookup' uses the binary string for the register as it is encoded in the opcode to return the actual register type:

register8Lookup :: (Integer,Integer,Integer) -> Register8
register8Lookup (1,1,1) = A
register8Lookup (0,0,0) = B
register8Lookup (0,0,1) = C
register8Lookup (0,1,0) = D
register8Lookup (0,1,1) = E
register8Lookup (1,0,0) = H
register8Lookup (1,0,1) = L

This had the advantage of letting the Haskell compiler work out exactly what instruction to decode the opcode value too. It's made the code quite readable.

A 'runInstruction' function is used to execute the instruction and modify the Cpu type. Some examples of implementations are:

runInstruction :: Cpu -> Instruction -> Cpu
-- Data transfer group
runInstruction cpu (MoveRegister r1 r2) =
 setRegister8 cpu r1 (getRegister8 cpu r2)

runInstruction cpu (MoveFromMemory r) =
   let hl = getRegister16 cpu HL
       v  = readMemory8 cpu hl in
     setRegister8 cpu r v

runInstruction cpu (MoveToMemory r) =
   let hl = getRegister16 cpu HL
       v  = getRegister8 cpu r in
         writeMemory8 cpu hl v

runInstruction cpu (MoveImmediate r v) =
   setRegister8 cpu r v

runInstruction cpu (MoveToMemoryImmediate v) =
   let hl = getRegister16 cpu HL in
     writeMemory8 cpu hl v

-- Branch Group
runInstruction cpu (Jump addr) = 
 setRegister16 cpu PC addr

runInstruction cpu (ConditionalJump condition addr) = 
   let c = isConditionTrue cpu condition in
     if c then setRegister16 cpu PC addr else cpu

The code that looks up the machine code byte from memory converts it to the 'Binary' datatype and calls 'opcodeToInstruction' to get the instruction out of it:

toBinary :: Word8 -> Binary
opcodeToInstruction :: Binary -> Word8 -> Word8 -> Instruction
readMemory8 :: Cpu -> Word16 -> Word8

step1 :: Cpu -> Cpu
step1 cpu = 
  let addr = pc cpu
      opcode  = readMemory8 cpu addr
      b2 = readMemory8 cpu (addr+1)
      b3 = readMemory8 cpu (addr+2) 
      instruction = opcodeToInstruction (toBinary opcode) b2 b3 in
    runInstruction cpu instruction

With all the instructions implemented I ran a simple benchmark of 1,000,000 instructions. Using '-O2' with GHC I got approximately 20 seconds on my machine (an Athlon 2500+) which is a tad slow.

As a comparison, my Factor implementation runs 1,000,000 instructions in about 1.7 seconds.

An easy change to make was to change the memory type to use a UArray which is an unboxed array. This immediately change the timing to 1.6 seconds. Which is much better than I expected. Remember that this is unoptimised code and purely functional. The only impure code is the I/O printing the CPU registers after each instruction (which I piped to /dev/null for the timings).

The implementation currently emulates about 90% of the 8080 CPU instruction set. I still need to handle interrupts and then add a simple GUI. Once that's done I'll make the code available.

Overall I found Haskell quite easy to develop with. I used 'ghci', the interpreter that is part of GHC, for most of the testing and iterative development. Using that was a lot like developing with a Lisp REPL. The static typing didn't get in my way and in fact it pointed to errors I'd made at compile time instead of run time in a lot of cases. I'll see if that remains the case when I tackle the GUI though which is likely to be a bit more difficult.

Building wxFruit and Yampa with GHC 6.4.x

2006-04-24T15:52:00+12:00

wxFruit is a Haskell GUI library based on Functional Reactive Programming concepts. It uses wxWindows as its underlying GUI framework. For FRP it uses the Haskell FRP framework Yampa.

Both libraries are relatively old and seem to have suffered bitrot as I couldn't get them to compile using a recent GHC build (GHC 6.4.x). A bit of searching came across this post to the haskell-jp list which had some patches to enable these to build. I've extracted the relevant patches here:

These can be applied by using 'patch' from within the original wxFruit and Yampa distributions:

cd afrp-0.4
patch -p1 &lt;afrp.diff
cd ../wxfruit-0.1
patch -p1 &lt;wxfruit.diff

Alternatively you can download the already patched versions I've made here:

To build and install the patched Yampa, the following should work:

cd afrp-0.4
runhaskell Setup.hs configure
runhaskell Setup.hs build
sudo runhaskell Setup.hs install

Once this is done the wxFruit paddle example builds with:

cd ../wxfruit-0.1
ghc --make -farrows -o paddle paddle.hs