Bluish Coder: fxos

Update on Tor on Firefox Proof of Concept

2014-06-13T18:00:00+12:00

Yesterday I wrote about Tor on Firefox OS. Further testing showed an issue when switching networks - a common thing to happen when carrying a mobile device. The iptables rule I was using didn't exclude the tor process itself from having traffic redirected. When a network switch occurred tor would attempt to reestablish connections and this would fail.

A fix for this is to exclude tor from the iptables rules or to use rules for specific processes only. The processes that belong to an Firefox OS application be be viewed with b2g-ps:

APPLICATION    SEC USER     PID   PPID  VSIZE  RSS     NAME
b2g              0 root      181   1     494584 135544 /system/b2g/b2g
(Nuwa)           0 root      830   181   55052  20420  /system/b2g/plugin-container
Built-in Keyboa  2 u0_a912   912   830   67660  26048  /system/b2g/plugin-container
Vertical         2 u0_a1088  1088  830   103336 34428  /system/b2g/plugin-container
Usage            2 u0_a4478  4478  830   65544  23584  /system/b2g/plugin-container
Browser          2 u0_a26328 26328 830   75680  21164  /system/b2g/plugin-container
Settings         2 u0_a27897 27897 830   79840  28044  /system/b2g/plugin-container
(Preallocated a  2 u0_a28176 28176 830   62316  18556  /system/b2g/plugin-container

Unfortunately the iptables that ships with Firefox OS doesn't seem to support the --pid-owner option for rule selection so I can't select specifically the tor or application processes. I can however select based on user or group. Each application gets their own user so the option to redirect traffic for applications can use that. I wasn't able to get this working reliably though so I switched to targeting the tor process itself.

In my writeup I ran tor as root. I need to run as a different user so that I can use --uid-owner on iptables. Firefox OS inherits the Android method of users and groups where specific users are hardcoded into the system. Since this is a proof of concept and I want to get things working quickly I decided to pick an existing user, system, and run tor as that. By setting the User option in the Tor configuration file I can have Tor switch to that user at run time. Nothing is ever that easy though as user does not have permission to do the many things that tor requires. It can't create sockets for example.

Enter Linux capabilities. It is possible to grant a process certain capabilities which give it the right to perform priviledged actions without being a superuser. There is an existing Tor trac ticket about this and I used the sample code in that ticket to modify tor to keep the required capabilities when it switches user, I put the code I cobbled together to patch tor in tor.patch.

To use this change the Building tor section of my original post to use these commands:

$ cd $HOME/build
$ wget https://www.torproject.org/dist/tor-0.2.4.22.tar.gz
$ cd tor-0.2.4.22
$ curl http://bluishcoder.co.nz/b2g/tor.patch | patch -p1
$ ./configure --host=arm-linux-androideabi \
              --prefix=$HOME/build/install \
              --enable-static-libevent
$ make
$ make install

Change the Tor configuration file to switch the user to system in the Packaging Tor for the device section:

DataDirectory /data/local/tor/tmp 
SOCKSPort 127.0.0.1:9050 IsolateDestAddr
SOCKSPort 127.0.0.1:9063
RunAsDaemon 1
Log notice file /data/local/tor/tmp/tor.log
VirtualAddrNetwork 10.192.0.0/10
AutomapHostsOnResolve 1
TransPort 9040
DNSPort 9053
User system

I've also changed the location of the data files to be in a tmp directory which needs to be given the system user owner. Change the steps in Running tor to:

$ adb shell
# cd /data/local/tor
# mkdir tmp
# chown system:system tmp
# ./tor -f torrc &
# iptables -t nat -A OUTPUT ! -o lo
            -m owner ! --uid-owner system \
            -p udp --dport 53 -j REDIRECT --to-ports 9053 
# iptables -t nat -A OUTPUT ! -o lo \
           -m owner ! --uid-owner system \
            -p tcp -j REDIRECT --to-ports 9040

Now tor should work in the presence of network switching. I've updated the b2g_tor.tar.gz to include the new tor binary, the updated configuration file, and a couple of shell scripts that will run the iptables commands to redirect traffic to tor and to cancel the redirection.

As before the standard disclaimer applies:

All files and modifications described and provided here are at your own risk. This is a proof of concept. Don't tinker on devices you depend on and don't want to risk losing data. These changes are not an official Mozilla project and do not represent any future plans for Mozilla projects.

This is probably as far as I'll take things for now with this proof of concept and see what happens from here after using it for a while.

Using Tor with Firefox OS

2014-06-12T18:00:00+12:00

Update - Please read my followup post for some additional information and updated steps on building and installing tor on Firefox OS.

Please read the disclaimer at the end of this article. This is a proof of concept. It's a manual process and you shouldn't depend on it. Make sure you understand what you are doing.

I'm a fan of Tor. The Tor site explains what it does:

Tor is free software and an open network that helps you defend against traffic analysis, a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and state security.

I make my personal website available as a Tor hidden service accessible from mh7mkfvezts5j6yu.onion. I try to make other sites I'm involved with also have a presence over Tor. I do a fair amount of my browsing over the Tor network for no reason other than I can and it limits the opportunity for people snooping on my data.

I want to be able to use Tor from Firefox OS. In particular I want it embedded as low level as possible so I have the option of all traffic going over Tor. I don't want to have to configure socks proxies.

Firefox OS doesn't allow native applications. The low level underlying system however is based on Linux and Android and can run native binaries. Starting with a rooted Firefox OS install I built Tor and used iptables to reroute all network traffic to work over it. This is a first step and is what this article demonstrates how to get going so power users can try it out. My next step would be to investigate integrating it into the build system of Firefox OS and providing ways to start/stop it from the OS interface.

The first stage of building is to have an Android standalone toolchain installed. I describe how to do this in my Wasp Lisp on Android post or you can use a Nix package I created for use with the Nix package manager.

Building libevent

Tor requires libevent to build. I'm using static libraries to make creating a standalone tor binary easier. The following will build libevent given the standalone toolchain on your path:

$ cd $HOME
$ mkdir build
$ cd build
$ wget https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz
$ tar xvf libevent-2.0.21-stable.tar.gz
$ cd libevent-2.0.21-stable
$ ./configure --host=arm-linux-androideabi \
              --prefix=$HOME/build/install \
              --enable-static --disable-shared
$ make
$ make install

Building zlib

Tor requires openssl which in turn requires zlib:

$ cd $HOME/build
$ wget http://zlib.net/zlib-1.2.8.tar.gz
$ tar xvf zlib-1.2.8.tar.gz
$ cd zlib-1.2.8
$ CC=arm-linux-androideabi-gcc ./configure --prefix=$HOME/build/install --static
$ make
$ make install

Building openssl

$ cd $HOME/build
$ wget http://www.openssl.org/source/openssl-1.0.1h.tar.gz
$ tar xvf openssl-1.0.1h.tar.gz
$ cd openssl-1.0.1h
$ CC=arm-linux-androideabi-gcc ./Configure android no-shared --prefix=$HOME/build/install
$ make
$ make install

Building tor

$ cd $HOME/build
$ wget https://www.torproject.org/dist/tor-0.2.4.22.tar.gz
$ cd tor-0.2.4.22
$ ./configure --host=arm-linux-androideabi \
              --prefix=$HOME/build/install \
              --enable-static-libevent
$ make
$ make install

Packaging Tor for the device

To run on the Firefox OS device I just installed the tor binary and a configuration file that enables transaparent proxing as per the Tor documentation on the subject. I put these in a directory that I push to an easily accessible place on the device:

$ mkdir $HOME/build/device
$ cd $HOME/build/device
$ cp $HOME/build/install/bin/tor .
$ cat >torrc
  ...contents of configuration file...
$ adb push $HOME/build/device /data/local/tor

The configuration file is:

DataDirectory /data/local/tor
Log notice file /data/local/tor/tor.log
RunAsDaemon 1
SOCKSPort 127.0.0.1:9050 IsolateDestAddr
SOCKSPort 127.0.0.1:9063
VirtualAddrNetwork 10.192.0.0/10
AutomapHostsOnResolve 1
TransPort 9040
DNSPort 9053

Running tor

I haven't integrated tor into the device at all so for this proof of concept I adb shell into it to run it and configure the iptables to redirect traffic:

$ adb shell
# cd /data/local/tor
# ./tor -f torrc &
# iptables -t nat -A OUTPUT ! -o lo -p udp --dport 53 -j REDIRECT --to-ports 9053 
# iptables -t nat -A OUTPUT ! -o lo -p tcp -j REDIRECT --to-ports 9040

Testing

The device should now be sending traffic over Tor. You can test by visiting sites like whatismyip.com or icanhazip.com to see if it reports a different IP address and location to what you normally have. You can also try out hidden services like mh7mkfvezts5j6yu.onion which should show this site.

Removing

Killing the Tor process and removing the iptables entries will set the network back to normal:

$ adb shell ps|grep tor
$ adb shell
# kill ...process id of tor...
# iptables -t nat -F

You can optionally delete the /data/local/tor directory to remove all tor files:

$ adb shell rm -r /data/local/tor

Future

This is just a proof of concept. Don't depend on this. You need to restart Tor and the iptables commands on reboot. I'm not sure how well interaction with switching to/from WiFi and GSM works. Ideally Tor would be integrated with Firefox OS so that you can start and stop it as a service and maybe whitelist or blacklist sites that should and shouldn't use Tor. I hope to do some of this over time or hope someone else gets excited enough to work on it too.

Another privacy aspect I'd like to investigate is whether TextSecure (or a similar service) could be integrated in the way it's done in CyanogenMod:

"The result is a system where a CyanogenMod user can choose to use any SMS app they'd like, and their communication with other CyanogenMod or TextSecure users will be transparently encrypted end-to-end over the data channel without requiring them to modify their work flow at all."

Ideally my end goal would be to have something close to that described in the hardening Android post on the Tor Project blog.

I'm not sure how possible that is though. But Firefox OS is open source, easy to build and hack on, and runs on a lot of devices, including multi booting on some. Adding things like this to build your own custom phone OS that runs web applications is one of the great things the project enables. Users should feel like they can dive in and try things rather than wait for an OS release to support it (in my opinion of course).

Test Builds

A tar file containing a precompiled tor and the torrc is available at b2g_tor.tar.gz.

Disclaimer

All files and modifications described and provided here are at your own risk. Don't tinker on devices you depend on and don't want to risk losing data. These changes are not an official Mozilla project and do not represent any future plans for Mozilla projects.

Dual Booting Android and Firefox OS on the Nexus 5

2014-06-11T18:00:00+12:00

I've gone through periods of using a Firefox OS phone as my main device but I've usually fallen back to Android due to needing to use some Android only programs and I don't like carrying two phones around. Today I decided to investigate how to get dual boot Android with custom Firefox OS builds. Thankfully it was actually pretty easy.

The boot manager I used to get this to work is MultiROM Manager, available from the Play store for rooted phones. The source is for MultiROM Manager is available on github. The phone I used was the Nexus 5. The instructions here assume you are familiar with adb and fastboot already.

Be aware that all these changes may lose the data you have on the device if you haven't already unlocked the boot loader and rooted the device.

Make a backup of your Android settings and applications

With the device plugged in and visible from adb:

$ adb backup -apk -shared -all

This can be restored later if needed with:

$ adb restore backup.ab

Unlock the bootloader

The Nexus 5, and other Google devices, make it easy to unlock the bootloader. With the device plugged in and visible from adb:

$ adb reboot bootloader
$ fastboot oem unlock

Follow the screen instructions. This will erase everything on the device!

Rooting the Nexus 5

I used CF-Auto-Root. I downloaded the version for the Nexus 5 and used fastboot to boot the image inside of it:

$ unzip CF-Auto-Root-hammerhead-hammerhead-nexus5.zip
$ fastboot boot image/CF-Auto-Root-hammerhead-hammerhead-nexus5.img

The device will reboot and perform the steps necessary to root it.

Install MultiROM Manager

Install MultiROM Manager from the Play store. Run the app and choose Install after ticking the MultiROM, Recovery and Kernel check boxes. Follow the onscreen instructions.

Build Firefox OS

The Mozilla Developer Network has instructions for building Firefox OS. Assuming all the pre-requisites are installed the steps are:

$ git clone git://github.com/mozilla-b2g/B2G b2g
$ cd b2g
$ ./config.sh nexus-5
$ PRODUCTION=1 MOZILLA_OFFICIAL=1 ./build.sh

Don't flash the device from here. We'll create a MultiROM compatible ROM file to boot from.

Create Firefox OS ROM file

Create a directory to hold the ROM contents and copy the results of the build into it:

$ mkdir rom
$ cd rom
$ rsync -rL ../out/target/product/hammerhead/system .
$ rsync -rL ../out/target/product/hammerhead/data .
$ cp ../out/target/product/hammerhead/boot.img .

For the rsync copy I deliberately choose not to copy symbolic links and to instead re-copy the original file. I had difficulty getting symbolic links working and need to investigate.

An Android ROM requires a META-INF directory containing a script that performs the update process. The following commands create this directory, copy the binary to run the script and the script itself:

$ mkdir -p META-INF/com/google/android/
$ cp ../tools/update-tools/bin/gonk/update-binary META-INF/com/google/android/
$ curl http://bluishcoder.co.nz/b2g/updater-script >META-INF/com/google/android/updater-script

The updater script is one I wrote based on existing ones. It's pretty easy to follow if you want to read and change it.

The final step is to ZIP the directories, sign them and push to a directory on the device:

$ zip -r9 b2g.zip *
$ java -jar ../prebuilts/sdk/tools/lib/signapk.jar \
            ../build/target/product/security/testkey.x509.pem \
            ../build/target/product/security/testkey.pk8 \
             b2g.zip signed_b2g.zip
$ adb push signed_b2g.zip /sdcard/

Install Firefox OS ROM

Boot into recovery mode by pressing volume down and the power on button at the same time (or run adb reboot recovery). From the recovery menu choose 'Advanced' followed by 'MultiROM', then Add ROM.

Make sure Android is selected and Don't Share is chosen for "Share Kernel with Internal ROM". Click Next, choose Zip file and select the file we created in the signing step previously. Swipe to confirm as requested.

If this succeeds, Reboot and touch the screen during the 'Auto boot' display to get the list of ROMS to run. Choosing the one we just installed should boot Firefox OS.

Other ROMs

With MultiROM you can install other ROMS and even Ubuntu Touch. I'd like to get Inferno OS running under MultiROM as well so I can boot between all the operating systems I like to tinker with on one device.

Try it

I've placed a complete Firefox OS ROM for use with MultiROM on the Nexus 5 in b2g_nexus5.zip. This was built from B2G master branch so may be broken in various aspects (The camera doesn't work for example) but will allow you to try the multi boot process out if you can't do builds. This is not an official Mozilla build and was generated by me personally. Use at your own risk.

Running Firefox OS on Nexus S

2013-04-22T20:00:00+12:00

Update: 2013-05-07: Minor changes to deal with new and rebased patches.

Last year I posted about how to build and run B2G on the Nexus S. Much has changed since then and this post attempts to provide up to date instructions on building Firefox OS and installing the latest Gaia version.

I wrote 'attempts' because the support for the Nexus S bitrots frequently due to (I assume) most development effort going towards developer devices and devices that intend to be shipped. These instructions worked for me on my Nexus S and gives me a phone that can use Wifi, make calls, use 3G data, take photo's, record video and install applications.

Getting the source

The main source repository is on github, https://github.com/mozilla-b2g/b2g. The first step is to clone this repository. I clone this into a directory called nexus-s:

$ git clone git://github.com/mozilla-b2g/B2G nexus-s
$ cd nexus-s

Create a file in this directory called .userconfig containing the following:

export CC=gcc-4.6
export CXX=g++-4.6
export HIDPI=1

The first two entries are required if you are building on Ubuntu 12.10 to use gcc version 4.6, which you must install. B2G does not build using gcc version 4.7.

The last entry, HIDPI, results in applications using the correct resources for the screen size on the Nexus S. Without this many things will be scaled incorrectly.

Configuring

Once the source is obtained you need to pull down the sub-repositories required for the Nexus S. This step downloads multiple gigabytes of data and can take a long time:

nexus-s$ BRANCH=v1-train ./config.sh nexus-s

Note the use of the BRANCH environment variable. This configures our build to use the relatively stable v1-train branch. This branch gets regular updates but tends to be better tested than master which is often broken for not-quite-supported devices like the Nexus S.

Although we are using the v1-train we are going to use the master branch of Gaia so we get the latest and greatest user interface. We'll change to this later.

Gecko changes

The branch of Gecko that v1-train uses has some bugs on the Nexus S. To fix this we need to apply some patches. The first is bug 832653, video playback broken on Nexus S. The patches relevant to Gecko from this bug that need to be applied are:

Fix graphic buffer crash

The following set of commands downloads these patches and applies them in a branch of the Gecko tree that the config.sh step previously obtained:

nexus-s       $ cd gecko
nexus-s/gecko $ git checkout -b nexus-s m/v1-train
nexus-s/gecko $ curl -k -L https://bug832653.bugzilla.mozilla.org/attachment.cgi?id=734988 >p1.patch
nexus-s/gecko $ patch -p1 <p1.patch
nexus-s/gecko $ git commit -a -m "Nexus S patches"
nexus-s/gecko $ cd ..

Gonk changes

Part of bug 832653 makes a change to the low level Gonk layer of B2G. This is in the libstagefright video decoding libraries. The change is to prevent output buffer starvation which results in video decoding stopping after a few frames. The required patch is:

Fix output buffer starvation

To apply this patch:

nexus-s                 $ cd frameworks/base
nexus-s/frameworks/base $ git checkout -b nexus-s 
nexus-s/frameworks/base $ curl -k -L https://bugzilla.mozilla.org/attachment.cgi?id=735002 >p1.patch
nexus-s/frameworks/base $ patch -p1 <p1.patch
nexus-s/frameworks/base $ git commit -a -m "Nexus S patches"
nexus-s/frameworks/base $ cd ../..

Gaia changes

There are Gaia changes required as part of bug 869289 as well. This change prevents the error where videos are not listed if the thumbnail cannot be generated:

List videos even if thumbnail generation fails

We also want to be on the Gaia master branch to get the latest and greatest code. The following comands switch to this branch and apply our patch:

nexus-s      $ cd gaia
nexus-s/gaia $ git checkout -b nexus-s mozillaorg/master
nexus-s/gaia $ curl -k -L https://bug869289.bugzilla.mozilla.org/attachment.cgi?id=746196 >p1.patch
nexus-s/gaia $ patch -p1 <p1.patch
nexus-s/gaia $ git commit -a -m "Nexus S video app thumbnail fix"

There is a bug related to video recording on the Nexus S. To fix this we need a patch from bug 832638:

Set video recording profile in camera app

To apply this:

nexus-s/gaia $ curl -k -L https://bugzilla.mozilla.org/attachment.cgi?id=704241 >p2.patch
nexus-s/gaia $ patch -p1 <p2.patch
nexus-s/gaia $ git commit -a -m "Nexus S video recording fix"

Building

Now that all the required changes have been made we can build:

nexus-s $ ./build.sh

Installing

Once the build is complete you should be able to flash the Nexus S with the results. Note that this completely wipes everything on your phone:

nexus-s $ ./flash.sh

The phone should reboot after this step with Firefox OS installed and prompting for information from the "First Run" app.

New Zealand changes

Some New Zealand sites don't detect Firefox OS and serve their mobile versions. This can be changed by adding overrides to gaia/build/ua-override-prefs.js. I add the entries in b2gnzuaoverride.txt to this file to get stuff.co.nz, mega.co.nz and trademe.co.nz serving the correct mobile site. Once added to this file, reinstall Gaia on the phone.

nexus-s $ ./flash.sh gaia

Updating

To update the source to latest versions we need to 'rebase' our changes on top of the most recent repository changes. The following steps will do this:

nexus-s $ git pull
nexus-s $ ./config.sh nexus-s
nexus-s $ cd gecko
nexus-s/gecko $ git rebase m/v1-train
nexus-s/gecko $ cd ../gaia
nexus-s/gaia  $ git rebase mozillaorg/master
nexus-s/gaia  $ cd ..
nexus-s $ ./build.sh

You may need to do some fixing up of merge conflicts if changes have occurred to the areas we've patched.

To flash the phone, overwriting everything on it, including all user data:

nexus-s $ ./flash.sh

To flash the phone, keeping (most of...) your existing user data:

nexus-s $ ./flash.sh system
nexus-s $ ./flash.sh gecko
nexus-s $ ./flash.sh gaia

Note that flashing gaia might remove some of your installed application icons. You'll need to reinstall those apps to fix that.