Bluish Coder: freenet

Distributed Wikipedia Mirrors in Freenet

2017-05-16T17:00:00+12:00

There was a recent post about uncensorable Wikipedia mirrors on IPFS. The IPFS project put a snapshot of the Turkish version of Wikipedia on IPFS. This is a great idea and something I've wanted to try on Freenet.

Freenet is an anonymous, secure, distributed datastore that I've written a few posts about. It wasn't too difficult to convert the IPFS process to something that worked on Freenet. For the Freenet keys linked in this post I'm using a proxy that retrieves data directly from Freenet. This uses the SCGIPublisher plugin on a local Freenet node. The list of whitelisted keys usable are at freenet.cd.pn. There is also a gateway available at d6.gnutella2.info. The keys can also be used directly from a Freenet node, which is likely to be more performant than going through my underpowered proxy. Keep in mind that the "distributed, can't be taken down" aspect of the sites on Freenet is only when accessed directly through Freenet. It's quite likely my clearnet proxy won't be able to handle large amounts of traffic.

I started with the Pitkern/Norfuk Wikipedia Snapshot as that was relatively small. Once I got the scripts for that working I converted the Māori Wikipedia Snapshot. The lastest test I did was the Simple English Wikipedia Snapshot. This was much bigger so I did the version without images first. Later I plan to try the version with images when I've resolved some issues with the current process.

The Freenet keys for these mirrors are:

USK@m79AuzYDr-PLZ9kVaRhrgza45joVCrQmU9Er7ikdeRI,1mtRcpsTNBiIHOtPRLiJKDb1Al4sJn4ulKcZC5qHrFQ,AQACAAE/simple-wikipedia/0/
USK@jYBa5KmwybC9mQ2QJEuuQhCx9VMr9bb3ul7w1TnyVwE,OMqNMLprCO6ostkdK6oIuL1CxaI3PFNpnHxDZClGCGU,AQACAAE/maori-wikipedia/5/
USK@HdWqD7afIfjYuqqE74kJDwhYa2eetoPL7cX4TRHtZwc,CeRayXsCZR6qYq5tDmG6r24LrEgaZT9L2iirqa9tIgc,AQACAAE/pitkern-wikipedia/2/

The keys are 'USK' keys. These keys can be updated and have an edition number at the end of them. This number will increase as newer versions of the mirrors are pushed out. The Freenet node will often find the latest edition it knows about, or the latest edition can be searched for using '-1' as the edition number.

The approach I took for the mirroring follows the approach IPFS took. I used the ZIM archives provided by Kiwix and a ZIM extractor written in Rust. The archive was extracted with:

$ extract_zim wikipedia_en_simple_all_nopic.zim

This places the content in an out directory. All HTML files are stored in a single directory, out/A. In the 'simple english' case that's over 170,000 files. This is too many files in a directory for Freenet to insert. I wrote a script in bash to split the directory so that files are stored in '000/filename.html' where '000' is the first three digits of a SHA256 hash of the base filename, computed with:

$ echo "filename.html"|sha256sum|awk '{ print $1 }'|cut -c "1,2,3"

The script then went through and adjusted the article and image links on each page to point to the new location. The script does some other things to remove HTML tags that the Freenet HTML filter doesn't like and to add a footer about the origin of the mirror.

Another issue I faced was that filenames with non-ascii characters would get handled differently by Freenet if the file was inserted as a single file vs being inserted as part of a directory. In the later case the file could not be retrieved later. I worked around this by translating filenames into ascii. A more robust solution would be needed here if I can't track down where the issue is occurring.

This script to do the conversion is in my freenet-wikipedia githib repository. To convert a ZIM archive the steps are:

$ wget http://download.kiwix.org/zim/wikipedia_pih_all.zim
$ extract_zim wikipedia_pih_all.zim
$ ./convert.sh
$ ./putdir.sh result my-mirror index.html

At completion of the insert this will output a list of keys. the uri key is the one that can be shared for others to retrieve the insert. The uskinsert key can be used to insert an updated version of the site:

$ ./putdir.sh result my-mirror index.html <uskinsert key>

The convert.sh script was a quick 'proof of concept' hack and could be improved in many ways. It is also very slow. It took about 24 hours to do the simple english conversion. I welcome patches and better ways of doing things.

The repository includes a bash script, putdir.sh, which will insert the site using the Freenet ClientPutDiskDir API message. This is a useful way to get a directory online quickly but is not an optimal way of inserting something the size of the mirror. The initial request for the site downloads a manifest containing a list of all the files in the site. This can be quite large. It's 12MB for the Simple English mirror with no images. For the Māori mirror it's almost 50MB due to the images. The layout of the files doesn't take into account likely retrieval patterns. So images and scripts that are included in a page are not downloaded as part of the initial page request, but may result in pulling in larger amounts of data depending on how that file was inserted. A good optimisation project would be to analyse the directory to be inserted and create an optimal Freenet insert for faster retrieval. pyFreenet has a utility, freesitemgr, that can do some of this and there are other insertion tools like jSite that may also do a better job.

My goal was to do a proof of concept to see if a Wikipedia mirror on Freenet was viable. This seems to be the case and the Simple English mirror is very usable. Discussion on the FMS forum when I announced the site has been positive. I hope to improve the process over time and welcome any suggestions or enhancements to do that.

What are the differences between this and the IPFS mirror? It's mostly down to how IPFS and Freenet work.

In Freenet content is distributed across all nodes in the network. The node that has inserted the data can turn their node off and the content remains in the network. No single node has all the content. There is redundancy built in so if nodes go offline the content can still be fully retrieved. Node space is limited so as data is inserted into Freenet, data that is not requested often is lost to make room. This means that content that is not popular disappears over time. I suspect this means that some of the wikipedia pages will become inaccessible. This can be fixed by periodically reinserting the content, healing the specific missing content, or using the KeepAlive plugin to keep content around. Freenet is encrypted and anonymous. You can browse Wikipedia pages without an attacker knowing that you are doing so. Your node doesn't share the Wikipedia data, except possibly small encrypted chunks of parts of it in your datastore, and it's difficult for the attacker to identify you as a sharer of that data. The tradeoff of this security is retrievals are slower.

In IPFS a node inserting the content cannot be turned off until that content is pinned by another node on the network and fully retrieved. Nodes that pin the content keep the entire content on their node. If all pinned nodes go offline then the content is lost. All nodes sharing the content advertise that fact. It's easy to obtain the IP address of all nodes that are sharing Wikipedia files. On the positive side IPFS is potentially quite a bit faster to retrieve data.

Both IPFS and Freenet have interesting use cases and tradeoffs. The intent of this experiment is not to present one or the other as a better choice, but to highlight what Freenet can do and make the content available within the Freenet network.

Introduction to the Freenet API

2017-03-28T17:00:00+13:00

Freenet is an anonymous, secure, distributed datastore. I've written about using Freenet before, including using it as the backend for static websites. In this post I'll demonstrate how to use the Freenet API to push data into the Freenet network and retrieve data from it.

Unfortunately the freenet protocol documentation is in a state of flux as it moves from a previous wiki to a github based wiki. This means some of the protocol information may be incomplete. The old wiki data is stored in Freenet under the following keys:

CHK@San0hXZSyCEvvb7enNUIWrkiv8MDChn8peLJllnWt4s,MCNn4eUbl5NW9quOm4JTU~0rsWu6QlIdek9VtpFpXe4,AAMC--8/freenetproject-oldwiki.tar
CHK@4sff2MgvexbsfgSuqNOwqVDkP~GaZPsZ1rJVKUg87g8,unU3TZ93pYGYPCoH7LC53dlc5~Rmar8SKF9fsZnQX-8,AAMC--8/freenetproject-wiki.tar

The API for external programs to communicate with a running Freenet daemon is FCPv2. It's a text based protocol accessed using a TCP socket connection on port 9481. The FCP protocol can be enabled or disabled from the Freenet configuration settings but it is enabled by default so the examples here should work in a default installation.

The basic protocol of FCP uses a unit called a 'message'. Messages are sent over the socket starting with a line for the start of the message, followed by a series of 'Key=Value' lines, and ending the message with 'EndMessage'. Some messages containing binary data and these end differently and I'll discuss this after some basic examples.

For the examples that follow I'll be using bash. I debated picking from my usual toolkit of obscure languages but decided to use something that doesn't require installation on Linux and Mac OS X and may also run on Windows 10. The examples should be readable enough for non-bash users to pick up and translate to their favourite language, especially given the simplicity of the protocol. I've found the ability to throw together a quick bash script to do inserts and retrievals to be useful.

Hello

The FCPv2 documentation lists the messages that can be sent from the client to the Freenet node, and what can be expected to be received from the node to the client. On first connecting to the node the client must send a ClientHello message. This looks like:

ClientHello
Name=My Client Name
ExpectedVersion=2.0
EndMessage

The Name field uniquely identifies the client to the node. Disconnecting and reconnecting with the same Name retains access to a persistent queue of data being inserted and retrieved. An error is returned if an attempt to connect is made when a client with the same Name is already connected.

The node returns with a NodeHello Message. This looks like:

NodeHello
Build=1477
ConnectionIdentifier=...
...
EndMessage

The various fields are described in the NodeHello documentation. This interaction can be tested using netcat or telnet:

$ nc localhost 9481
ClientHello
Name=My Client Name
ExpectedVersion=2.0
EndMessage

NodeHello
CompressionCodecs=4 - GZIP(0), BZIP2(1), LZMA(2), LZMA_NEW(3)
Revision=build01477
Testnet=false
...
ExtRevision=v29
EndMessage

You can connect to a socket from bash using 'exec' and file redirection to a pseudo-path describing the tcp socket. See HACKTUX notes from the trenches for details. The above netcat interaction looks like this from bash:

#! /bin/bash
    function wait_for {
      local line
      local str=$1
      while read -r line
      do
        >&2 echo "$line"
        if [ "$line" == "$str" ]; then
          break
        fi
      done
    }
       
    exec 3<>/dev/tcp/127.0.0.1/9481
       
    cat >&3 <<HERE
    ClientHello
    Name=My Client Name
    ExpectedVersion=2.0
    EndMessage
    HERE
       
    wait_for "NodeHello" <&3
    wait_for "EndMessage" <&3
       
    exec 3<&-
    exec 3>&-

The exec line opens a socket on port 9481, the FCP port, and assigns it to file descriptor '3'. Then we use cat to write the ClientHello message to that file descriptor. wait_for reads lines from the socket, displaying them on standard error (file descriptor '2'), until it reaches a specifc line passed as an argument. Here we wait for the NodeHello line and then the EndMesage line to cover the NodeHello response from the server. The remaining two exec lines close the socket.

The full bash script is available in hello.sh.

Retrieving data inline

The FCP message ClientGet is used to retrieve data stored at a specific key. The data can be returned inline within a message or written to a file accessable by the node. An example message for retrieving a known key is:

ClientGet
URI=CHK@otFYYKhLKFzkAKhEHWPzVAbzK9F3BRxLwuoLwkzefqA,AKn6KQE7c~8G5dLa4TuyfG16XIUwycWuFurNJYjbXu0,AAMC--8/example.txt
Identifier=1234
Verbosity=0
ReturnType=direct
EndMessage

This retrieves the contents of a particular CHK key where I stored example.txt. The Verbosity is set to not return any progress messages, just send messages when the entire contents are retrieved. A ReturnType of direct means return the data within the AllData message which is received when the retrieval is complete. The result messages are:

DataFound
Identifier=1234
CompletionTime=1490614072644
StartupTime=1490614072634
DataLength=21
Global=false
Metadata.ContentType=text/plain
EndMessage

AllData
Identifier=1234
CompletionTime=1490614072644
StartupTime=1490614072634
DataLength=21
Global=false
Metadata.ContentType=text/plain
Data
Hello Freenet World!

The first message received is DataFound giving information about the completed request. The following message, AllData, returns the actual data. Note that it does not include an EndMessage. Instead it has a Data terminator followed by the data as a sequence of bytes of length DataLength.

To process AllData from bash I use a function to extract the DataLength when it finds it:

    function get_data_length {
      local line
      while read -r line
      do
       if [[ "$line" =~ ^DataLength=.* ]]; then
         echo "${line##DataLength=}"
         break
       fi
      done
    }

This is called from the script after the ClientHello and NodeHello exchange:

    cat >&3 <<HERE
    ClientGet
    URI=CHK@otFYYKhLKFzkAKhEHWPzVAbzK9F3BRxLwuoLwkzefqA,AKn6KQE7c~8G5dLa4TuyfG16XIUwycWuFurNJYjbXu0,AAMC--8/example.txt
    Identifier=1234
    Verbosity=0
    ReturnType=direct
    EndMessage
    HERE
        
    wait_for "AllData" <&3
    len=$(get_data_length <&3)
    wait_for "Data" <&3
    dd status=none bs="$len" count=1 <&3 >&2

The dd command reads the specified number of bytes from the socket and outputs it to standard output. This is the contents of the key we requested:

$ ./getinline
Hello Freenet World!

The full bash script is available in getinline.sh.

The main downside of using inline data requests is that large files can exhaust the memory of the node.

Request Direct Disk Access

A variant of ClientGet requests the node to write the result to a file on disk instead of sending it as part of the AllData message. This is useful for large files that don't fit in memory. The data is written to the filesystem that the node has access to so it's most useful when the FCP client and the freenet node are on the same system.

Being able to tell the server to write directly to the filesystem is a security issue so Freenet requires a negotiation to happen first to confirm that the client has access to the directory that you are requesting the server to write to. This negotiation requirement, known as TestDDA can be disabled in the configuration settings of the node but it's not recommended.

First the client must send a TestDDARequest message listing the directory it wants access to and whether read or write access is being requested.

TestDDARequest
Directory=/tmp/
WantWriteDirectory=true
WantReadDirectory=true
EndMessage

The server replies with a TestDDAReply:

TestDDAReply
Directory=/tmp/
ReadFilename=/tmp/testr.tmp
WriteFilename=/tmp/testw.tmp
ContentToWrite=RANDOM
EndMessage

The script should now write the data contained in the ContentToWrite key into the file referenced by the WriteFilename key. It should read the data from the file referenced in the ReadFilename key and send that data in a TestDDAResponse:

TestDDAResponse
Directory=/tmp/
ReadContent=...content from TestDDAReply...
EndMessage

The server then responds with a TestDDAComplete:

TestDDAComplete
Directory=/tmp/
ReadDirectoryAllowed=true
WriteDirectoryAllowed=true
EndMessage

Once that dance is complete then put and get requests can be done to that specific directory. The bash code for doing this is:

    cat >&3 <<HERE
    TestDDARequest
    Directory=/tmp/
    WantWriteDirectory=true
    WantReadDirectory=true
    EndMessage
    HERE
    
    wait_for "TestDDAReply" <&3
    content=$(process_dda_reply <&3)
    
    cat >&3 <<HERE
    TestDDAResponse
    Directory=/tmp/
    ReadContent=$content
    EndMessage
    HERE
    
    wait_for "TestDDAComplete" <&3
    process_dda_complete <&3

It uses a helper function process_dda_reply to handle the TestDDAReply message from the server:

    function process_dda_reply {
      local readfile=""
      local writefile=""
      local content=""
    
      while read -r line
      do
       if [[ "$line" =~ ^ReadFilename=.* ]]; then
         readfile="${line##ReadFilename=}"
       fi
       if [[ "$line" =~ ^WriteFilename=.* ]]; then
         writefile="${line##WriteFilename=}"
       fi
       if [[ "$line" =~ ^ContentToWrite=.* ]]; then
         content="${line##ContentToWrite=}"
       fi
       if [[ "$line" == "EndMessage" ]]; then
         echo -n "$content" >"$writefile"
         cat "$readfile"
         break
       fi
      done
    }

This function reads the fields of the TestDDAReply and writes the required content to the write file and returns the data contained in the read file. This returned data is sent in the TestDDAResponse. The process_dda_complete function checks the TestDDAComplete fields to ensure that access was granted. The full bash script is available in testdda.sh.

Retrieving data to disk

The ReturnType field of the ClientGet message can be set to disk to write directly to a disk file once the TestDDA process is complete. The message looks like this:

    cat >&3 <<HERE
    ClientGet
    URI=CHK@HH-OJMEBuwYC048-Ljph0fh11oOprLFbtB7QDi~4MWw,B~~NJn~XrJIYEOMPLw69Lc5Bv6BcGWoqJbEXrfX~VCo,AAMC--8/pitcairn_justice.jpg
    Identifier=1234
    Verbosity=1
    ReturnType=disk
    Filename=/tmp/pitcairn_justice.png
    EndMessage
    HERE

In this case we're retreving a file I've inserted previously. The Verbosity key is set to 1 to enable SimpleProgress messages to be received. These messages contain fields showing the total number of blocks that can be fetched for that file, the required number of blocks that we need to get, how many we've successfully retrieved so far, and a few other fields. The following bash function processes this and prints the result:

    function handle_progress {
      local total=0
      local succeeded=0
      local required=0
      local final=""
    
      while read -r line
      do
       if [[ "$line" =~ ^Total=.* ]]; then
         total="${line##Total=}"
       fi
       if [[ "$line" =~ ^Required=.* ]]; then
         required="${line##Required=}"
       fi
       if [[ "$line" == "FinalizedTotal=true" ]]; then
         final="final"
       fi
       if [[ "$line" =~ ^Succeeded=.* ]]; then
         succeeded="${line##Succeeded=}"
       fi
       if [[ "$line" == "EndMessage" ]]; then
         echo "Progress: retrieved $succeeded out of $required required and $total total ($final)"
         break
       fi
      done
    }

The FinalizedTotal field indicates if the Total field is accurate and will not change. Otherwise that field can increase as more knowledge about the file is gained. The Required field is the number of blocks that need to be received to reconstruct the file. It is less than Total due to redundancy in the way freenet stores data to account for nodes going away and data being lost.

The handle_progress function is called from within wait_with_progress, which waits for a particular message (usually the one indicating the end of the transfer), displays progress, and ignores everything else.

    function wait_with_progress {
      while read -r line
      do
        if [ "$line" == "SimpleProgress" ]; then
          handle_progress
        fi
        if [ "$line" == "$1" ]; then
          break
        fi
      done
    }

These are called as follows:

    cat >&3 <<HERE
    ClientGet
    URI=CHK@HH-OJMEBuwYC048-Ljph0fh11oOprLFbtB7QDi~4MWw,B~~NJn~XrJIYEOMPLw69Lc5Bv6BcGWoqJbEXrfX~VCo,AAMC--8/pitcairn_justice.jpg
    Identifier=1234
    Verbosity=1
    ReturnType=disk
    Filename=/tmp/pitcairn_justice.png
    EndMessage
    HERE
    
    wait_with_progress "DataFound" <&3
    wait_for "EndMessage" <&3

The DataFound message is sent by the server when the file has been successfully retrieved. It can be found at the location specified in the Filename field of the ClientGet.

The full bash script is available in getdisk.sh.

$ bash getdisk.sh
Progress: retrieved 0 out of 1 required and 1 total ()
Progress: retrieved 1 out of 1 required and 1 total ()
Progress: retrieved 1 out of 5 required and 10 total ()
Progress: retrieved 1 out of 5 required and 10 total (final)
Progress: retrieved 5 out of 5 required and 10 total (final)

Inserting Data Inline

When storing data using FCP you can provide the data directly in the message or reference a file on disk that the node will read and store. They are both done using the ClientPut message. Sending this message looks like:

    file="$1"
    size=$(stat -c%s "$file")
    mime=$(file --mime-type "$file" |awk '{print $2}')
    
    cat >&3 <<HERE
    ClientPut
    URI=CHK@
    Metadata.ContentType=$mime
    Identifier=1234
    Verbosity=1
    GetCHKOnly=false
    TargetFilename=$(basename "$file")
    DataLength=$size
    UploadFrom=direct
    Data
    HERE
    
    dd status=none if="$file" bs="$size" count=1 |pv -L 500k >&3
    
    wait_with_progress "PutSuccessful" <&3
    uri=$(get_uri <&3)
    wait_for "EndMessage" <&3

ClientPut requires the mime type of the file and this is obtained using file. The size of the file is retrieved with stat. These are placed in the ClientPut message directly. The binary data for the file needs to be sent after a Data terminator similar to how we retrieved the data when doing an inline get. dd is again used for this but it's piped to pv to limit the data transfer rate otherwise the network gets swamped due to buffer bloat.

The URI for inserting is generated as a CHK key. This is a key based on a hash of the file content. Inserting the same file will result in the same key. get_uri reads the PutSuccessful message to find the full key of the insert. This is displayed to the user later in the script.

The full bash script is available in putinline.sh.

$ bash putinline.sh /tmp/example.txt 
Progress: inserted 0 out of 1 ()
Progress: inserted 0 out of 2 ()
Progress: inserted 0 out of 2 (final)

Inserting Data from Disk

Inserting data directly from a disk file works very similar to requesting from a disk file. It requires the TestDDA process followed by a ClientPut using a UploadFrom set to disk:

    cat >&3 <<HERE
    ClientPut
    URI=CHK@
    Metadata.ContentType=$mime
    Identifier=1234
    Verbosity=1
    GetCHKOnly=false
    TargetFilename=$(basename "$file")
    Filename=$file
    UploadFrom=disk
    EndMessage
    HERE
    
    wait_with_progress "PutSuccessful" <&3
    uri=$(get_uri <&3)
    wait_for "EndMessage" <&3

The full bash script is available in putdisk.sh.

Other Messages

There are other interesting messages that are useful. Using ClientPut if you set the field GetCHKOnly to true then the file isn't inserted but the CHK key is generated. Since CHK is based on the file contents it will be the same key if the file is inserted using the same mime type, filename and other parameters. This allows generating a key, sending it to a third party and they can start a file retrieval before the file completes inserting. There are security issues with this in that if an attacker knows the key while it is being inserted they may be able to narrow down the location of the inserting node.

Another useful message is GenerateSSK:

GenerateSSK
Identifier=1234
EndMessage

This results in a SSKKeypair reply containing a randomly generated SSK insert and request key:

SSKKeypair
InsertURI=SSK@AK.../
RequestURI=SSK@Bn.../
Identifier=1234
EndMessage

These can be used to insert data with ClientPut by setting the URI to the InsertURI, and retrieved by a third party using the RequestURI as the URI in ClientGet.

ClientPutDiskDir inserts an entire directory. This is the basis of inserting 'Freesites' - Freenet websites. I wrote a mirror.sh utility that mirrors an online website or page and inserts it into Freenet. This is useful for linking to news articles in Freenet without having to leave the Freenet environment. It uses a putdir.sh script that inserts a directory.

Conclusion

The Freenet API has a lot of functionality beyond what I've shown here. I used bash for the examples because it has few dependancies but more robust scripts would be easier in a full featured programming language. I'm not a bash expert and welcome corrections and additions. I've put the code in an fcp-examples github repository.

There are some client libraries with pyFreenet being an example. I recommend the following articles for a deeper dive into Freenet programming:

Using Freenet over Tor

2016-08-18T18:00:00+12:00

This post outlines a method of using Freenet over Tor based on posts I wrote on my Freenet hosted blog and subsequent discussions about it. If you read my Freenet hosted blog there's little new here, I'm just making it available on my non-freenet blog.

One issue I've had with Freenet is that it exposes your IP address to peers. Recent law enforcement efforts to monitor Freenet have shown that they have been able to obtain search warrants based on logging requests for blocks of known data and associating them with IP addresses. If law enforcement can do this, so can random bad people.

You can avoid exposing your IP address to random strangers on opennet by using darknet but even then you have to trust your friends aren't monitoring your requests. If it was possible to run Freenet over Tor hidden services then only the hidden service address would be exposed using this logging method. A problem is that Freenet uses UDP which Tor does not support.

A recent post on the Freenet development mailing list pointed out that onioncat provides a virtual network over Tor and tunnels UDP. Using the steps they provided, and some tweaks, it's possible to set up a darknet node that doesn't expose its IP address. It uses the onioncat generated IPv6 address for communicating with peers - and this address is backed by a Tor hidden service.

The steps below outline how to set this up. Note that this is quite experimental and requires care to not expose your IP address. There are some Freenet issues that make things difficult so you should be aware that you do this at your risk and understand it may still expose your identity if things go wrong.

I'm assuming a Debian/Ubuntu like system for the steps.

Install Tor

Install Tor:

$ sudo apt-get install tor

Edit the /etc/tor/torrc file to enable a Hidden Service with an entry like:

HiddenServiceDir /var/lib/tor/freenet/
HiddenServicePort 8060 127.0.0.1:8060

Restart Tor and find your hidden service hostname:

$ sudo systemctl restart tor
$ sudo cat /var/lib/tor/freenet/hostname

Install onioncat

Install onioncat:

$ sudo apt-get install onioncat

Edit /etc/default/onioncat and change the lines matching the following:

ENABLED=yes
DAEMON_OPTS="-d 0 hiddenservicename.onion -U"

Restart onioncat:

$ sudo systemctl stop onioncat
$ sudo systemctl start onioncat

Find your onioncat IP address with:

$ ocat -i hiddenservicename.onion

Install Freenet

Install Freenet in the usual way and go through the browser based setup wizard. Choose "Details settings: (custom)" for the security option. On the subsequent pages of the wizard:

Disable the UPnP plugin.
Choose "Only connect to your friends"
Choose "High" for "Protection against a stranger attacking you over the internet"
Click the "I trust at least one person already using Freenet" checkbox.
For "Protection of your downloads..." pick any option you want.
Pick a node name that your darknet friends will see.
Pick a datastore size that you want.
Choose the bandwidth limit.

The node will now be started but have no connections. There will be warnings about this.

Configure Freenet over Tor

The following settings need to be changed in "Configuration/Core Settings" - make sure you have clicked "Switch to advanced mode".

Change "IP address override" to your onioncat IP address retrieved in the previous section.
Apply the changes.

Shut down Freenet and edit the wrapper.conf file in the Freenet installation directory. Change the line that contains java.net.preferIPv4Stack=true to java.net.preferIPv4Stack=false. In my wrapper.conf this is:

wrapper.java.additional.3=-Djava.net.preferIPv4Stack=false

Edit freenet.ini file in the Freenet installation directory. Change or add the following (replace "onioncat IP address" with the IP address obtained installing onioncat):

node.opennet.bindTo=onioncat IP address
node.bindTo=onioncat IP address
node.load.subMaxPingTime=2500
node.load.maxPingTime=5k

Save the file and restart Freenet. There might be a warning about "Unknown external address". Ignore this as you've explictly set one. I provide a patch later in this post if you want to get rid of the warning.

Add a friend

Now is the time to add a Darknet friend who is also using Tor/Onioncat. Go to "Friends/Add a friend". Choose your trust and ability to see other friends settings and enter a description of the friend. Paste their noderef in the "Enter node reference directly" box.

Give your noderef to your friend and have them add it. Once both connections have been added you should see "Connected" in the Friends list for that connection. The IP address should show the onioncat IPv6 address, beginning with "fd".

Optional Freenet patch

When running a Tor based node Freenet thinks the onioncat IP address is a local address. Some places in the Freenet code base check for this and reject it as a valid global routable address. In the FProxy user interface a large warning appears on each page that it couldn't find the external IP address of the node. The other issue is that local addresses aren't counted for bandwidth statistic reporting. The bandwidth box on the statistics page is empty as a result.

I use a patch, onioncat.txt, that provides a workaround for these two issues. The patch is optional as the node works without it but it's a useful improvement if you plan to run a Tor based node long term. You should check the patch before applying it blindly and assure that it's not doing anything nefarious.

Hybrid nodes

If you run a Tor based darknet node then at least one hybrid node must be in the darknet to bridge to the non-tor nodes. These hybrid nodes will have a public clearnet IP address exposed. I outline how to set up a hybrid node later below. For those that trust me, if you send a darknet tor noderef to me at the freemail address on the bottom of this page, or via normal email, I'll connect and send you a noderef of a hybrid node setup in this manner.

Install Tor and Onioncat as described previously. Install Freenet in the usual way and go through the browser based setup wizard. Choose "Details settings: (custom)" for the security option. On the subsequent pages of the wizard:

Enable or Disable the UPnP plugin as necessary depending on what you need for your clearnet connection to work.
Choose "Connect to strangers"
Choose "Low" or "Normal" security as desired.
For "Protection of your downloads..." pick any option you want.
Pick a datastore size that you want.
Choose the bandwidth limit.

The node will start and connect to opennet.

wrapper.java.additional.3=-Djava.net.preferIPv4Stack=false

Edit freenet.ini file in the Freenet installation directory. Change or add the following:

node.load.subMaxPingTime=2500
node.load.maxPingTime=5k

Save the file and restart Freenet. If you base64 decode the "physical.udp" section of the noderef for the node you should see that it now contains the onioncat IP address as well as the public clearnet IP address.

Adding friends to this node will give those friends access to the wider Freenet datastore when they reciprocate.

Don't forget to check your noderefs to ensure that the ARK and the public IP address contain data you are willing to reveal. Check both the darknet noderef and the opennet noderef. You can decode the base64 of the "physical.udp" line with the GNU base64 command:

$ echo "physical.udp base64 here" |base64 -d

Final steps and caveats

Try visiting a Freenet index site and see if it loads. If it does then the Freenet over Tor setup is working. It will be slower than normal Freenet usage due to Tor latency. If you connect to more darknet nodes it will get faster.

When adding a friends noderef you can check what IP addresses it will connect to by looking at the "physical.udp" line. This is a base64 encoded list of IP addresses. You might want to check this to ensure that there are no clearnet addresses in there. If there is a clearnet address then it could deanonymize your node when it tries to connect to that in preference to the onioncat address.

The "ark.pubURI" portion of the noderef is an SSK that points to updated IP address information. A node can subscribe to the USK version of this and learn about IP address changes. Your friends node could change their IP address to a clearnet address resulting in you connecting to that.

To avoid the above two issues it's worthwhile running Freenet in a VM or container that does not have clearnet network access and only has access to the onioncat network setup. Alternatively you can use iptables to only allow onioncat traffic for the Freenet process or user running it.

The IP addresses exposed in the noderef include all local link addresses and their scopes. This is Freenet bug 6879. This may leak information you don't want leaked. It pays to check the "physical.udp" and "ark.pubURI" to see what you are exposing. Remember that any IP addresses exposed over the ARK is discoverable by looking at previous editions of the USK.

The traffic footprint of Freenet may make it easier to track down your IP address from your Tor ID. The volume of data and the nature of the traffic may make certain types of Tor de-anonymization techniques more effective.

Ideally it would be possible to have an opennet of Tor nodes so the exchange of darknet noderefs wouldn't be needed. I haven't been able to get this working yet but I'll continue to investigate it.

I've been running a Tor darknet node for the past week to test how well it works. With three darknet connections it runs well enough for browsing freesites. Sone and the Web of Trust took quite a while to bootstrap due to the lower speed but once it was running it works well. FMS and Flip are also usable. I'd expect performance to be even better with more connections.

Using Freenet for Static Websites

2015-09-14T19:00:00+12:00

This website is generated from markdown to static HTML and I mirror it on Freenet. Data on Freenet slowly disappears if it is not regularly requested and this happens to parts of the mirror of my blog since many posts have a small target audience and the cross section of Freenet users and that target audience results in a low number of requests.

I've thought about changing the clearnet site so it is a thin proxy in front of a Freenet node and retrieves the data from Freenet. This enables all the clearnet requests to contribute to the healing of the Freenet data. It also means an update to the site on Freenet will automatically be reflected in the clearnet version.

The recent announcement of Neocities mirroring their sites on IPFS prompted me to try this on Freenet to see how viable it was.

I've been able to get something working and this site is now being served directly from the Freenet data with nginx acting as a caching reverse proxy. Performance is acceptable. Taking this approach has a security tradeoff in that I've had to lock down internal node pages that may allow manipulating the node directly. See the end of this post for details on this.

Freenet has an API called FCP that allows retrieval of content programatically. I thought about writing a simple HTTP proxy server that would retrieve the requests from Freenet via FCP and send them back to the requester. I didn't want to invest too much effort into a proof of concept so I looked to see if there are existing tools to do this.

SCGIPublisher is a plugin for Freenet that provides an SCGI interface to Freenet content using a whitelist to expose only the desired data. It expects to be exposing actual Freenet URIs and keys. I want to hide all this behind my standard domain and I couldn't work out how to prevent the filtering of data and rewriting of URLs that it does. An example of SCGIPublish usage is d6.gnutella2.info - it's a proxy that provides access to a number of Freenet sites from clearnet.

Freenet already has a built in proxy, FProxy. It does filtering of the requested data to remove JavaScript and detect potentially malicious file formats. If I could disable this filtering and use nginx as a reverse proxy I'd be able to get what I wanted without writing any code. It turns out this can be disabled by doing the following:

In the Freenet node Configuration/Web Interface menu, set "Disable progress page when loading pages?" to false.
In the same menu, set "Maximum size of transparent pass-through in the web interface where we cannot show progress" and "Maximum size for transparent pass-through in the web interface where we can show a progress bar" to something higher than the maximum file size of the site you are exposing. Without this the user will receive an HTML page instead of the required content if the content is large.
Append "?forcedownload=true" to all requested URLs.

With this setup an nginx reverse proxy can be created that uses the Freenet node web interface as the upstream. Unfortunately setting ?forcedownload=true results in Freenet not sending the mime type for the content so I had to create a lookup table in nginx to compute the mime type. This table looks like:

map $uri $custom_content_type {
     default         "text/html";
     ~(.*\.xml)$  "text/xml";
     ~(.*\.rss)$  "application/rss+xml";
     ~(.*\.png)$  "image/png";
     ~(.*\.gif)$  "image/gif";
     ~(.*\.jpg)$  "image/jpeg";
     ~(.*\.pdf)$  "application/pdf";
     ..etc...
 }

In the server section of the configuration I set up some high timeout values to cater for the inital slowness of the freenet node. I intercept the 404 and 500 error pages to display some static HTML error messages. This stops the Freenet proxy error pages from having internal links allowing doing things on the node.

server {
    listen       80;
    server_name proxy.example.com;

    proxy_intercept_errors on;
    error_page 404 /404.html;
    error_page 500 /500.html;
    proxy_connect_timeout 300;
    proxy_send_timeout 300;
    proxy_read_timeout 300;
    send_timeout 300;
    try_files $uri $uri/index.html;
    location /404.html {
        root /var/www/html/;
        allow all;
        internal;
    }
    location /500.html {
        root /var/www/html/;
        allow all;
        internal;
    }
    ... location blocks ...
}

Following that comes the location blocks. These are hardcoded for the Freenet keys being exposed to prevent the proxy being used to browse any Freenet site. I've shortened the actual key below with .... to keep the example short.

This block hides headers returned by the Freenet proxy, adds the ?forcedownload=true query parameter and sets the proxy_pass to go to the Freenet node with the hardcoded key.

location /freenet/USK@..../bluishcoder/ {
        proxy_intercept_errors on;

        index index.html;
        proxy_redirect
            ~^/freenet:USK@..../bluishcoder/(?<edition>[0-9]+)/(.*)\?forcedownload=true$
            /freenet/USK@..../bluishcoder/$edition/$2;
        set $args forcedownload=true;
        proxy_hide_header Content-Type;
        proxy_hide_header Content-Transfer-Encoding;
        proxy_hide_header Content-Disposition;
        proxy_hide_header X-Content-Type-Options;
        proxy_hide_header X-Content-Security-Policy;
        proxy_hide_header X-Webkit-Csp;
        proxy_hide_header Content-Security-Policy;
        proxy_hide_header Cache-Control;
        proxy_hide_header Pragma;
        add_header Content-Type $custom_content_type;
        error_page 301 302 307 =200 @redir;

        proxy_pass http://127.0.0.1:8888/USK@..../bluishcoder/;
}

USK keys have an edition number that is incremented every time a site is updated. I bookmark the USK key in the node which results in it subscribing for updates and it will automatically pick up the latest edition, even if a request in the nginx configuration file is coded to use a specific edition number.

I don't include the USK edition number in the proxy_pass request here to make handling edition updates easier. If a request is made for an edition where a later one is available the node will send a 301 redirect. The Location header in the redirect is of the format freenet:USK@.../bluishcoder/edition/.... The edition is a numeric value of the latest edition number. The proxy_redirect statement rewrites this into the URL format that our location block uses.

At this point the error_page statement is hit which converts the 301 location moved response to a 200 (HTTP OK response) and passes it to a redir block:

location @redir {
  set $args forcedownload=true;
  proxy_hide_header Content-Type;
  proxy_hide_header Content-Transfer-Encoding;
  proxy_hide_header Content-Disposition;
  proxy_hide_header X-Content-Type-Options;
  proxy_hide_header X-Content-Security-Policy;
  proxy_hide_header X-Webkit-Csp;
  proxy_hide_header Content-Security-Policy;
  proxy_hide_header Cache-Control;
  proxy_hide_header Pragma;
  add_header Content-Type $custom_content_type;

  set $foo $upstream_http_location;
  proxy_pass http://127.0.0.1:8888$foo;
}

This block saves the original upstream HTTP Location header (The rewritten one in our own URL format) and passes it back to the proxy to get the data. In this way updated USK editions are handled even though requests are made for earlier editions.

With this in place we have a Freenet proxy for whitelisted URLs that work with requests lke http://proxy.example.com/freenet/USK@.../bluishcoder/17/.... Putting another reverse proxy in front of this allows standard clearnet URLs to be used:

server {
    listen       80;
    server_name example.com;

    proxy_intercept_errors on;
    error_page 404 /404.html;
    error_page 500 /500.html;
    proxy_connect_timeout 300;
    proxy_send_timeout 300;
    proxy_read_timeout 300;
    send_timeout 300;
    try_files $uri $uri/index.html;
    location /404.html {
        root /var/www/html/;
        allow all;
        internal;
    }
    location /500.html {
        root /var/www/html/;
        allow all;
        internal;
    }

    location / {
        index index.html;
        proxy_pass http://proxy.example.com/freenet/USK@..../bluishcoder/17/;
    }
}

Now requests to http://example.com proxy to the Freenet USK and avoids the internal filtering and progress pages.

I've got this site running with this setup and it works pretty well. I use nginx proxy caching to cache requests for a period of time to ease the load on the Freenet node. There are some rough edges. The first request for a large file, video for example, takes a while as it has to download the entire video - it can't stream it. Once it is cached by nginx then streaming to other requests is fine.

Taking this approach got things working quickly but I think in the long term it would be better to take the approach of writing a proxy that utilizes FCP as described earlier. This would enable using the mime type that Freenet knows for the files, avoiding the manual table in the nginx configuration, and avoids any possible security issues from accidentally leaking internal Freenet proxy pages. It helps prove the approach as viable however.

Security Tradeoffs

If an internal node page somehow became available to the user they may be able to access any Freenet URL, download files to any location on the machine and upload any files on the machine. They can reconfigure the node and create connections to other nodes.

To restrict the damage they can do I've changed the following settings on the node:

In Configuration/Core Settings change the "Directories downloading is allowed to" to empty. This prevents the node being used to download files to disk.
In Configuration/Core Settings change the "Directories uploading is allowed from" to empty. This prevents the node being used to upload files.
In Configuraiton/Web Interface change "Public Gateway Mode" to true. This will prevent the user from being able to change any node settings. You should configure an IP address for an admin user to access the node with full administation settings. Optionally, and what I do, is enable it for all users. If I want to temporarily administer the node I shut it down from a shell, edit freenet.ini to change fproxy.publicGatewayMode to false, and restart.

These issues would go away if a proxy that uses FCP, or a Freenet plugin that does similar, was created.

Using Freenet

2014-12-18T17:00:00+13:00

I've been following the Freenet project for many years, occasionally firing it up and seeing if I can do anything useful with it. I've been using it regularly over the last month and it has come a long way since I first tried it. It's much faster than what it was in the past. This post describes a bit of how I use it and some of the issues I worked around when publishing content.

Overview

There are no dynamic servers on Freenet. No user hosts a site. It's a data store and users push the data into the content store which then becomes retrievable by anyone with the key. Freenet is essentially a large encrypted distributed hash table.

Nodes set aside an amount of disk space and users choose to store data under a key. Retrieval of the key goes out into the distributed hash table and returns the data associated with it. Inserting data into the store pushes that data out into other nodes and is not generally stored in your own node. Requesting data sends the request over the network and the data migrates to your node to return the data. A scheme is used to enable recovery from data loss. M of N segments of your data can be lost but the full data can still be recovered. The network is lossy in that as more data is inserted, less frequently requested data drops out. Data stored is immutable. Once it is in the store with a key it will always be associated with that data.

Freenet data is requested using keys. There are different types of keys. A high level overview would be:

KSK@somewhere: A KSK key can be chosen by the inserter. The 'somewhere' portion of the key can be any value. This allows generating keys to access data using easier to remember words or phrases. The downside is they can be re-inserted by anyone with different data. What you get when you request the key depends on what data has been inserted by different users under it.
CHK@...: CHK keys have the `...` portion computed based on the hash of the data content. These have the advantage that the key is always the same for the same data. If data for a CHK key has dropped out of the network anyone can 'heal' that data by reinserting the same file. The hash for the CHK key will be the same and the data will become available again under that same key. This is like being able to have any user fix a 404 response on the standard internet by reuploading the same file anywhere.
SSK@...: An SSK key has a unique cryptographically generated hash that is different for any given insert of data. These cannot be 'healed' if the data drops out as a re-insert will have a different key.
USK@.../foo/1: A USK key allows updateable content. Note the number at the end. This increments everytime new data for the key is inserted. When requesting data freenet can look for the highest number available and return that. It's useful for freenet hosted blogs which have reguarly updated content.

Setup

The freenet software really needs to run 24x7 to be effective. I followed the headless install instructions to install on a server machine and access the freenet proxy on my client machines using an SSH tunnel. An SSH commmand like the following sets up local ports that tunnel to the server so they can be accessed locally:

ssh -L 8888:127.0.0.1:8888 -L 8080:127.0.0.1:8080 -L 9481:127.0.0.1:9481 me@myserver.local

The 8888 port is for the freenet proxy software where you access most freenet functionality from the browser. Port 8080 is for the Freenet Message System if you install that and 9481 is for the API interface that jSite uses.

It takes a few hours for a new freenet node to establish itself and get up to speed. Expect much slowness initially. It gets better though.

Social Networking on Freenet

Freenet has some social networking functionality. There is a web of trust for identities, distributed anonymous email, twitter-like microblogging, forums and IRC like chat. How to set these up is described in the Freenet Social Networking Guide. Setting up an identity in the web of trust and Sone for the microblogging will give a good start to using freenet socially.

You can create as many web of trust identities as you want and switch between them for different purposes. I use Freenet non-anonmously and my identity on there is associated with my real world identity but I could also have anonymous ones for other purposes.

Freenet Sites

A freenet site is usually stored under a USK key so it can be updated. Software to insert a directory of HTML as a USK is the easiest way of uploading a site or blog. I use jSite. I mirror this blog to freenet under the key USK@1ORdIvjL2H1bZblJcP8hu2LjjKtVB-rVzp8mLty~5N4,8hL85otZBbq0geDsSKkBK4sKESL2SrNVecFZz9NxGVQ,AQACAAE/bluishcoder/-7. Note the negative number at the end. When requested this results in freenet starting from edition '7' and looking for the most recent published edition from there. Sites can be bookmarked in the freenet proxy and it will automatically look for and update the bookmark when a new edition is inserted.

There were some issues I had to workaround when mirroring my Jekyll based blog. I have absolute links in the blog that references other pages. These don't work if copied directly to a freenet site as the freenet proxy has the content key as the initial part of the URL. So a link to a page in the proxy looks like /USK@longhash/bluishcoder/7/2014/12/17/changing-attributes-in-self-objects.html. An internal link that starts with / to go to a page will not work as it doesn't contain the USK key prefix. I tried modifying Jekyll to use relative URLs but wasn't successful. The approach I ended up taking was to follow the advice in this github issue. My _config.yml file contains these baseurl entries:

baseurl: "file:///some/path/bluishcoder/_site"
#baseurl: /USK@longlonghash/bluishcoder/7
#baseurl: "http://bluishcoder.co.nz"

All my internal links in blog posts have the baseurl prefixed. For example (Remove the backslash - I had to add it to prevent Jekyll from replacing it with the baseurl here):

[link to a video]({\{site.baseurl}}/self/self_comment.webm)

This gets replaced at blog generation time by the baseurl entry in _config.yml. I generate my internet based blog with the relevant baseurl, copy that to my webserver, then generate the freenet based one with the correct baseurl and push that to freenet using jSite. This is a bit tedious but works well. A blog system that uses only relative URLs would be a lot easier as you can just insert the site directly.

Note that freenet sites cannot use JavaScript and some content is filtered out for security reasons. Simple HTML and CSS works best.

Photo heavy sites

I have a site heavy in photos which is a mirror of some photos from my Pitcairn Island trip. This is under key USK@2LK9z-pdZ9kWQfw~GfF-CXKC7yWQxeKvNf9kAXOumU4,1eA8o~L~-mIo9Hk7ZK9B53UKY5Vuki6p4I4lqMQPxyw,AQACAAE/pitcairnisland/3. The interesting problem with photo heavy sites is how best to present the photos while also preventing them from dropping out of the network.

If the main page of the site has thumbnail images and allows the user to see the full image by selecting the thumbnail then the thumbnails tend to stay alive as they are most requested. Unfortunately some of the full images will tend to drop out eventually. A recommended approach by long time freenet users is to link to the full photo in the IMG tag but scale it to thumbnail size. This causes the page to have all the full size images scaled and is slow to load. But all the images stay alive.

I like the fast loading approach of thumbnails though so tried to find a middle ground. Image preloading using CSS seemed like a viable solution but Freenet's content filter has issues with it. With some tweaking of that this approach would work well. The thumbnails would load for quick viewing and the full images would pre-load without the user noticing that the page is still loading. This should result in most images staying around.

The approach I ended up using was to have a hidden DIV at the end of the page with the full sized images. They don't display and cause the full size images to be retrieved while the user sits on the main page. The downside is the page still shows that it's loading which isn't optimal. I also link to a page that has the full sized images scaled to thumbnail size as a viewing option. Hopefully the issue with the CSS preloading approach can be resolved as that has a better user experience.

Conclusion

Other than mirring my blog and using Sone I haven't done too much else. There is a 'bitcoin over freenet' program that mirrors the blockchain in freenet and allows submitting and retrieving transactions that looks interesting to explore. Freenet would seem to be useful for some things Tor is used for (dissemination of information under oppressive regimes) without the requirement of needing an active server that can be located and attacked.

There's a great set of PDF slides that cover more about what Freenet can do if you're interested in looking into it more.

My interest has been more about looking at how freenet can be used as a more encrypted and non-hosted distributed alternative to services like Twitter, Facebook, hosted email and the like. As long as you can put up with higher latency and the different idioms an 'immutable internet that decays' requires it seems that this is viable.

I'm curious what other services people could build on top of it.

Bluish Coder: freenet

Distributed Wikipedia Mirrors in Freenet

Introduction to the Freenet API

Hello

Retrieving data inline

Request Direct Disk Access

Retrieving data to disk

Inserting Data Inline

Inserting Data from Disk

Other Messages

Conclusion

Using Freenet over Tor

Install Tor

Install onioncat

Install Freenet

Configure Freenet over Tor

Add a friend

Optional Freenet patch

Hybrid nodes

Final steps and caveats

Using Freenet for Static Websites

Security Tradeoffs

Related Links

Using Freenet

Overview

Setup

Social Networking on Freenet

Freenet Sites

Photo heavy sites

Conclusion