Bluish Coder

I've been following the Freenet project for many years, occasionally firing it up and seeing if I can do anything useful with it. I've been using it regularly over the last month and it has come a long way since I first tried it. It's much faster than what it was in the past. This post describes a bit of how I use it and some of the issues I worked around when publishing content.

Overview

There are no dynamic servers on Freenet. No user hosts a site. It's a data store and users push the data into the content store which then becomes retrievable by anyone with the key. Freenet is essentially a large encrypted distributed hash table.

Nodes set aside an amount of disk space and users choose to store data under a key. Retrieval of the key goes out into the distributed hash table and returns the data associated with it. Inserting data into the store pushes that data out into other nodes and is not generally stored in your own node. Requesting data sends the request over the network and the data migrates to your node to return the data. A scheme is used to enable recovery from data loss. M of N segments of your data can be lost but the full data can still be recovered. The network is lossy in that as more data is inserted, less frequently requested data drops out. Data stored is immutable. Once it is in the store with a key it will always be associated with that data.

Freenet data is requested using keys. There are different types of keys. A high level overview would be:

KSK@somewhere

A KSK key can be chosen by the inserter. The 'somewhere' portion of the key can be any value. This allows generating keys to access data using easier to remember words or phrases. The downside is they can be re-inserted by anyone with different data. What you get when you request the key depends on what data has been inserted by different users under it.

CHK@...

CHK keys have the `...` portion computed based on the hash of the data content. These have the advantage that the key is always the same for the same data. If data for a CHK key has dropped out of the network anyone can 'heal' that data by reinserting the same file. The hash for the CHK key will be the same and the data will become available again under that same key. This is like being able to have any user fix a 404 response on the standard internet by reuploading the same file anywhere.

SSK@...

An SSK key has a unique cryptographically generated hash that is different for any given insert of data. These cannot be 'healed' if the data drops out as a re-insert will have a different key.

USK@.../foo/1

A USK key allows updateable content. Note the number at the end. This increments everytime new data for the key is inserted. When requesting data freenet can look for the highest number available and return that. It's useful for freenet hosted blogs which have reguarly updated content.

Setup

The freenet software really needs to run 24x7 to be effective. I followed the headless install instructions to install on a server machine and access the freenet proxy on my client machines using an SSH tunnel. An SSH commmand like the following sets up local ports that tunnel to the server so they can be accessed locally:

ssh -L 8888:127.0.0.1:8888 -L 8080:127.0.0.1:8080 -L 9481:127.0.0.1:9481 me@myserver.local

The 8888 port is for the freenet proxy software where you access most freenet functionality from the browser. Port 8080 is for the Freenet Message System if you install that and 9481 is for the API interface that jSite uses.

It takes a few hours for a new freenet node to establish itself and get up to speed. Expect much slowness initially. It gets better though.

Social Networking on Freenet

Freenet has some social networking functionality. There is a web of trust for identities, distributed anonymous email, twitter-like microblogging, forums and IRC like chat. How to set these up is described in the Freenet Social Networking Guide. Setting up an identity in the web of trust and Sone for the microblogging will give a good start to using freenet socially.

You can create as many web of trust identities as you want and switch between them for different purposes. I use Freenet non-anonmously and my identity on there is associated with my real world identity but I could also have anonymous ones for other purposes.

Freenet Sites

A freenet site is usually stored under a USK key so it can be updated. Software to insert a directory of HTML as a USK is the easiest way of uploading a site or blog. I use jSite. I mirror this blog to freenet under the key USK@1ORdIvjL2H1bZblJcP8hu2LjjKtVB-rVzp8mLty~5N4,8hL85otZBbq0geDsSKkBK4sKESL2SrNVecFZz9NxGVQ,AQACAAE/bluishcoder/-7. Note the negative number at the end. When requested this results in freenet starting from edition '7' and looking for the most recent published edition from there. Sites can be bookmarked in the freenet proxy and it will automatically look for and update the bookmark when a new edition is inserted.

There were some issues I had to workaround when mirroring my Jekyll based blog. I have absolute links in the blog that references other pages. These don't work if copied directly to a freenet site as the freenet proxy has the content key as the initial part of the URL. So a link to a page in the proxy looks like /USK@longhash/bluishcoder/7/2014/12/17/changing-attributes-in-self-objects.html. An internal link that starts with / to go to a page will not work as it doesn't contain the USK key prefix. I tried modifying Jekyll to use relative URLs but wasn't successful. The approach I ended up taking was to follow the advice in this github issue. My _config.yml file contains these baseurl entries:

baseurl: "file:///some/path/bluishcoder/_site"
#baseurl: /USK@longlonghash/bluishcoder/7
#baseurl: "http://bluishcoder.co.nz"

All my internal links in blog posts have the baseurl prefixed. For example (Remove the backslash - I had to add it to prevent Jekyll from replacing it with the baseurl here):

[link to a video]({\{site.baseurl}}/self/self_comment.webm)

This gets replaced at blog generation time by the baseurl entry in _config.yml. I generate my internet based blog with the relevant baseurl, copy that to my webserver, then generate the freenet based one with the correct baseurl and push that to freenet using jSite. This is a bit tedious but works well. A blog system that uses only relative URLs would be a lot easier as you can just insert the site directly.

Note that freenet sites cannot use JavaScript and some content is filtered out for security reasons. Simple HTML and CSS works best.

Photo heavy sites

I have a site heavy in photos which is a mirror of some photos from my Pitcairn Island trip. This is under key USK@2LK9z-pdZ9kWQfw~GfF-CXKC7yWQxeKvNf9kAXOumU4,1eA8o~L~-mIo9Hk7ZK9B53UKY5Vuki6p4I4lqMQPxyw,AQACAAE/pitcairnisland/3. The interesting problem with photo heavy sites is how best to present the photos while also preventing them from dropping out of the network.

If the main page of the site has thumbnail images and allows the user to see the full image by selecting the thumbnail then the thumbnails tend to stay alive as they are most requested. Unfortunately some of the full images will tend to drop out eventually. A recommended approach by long time freenet users is to link to the full photo in the IMG tag but scale it to thumbnail size. This causes the page to have all the full size images scaled and is slow to load. But all the images stay alive.

I like the fast loading approach of thumbnails though so tried to find a middle ground. Image preloading using CSS seemed like a viable solution but Freenet's content filter has issues with it. With some tweaking of that this approach would work well. The thumbnails would load for quick viewing and the full images would pre-load without the user noticing that the page is still loading. This should result in most images staying around.

The approach I ended up using was to have a hidden DIV at the end of the page with the full sized images. They don't display and cause the full size images to be retrieved while the user sits on the main page. The downside is the page still shows that it's loading which isn't optimal. I also link to a page that has the full sized images scaled to thumbnail size as a viewing option. Hopefully the issue with the CSS preloading approach can be resolved as that has a better user experience.

Conclusion

Other than mirring my blog and using Sone I haven't done too much else. There is a 'bitcoin over freenet' program that mirrors the blockchain in freenet and allows submitting and retrieving transactions that looks interesting to explore. Freenet would seem to be useful for some things Tor is used for (dissemination of information under oppressive regimes) without the requirement of needing an active server that can be located and attacked.

There's a great set of PDF slides that cover more about what Freenet can do if you're interested in looking into it more.

My interest has been more about looking at how freenet can be used as a more encrypted and non-hosted distributed alternative to services like Twitter, Facebook, hosted email and the like. As long as you can put up with higher latency and the different idioms an 'immutable internet that decays' requires it seems that this is viable.

I'm curious what other services people could build on top of it.