Reinventing Web Storage

by Zef Hemel

I had this great idea a month or two ago: We are storing more and more data on servers on the web, be it email, pictures, music, calendars or documents. And most of it is scattered all around on different services with different interfaces. Can’t we come up with a unified interface to storage and make our data mobile again?

That was the secret project I have been working on, which I dubbed WebFS. It was a very simple REST webservice, which basically allowed you to GET and POST data items (documents). Each item had metadata associated with it (in RDF). There is a special kind of item, called the Container, which has a directory-like function. Every service supporting WebFS should allow the user to store arbitrary RDF metadata for its items, which would allow all kinds of neat services (such as synchronization) to be built.

I got quite a bit of it to work. I wrote a little bash-like shell (webshell) which allowed you to browse through a WebFS share and allowed you to copy items, view them, and get and set metadata. I built a WebFS interface for a simple local filesystem, but also for Jungle Disk. So I could access my files on Amazon’s S3 (where JungleDisk stores the files) through WebFS. The idea was that I would also build a FlickrFS which exposed flickr albums as a WebFS system and so you can easily copy files from there to your local driver, or other WebFS-enabled storage systems. In the future other systems could be built, like IMAPFS, CalendarFS, and so on and so forth. WebFS would be the great unifier.

Nice idea huh?

Yeah, so great that it already exists. WebDAV does almost exactly this. It wasn’t really built to create wrappers around existing systems, but it does allow you to access files on remote webservers pretty much in the way WebFS allows you to do that. In addition it allows you to store arbitrary XML data as meta data for your files. And the best thing, it’s already integrated in all major operating systems. OS X allows you to mount WebDAV shares (actually that’s how Jungle Disk works) and so does Windows and Linux.

Crap.

There are differences between WebDAV and WebFS, and choices in WebFS which can be considered nicer, but they surely do not constitute a new protocol while WebDAV is already well established. So basically I spent dozens of hours implementing something really elegant that already existed.

Great huh.

But what it did learn me is that we should have a better look at WebDAV for our services. Many services have, beside the standard web interface, some kind of API (usually a REST one). Wouldn’t it be an idea to also add a WebDAV interface to those? This way data is truly open, truly free to the user and we wouldn’t have a data lock-in, which I think nobody likes and is not very web-2.0-ish. Another possibility is to allow to store the data on a WebDAV store of choice for the user. Flickr then would just be a front-end to the Pictures directory I have somewhere on a WebDAV server. This is kinda what Zoho and Omnidrive are doing. It is possible to open word documents stored in Omnidrive directly in Zoho Writer (an online word processor). Problem is that they use Omnidrive’s non-standard protocols. Not that I have anything against Omnidrive, but as of yet I have no reason to believe Omnidrive will be the ultimate storage solution for everyone.

Record Skype Calls (Windows and Mac)

by Zef Hemel

One of the most requested features of Skype is the possibility to record calls. As of yet, it still has to be implemented into Skype itself. But that is no longer that big of an issue, because there are now third-party plug-ins to do this.

For Windows there’s the free Skype Recorder. Apparently this program can also record other VoIP phone converstations (such as Google Talk). Personally I haven’t tested this software, but it should work.
Skype Recorder (Windows)

For the Mac there is Ecamm’s Call Recorder, which costs $13.46. I tested this myself and it works really nicely. It integrates into Skype itself (also adds a tab to its settings window). It records the conversations into a Quicktime movie, which sounds weird, because it’s only audio. The reason they do this is to be able to record two tracks separately. On one track it will be you talking and on the other the person(s) you’re talking to. This way you can edit each track separately. Little utilities are included to easily mix-down this movie into AAC or MP3.
Call Recorder (Mac)

Flickr, Is That RDF You’re Cooking?

by Zef Hemel

I was reading some articles at O’Reilly’s excelent O’Reilly Radar weblog. Then I read this:

Flickr, Yahoo’s amazing photo-sharing site, has added another tag based feature — this one aimed at developers. They are now supporting machine tags (or triple-tags). What are those you might ask?

Triple tags? Triples, that rings a bell. So I decided to follow the link to the announcement at the flickr site. And guess what do I read?

# What is the spec for machine tags?

Machine tags are divided in to three parts :

1) A “namespace” :

Namespaces MUST begin with any character between a - z; remaining
characters MAY be a - z, 0 - 9 and underbars. Namespaces are
case-insensitive.

2) A “predicate” :

Predicates MUST begin with any character between a - z; remaining
characters MAY be a - z, 0 - 9 and underbars. Namespaces are
case-insensitive.

3) A “value” :

Values MAY contain any characters that a “plain vanilla” tags
use. Values may also contain spaces but, like regular tags, they
need to wrapped in quotes.

Namespace and predicates are separated by a colon : “:”

Predicates and values are separated by an equals symbol : “=”

For example :

* flickr:user=straup

* geo:locality=”san francisco”

Still… that reminds me of something… Hmm… So for each picture, you can add any random predicate (property) with a value. So basically you end up with a bunch of ([picture], [predicate], [value]) tuples right. Hmm.

So let me paraphrase this thing. Flickr always has known tags, which could give you some more hints about what the picture was about. Keywords if you will. These were additional pieces of metadata, information about data (pictures in flickr’s case). Right now these tags can be structured, so there is room for a property/field/predicate/whatever-you-want-to-call-it name and a value. And this predicate has a namespace. Hmm. So what they create is a kind of framework to be able to describe pictures. Interesting. Wouldn’t it be a nice idea to generalize this and develop a framework like this to describe any resource?

We could call it, hmm, say… RDF?

OpenID, baby!

by Zef Hemel

OpenID LogoA post on Intertwingly (Sam Ruby’s blog) brought me back to the idea of single sign-ons. A year and a half ago I came up with SPTP, the Simple Profile Transfer Protocol, which turned out to have a major flaw. Then I looked at SINP. But the problem is that it’s kind of hard to get a new system off the ground.

And honestly, there already is a system that is pretty slick.

It is called OpenID and it gets more and more support. The idea is that you login to OpenID-enabled sites using a URL that you own. This URL links to an OpenID server where you have an account to authenticate you. If you look at the HTML code of zefhemel.com right now you will see two new tags in the head:

<link rel="openid.server" href="http://www.myopenid.com/server"/>
<link rel="openid.delegate" href="http://zef.myopenid.com/"/>

This means that if somebody (me) uses http://www.zefhemel.com to authenticate somewhere, the MyOpenID servers will handle this authentication. MyOpenID is a free service where anybody can get an OpenID, you don’t need a website, by default you get a http://username.myopenid.com address. You can use this address to login to any OpenID-enabled website, or you can link your current website to your MyOpenID account (like I did). On MyOpenID you just fill in your profile (you can create multiple), with information that you want to make available to the services you will use your OpenID for.

Authenticating on OpenID services is easy. You simply type in your OpenID URL and press login. That’s it. The first time you do this, the OpenID service will ask you to authorize this service to get access to your user information, you can authorize once, forever or decline. It’s that easy.

What makes or breaks a system like this, is whether services actually use this. More and more services start to support OpenID. A few examples:

There are a number more on MyOpenID’s directory page. There is also a WordPress plugin.

If you’re developing a web application, consider supporting OpenID. There are libraries available for most programming languages so it shouldn’t be that hard to integrate. It would make a lot of lives easier if you would.

The Browser as the Platform

by Zef Hemel

I wrote about this a couple of times before, but as time progresses I start to see the picture clearer and clearer: operating systems will increasingly become less important. Forget about what OS you’re using, it’s just a frame around what really matters: the browser. Sun used to say: the network is the computer. I would add that indeed the (inter)net is the computer, with the browser as in increasingly thick client.

The browser used to be fairly limited in what it could do. It could show text and images, and that was about it. Then there were some companies that helped making websites more dynamic, some succeeded (notably Flash) and many failed (I would say, including Java applets). However gradually things were added to the browser that allowed for very dynamic behavior. Early on there was this little scripting language called Javascript, which could show simple dialogs and could manipulate the webpage a bit on the user’s screen. When all browsers started to support some kind of means to make HTTP calls back to the server through Javascript, this turned the browser into an incredibly powerful platform. The application platform of the future, I would say.

The current trend is that more and more of the web applications move a lot of their logic to the browser. This is what I predicted a long time ago already. I think that soon most of the application will run inside the browser and will only make callbacks to the server for storing and retrieving data. In fact this is basically what Gmail does.

It is important to make a distinction between two kinds of dynamic websites:

  1. Sites with some dynamic content. These are sites that use Javascript and AJAX techniques just to make the experience more smooth, but do not rely on it. These sites have multiple pages and reload the whole content for most things you do. Wikis are an example of this, as are blogs and most content websites.
  2. Single-Page Interfaces (SPIs). These are the web applications that were built to replace the traditional desktop applications. You go to a single page and do not leave it for as long as you use the application. Examples of this are Gmail, the new Yahoo Mail, and many online feed readers, like Google Reader.

The real revolution is happening with the SPI web applications. These applications have important implications however and pose a number of problems:

  1. Current SPIs assume an internet connection at all times. This is unrealistic and will be unrealistic for some time. So there should be some means to download data to the browser, change it and push it back to the server when there is internet connectivity again. Dojo’s storage framework makes this browser-side storage of data possible.
  2. Javascript is going to be the dominant programming language. I think Javascript is fit for the job. It has to be, developers will have to build on the current Javascript engines currently around for many years, most people do not update their browser often. A lot of interesting work is being done on making writing software in Javascript easier and more pleasant. For browser makers it means they really should start focusing on their Javascript interpreters. As more and more code will run inside the browser this is of increasing importance.
  3. Where is the user’s data stored? Currently most web application providers store the user’s data on their servers. The question is, do users want this? At least it should be possible to move this data around, from the service to the user, but also between services I think. This is an issue not currently addressed, and I will talk about this more in a future post.

To be continued…

The Joy of Reinventing the Wheel, part I

by Zef Hemel

Lately I have discovered two of my ideas have already been invented before. The first one concerns this secret project that I mentioned a while ago, I will post about that later some time. The other one concerns this TV Torrent Feed idea I had.

Yes, this idea already had been around since 2003 apparently and was dubbed broadcatching, yes, there’s even a wikipedia page. How nice.

In December of 2003, the term broadcatching was used by Steve Gillmor to describe the combination of RSS and BitTorrent peer-to-peer file sharing as a method for subscribing to an ongoing series of media files from a website. The combination of these technologies allows a computer connected to the Internet to act like a digital video recorder (DVR) such as TiVo connected to cable.

[...]

Today, RSS and BitTorrent based broadcatching provides a web based distribution channel capable of delivering broadcast media to a large group of consumers at a low cost. BitTorrent provides the low cost method for distributing large files to a large group, and RSS enables a website to easily provide a subscription to a series of BitTorrent files.

D’oh!

JSON vs XML

by Zef Hemel

Interesting article on Quoderat:

In the current JSON vs. XML debate (see Bray, Winer, Box, Obasanjo, and many others), there are three things that important to understand:

  1. There is no information that can be represented in an XML document that cannot be represented in a JSON document.
  2. There is no information that can be represented in a JSON document that cannot be represented in an XML document.
  3. There is no information that can be represented in an XML or JSON document that cannot be represented by a LISP S-expression.

They are all capable of modeling recursive, hierarchical data structures with labeled nodes. Do we have a term for that, like Turing completeness for programming languages? It would certainly be convenient in discussions like this.

Next Page »