I worked on a network visualization tool in processing for my Ubiquitous Computing class. It observes network traffic and attempts to geolocate the servers you communicate with. It isn’t easily installable at the moment, but I hope to rectify that in the near future.

To us, the Web is a sort of shared external memory. We do not have to remember unnecessary details: dates, sums, formulas, clauses, street names, detailed definitions. It is enough for us to have an abstract, the essence that is needed to process the information and relate it to others. Should we need the details, we can look them up within seconds. Similarly, we do not have to be experts in everything, because we know where to find people who specialise in what we ourselves do not know, and whom we can trust. People who will share their expertise with us not for profit, but because of our shared belief that information exists in motion, that it wants to be free, that we all benefit from the exchange of information. Every day: studying, working, solving everyday issues, pursuing interests. We know how to compete and we like to do it, but our competition, our desire to be different, is built on knowledge, on the ability to interpret and process information, and not on monopolising it.

Geographic Efficiency.

There is an ultimatum that it will take at least 67ms for my traffic to get to the other side of the world – namely the speed of light.  If that was true, we would be in great shape.  In reality, it generally takes 200 to 250ms to go around the world, and if you want to get a response that means waiting 400-500ms.  There are two reasons for inefficiency.  First, data is not going to travel on the shortest path to the destination – there isn’t a wire on that path.  Instead it will travel through the various autonomous systems which form the Internet.  These systems are connected in an economically driven way, which compete over choosing the true shortest path.  Second, there isn’t a single wire connection you and the destination.  Instead, there are many (expect 10-15 for continental US, more for other continents) routers that the data will pass through.  Each router need to figure out which of the many wires connected to it my data needs to be sent out on.  To do this, they actually have to spend the time to read the data, check that it has a valid checksum and  is ‘intact’, and then put it in queue after any previously routed data to the outgoing wire.  A substantial fraction of transfer time is spent on routers, rather than on the wire.

Technical Efficiency.

Applications are built on top of this network, and have to cope with delays that are noticeable by users each time data needs to be transferred.  How do they do it?  The first realization is that there are a couple common types of ways that data will need to be transferred.  One is in ‘near realtime’, with examples like phone calls, video chats, and the like.  For this, there is a UDP protocol, which offers best-effort delivery – you bundle your data into small bundles and send them out.  There is no guarantee of if or when they’ll get to the other side – okay, because if there is a problem it isn’t worth waiting for that data to show up much later, the damage has already been done.  On the flip side, the other common protocol is TCP, which guarantees the reconstruction of a data stream.  The potential downside in this conversation is that a tcp connection has an initiation phase.  This takes 1 round trip (you send a request, the destination acknowledges it, and then you can begin sending data).   Tick off 400ms if you were trying to get somewhere in europe or asia.

Once a connection is made, you have to actually request the web page you want.  This is done using the HTTP protocol.  The main cost that is worth noting with HTTP is that it allows fetching of individual resources.  If I want the page, I ask for it, and the server responds.  In the response, there may be a reference to an image.  I will need to the request that image, and get it in a response – the protocol doesn’t let the server send me additional resources until I ask for them.  (Why, you might ask, would you make pages composed of multiple resource if you care about speed?  The answer seems to be that I might already have some of those resources saved in a local cache, and won’t necessarily need to request them from you.)  The upshot is that at best there are two additional rounds of communication required to get all the resources together to actually show a web page.

Reality.

This chart shows averages for how long it took to load each of 100 different web sites.  The requests were made from planet lab, a geographically-diverse set of university computers (which are not terribly reliable.).  Ignore the tails, which are products of requests being banned.  This is indicating that for an average (around the world, not average in the US) user, it takes 3-5 seconds to load a web page.  That’s horrible!

Otherwise, this site is now serving dual purpose as both a stream-of-consciousness dump, and as my primary identity hub.