When presented with the task of distributing large volumes of data over the internet to large audiences, application architects encounter data integerity, performance and reliablity hurdles. These challenges are often addressed using sophisticated hardware solutions or external content based networks. Both approaches, while proven and tested solutions, are also quite expensive and often outside the reach of many projects.

Justin Chapweske, CTO of Onion Networks, presents swarming as an alternative that promises to deliver scalable, reliable and fast content delivery without a sophisticated architecture, without content network fees and without disrupting existing web clients.

After reviewing both the benefits and limitations of HTTP, Justin describes how Onion Networks' swarming technology can be used to enhance new and existing web applications where the requirement to transfer large data files and stream large volumes of data is becoming ever more common. In particular Justin describes how Onion Networks uses, as a basic block, "self healing" transfers to transparently correct corruption or tampering of transferred data. By comparing swarming for data content to RAID for data storage Justin describes how remixing existing HTTP, cheap bandwidth and commodity hardware can produce "golden bandwidth" and reduce implementation costs significently.

After suggesting some simple best practise tips for web application designers in order to allow their sites take advantage of swarming, Justin closes his presentation by encouraging us to learn from Google and to imagine the business oppertunities open to us by combining cheap commodity resources to produce valuable services.

---

IT Conversations' publication of this session is underwritten by your donations and:

"GoToMeeting is one of the best online conferencing tools I've come across so far. It is also one of the most economical. GoToMeeting lets you share your screen safely and easily through corporate firewalls almost instantly." Torsten Jacobi, Editor tjacobi.com

---

Justin Chapweske is the Founder and CEO of Onion Networks, Inc., a company based in Minneapolis, Minnesota that provides file transfer acceleration and network infrastructure software for enterprise applications. Justin is the original inventor of Swarmcast, the first ever "swarming" content delivery network and more recently has unveiled "Swarmstreaming", a new swarming technology that allows smooth progressive playback of video and skipping ahead without downloading the entire file.

Resources:

• The New Scientist Swarming Article

This presentation is one of a series from the O'Reilly Emerging Technology Conference held in San Diego, California, March 14-17, 2005.

For Team ITC:

• Description editor: Paul Power
• Post-production audio engineer: Bruce Sharpe
• Series Producer: Cori Schlegel