Inkwell: Authors and Artists

    
Our next interview is with Cory Doctorow.
  
Cory is a self-described "Renaissance geek" with fingers in a lot of pies:
author, journalist, critic, technocrat, entrepreneur, obsessive
collector...obviously, he doesn't get a lot of sleep.
  
Cory won the John W. Campbell Award for best new science fiction writer at
the 2000 Hugo Awards in Chicago, and his short fiction appears regularly
in all the notable magazines and anthologies in the field -- most
recently, a story in the December Asimov's, "I Love Paree," written with
Michael Skeet. His first book, "The Complete Idiot's Guide to Publishing
Science Fiction" (written with Karl Schroeder) is on shelves now, and his
second, a novel called "Down and Out in the Magic Kingdom," is at a
publisher now.
  
Cory has written feature stories for Wired Magazine and elsewhere, and is
a regular book and technology reviewer for Wired. His journalism is
fuelled by good narratives and wicked-cool technology.
  
The venture that's currently eating Cory's life is openCOLA
(http://www.opencola.com), a company he started with two partners in June
1999. Now with 50 employees and offices in Toronto and San Francisco,
openCOLA is a funded, commercial open-source initiative that's building a
"relevance-switching" mechanism for peer-to-peer networks. The idea is to
marry collaborative search technology to self-organizing networks and in
so doing to create a viable means to evaluate every document in the world
and determine its idiosyncratic relevance to every human being on the
Internet. This is just as ambitious as it sounds, and is very, very
exciting stuff.

Cory's title at openCOLA is "Chief Evangelist." Where traditional software
companies have sales forces that sell licenses, openCOLA has evangelists
who sell the ideology -- the software is free. Chief Evangelist is not so
far from Spokesmodel, with a little bit of bottlewasher and oddsbody
thrown in. He's just moved from Toronto to San Francisco to take on the
role, and is getting to feel like a for-real Californian (he was just
named "Customer of the Week" at the local Peet's Coffee!).
 
Cory will be interviewed by Mark Frauenfelder, whose excellent piece about
openCOLA for The Industry Standard, can be read at
http://www.thestandard.com/article/display/0,1151,19498,00.html
  
Mark, a writer and illustrator from Los Angeles, is the founder of the pop
oddity zine Boing Boing, and the co-editor of The Happy Mutant Handbook,
which was published in 1995 by Putnam Berkley. He worked at Wired Ventures
from 1993 to 1998, where was an associate editor for Wired, the founding
editor-in-chief of Wired Online, and an editor at Wired Books. Today, Mark
writes for Wired and The Industry Standard, and writes a monthly column
for Playboy magazine called Living Online. He also draws cartoons for
Guru.com and Digitallivingtoday.com. Mark has been a member of the Well
since 1988. You can see his work at http:/boingboing.net/markf.html.

Please join me in welcoming Cory and Mark to inkwell.vue!
 
  

    
Well, I'm here, having just wrapped upa fine week at Comdex, culiminating in
the development for a cold. Flying to Philadelphia for my annual appearance
at Philcon this afternoon, then to NYC for the Science Fiction Writers of
America booze-n-shmooze, then to Montreal, then, (maybe) back to San
Francisco long enough to change suitcases before I leave for Hong Kong.

Yikes.
  

    
Cory has pockets full of gadgets that keep him connected while he's on
the road and in the air. Cory, why don't you empty out your pockets
right now and tell everybody what you are carrying?
  

    
(And I want to know how you can do the Comdex to SFWA transition... I hope
you have a half gallon of Cuervo Gold somewhere in your luggage!!)
  

    
In my pocketses:

* RIM Blackberry pager

* Gerber BYO Multitool

* Gerber Legend multitool

* White-light ultraminiature 10,000h LED flashlight/zipperfob

* Wallet

* Change

* Eyemodule

* Backup module for Visor

* Modem for Visor

* 8MB flash for Visor

* Motorola i700 plus ruggedized cellphone

* Xtreme CX-20 rugged cigarette lighter (rated for 500' drops!)

* Visor Platinum in Concept Kitchen Bumper case

* IDEO dual-mode Visor stylus

The hard part abot shlepping all this crap around is going through airport
security. I've switched to wearing a jacket with a big marsupial pocket so
that I can fill it up with all this stuff and drop it on the x-ray.
Otherwise, it can take up to 20 minutes to divest myself of all the metal I
habitually carry.
  

    
The Comdex transition wasn't easy, but mostly because I'm coming down with a
cold -- but my panel last night was "Hackers, Crackers and Wackos" with Eric
Raymond and Bud Sparhawk and Ann something-or-other. A room full of
sysadmins and a panel full of nerdy writers. It was gold.
  

    
Bagginses!
  

    

Hi, Cory!

I met Cory at Philcon yesterday and 'evangelism' barely describes his
energy. He didn't talk like he was suffering from a cold and his enthusiasm
for openCOLA helped me make sense of it.  But, Philcon did not have his book
for
sale and this must be fixed.
  

    
So, Cory, what's all this hubub about peer-to-peer, how is it different from
host/client, and what's cooking at openCOLA?
  

    
Thanks, (viv) -- it's great to hear that I can make openCOLA sound sensical!

--

(satyr), that's a giant question! I'll tackle it in pieces: some now and
some later.

First of all, let's start with what's wrong with the acronym P2P. I wrote a
piece of Mindjack where I sum this up:

http://www.mindjack.com/feature/p2p.html

>The worst day in short history of modern peer-to-peer computing
>was the day that some wag coined the banal acronym "P2P." On that
>day, an entire field of wonderful, mind-bending, world-changing
>networking models were reduced to a single, oversimplified
>acronym-with-a-two-in-the-middle (AWATITM).
>
>The other AWATITMs -- B2B, B2C -- were used to make simple ideas
>(selling stuff to people and selling stuff to companies)
>excitingly complex. Entire PowerPoint cosmologies were
>constructed around B2B and B2C, and MBA programs rejigged their
>curriculum to explain the theory and practice of AWATITMs. The
>dirty suits grabbed ahold of the idea, pumped and dumped billions
>of dollars in bad paper on the NASDAQ and then hastily declared
>B2B and B2C dead.
>
>Turning peer-to-peer computing into P2P implies some common
>lineage with B2B and B2C. It places peer-to-peer computing
>squarely in the safe and comprehensible realm of the monetized,
>commodified Internet of Superbowl ads and Times Square
>billboards. But peer-to-peer computing isn't a business model,
>it's a *technological* one. That's not to say that you can't make
>money with peer-to-peer technology. (You can. People will. Lots
>of it.) But peer-to-peer computing isn't a way of making money
>per se, any more than packet-switching or error-correction is a
>way of making money.
>
>What peer-to-peer computing is, is a way of making really cool
>decentralized networks. In a peer-to-peer universe, servers are
>irrelevant. Peer-to-peer networks are composed of personal
>computers tied together with consumer Internet connections, each
>node a quantum zone of uncertainty, prone to going offline
>whenever its owner closes his laptop and chucks it into a
>shoulder-bag. In peer-to-peer computing, reliability is ensured
>through massive redundancy; bottlenecks are dealt with in the
>same fashion. Peer-to-peer networks aren't owned by any central
>authority, nor can they be controlled, killed, or broken by a
>central authority. Companies and concerns may program and release
>software for peer-to-peer networking, but the networks that
>emerge are owned by everyone and no one.
>
>They're faery infrastructure, networks whose maps form weird
>n-dimensional topologies of surpassing beauty and chaos; mad
>technological hairballs run by ad-hocracies whose members each
>act in their own best interests.
>
>In a nutshell, peer-to-peer technology is goddamn wicked. It's
>esoteric. It's unstoppable. It's way, way cool.

That's the rant on why P2P is a stupid acronym.

As to why it's good, well, here's whaqt makes it awesome for openCOLA:

When we started out, we were thinking that oC would run on a centralized
server, and that the distributed part would be the "spider" -- the part that
fetches new pages from the Web and other resources to be integrated into a
database and measure for relative relevance to various user-profiles.

But when we started adding correlative features -- the ability to search for
a robot based on your "relevance-proximity" to other users, the ability to
automatically discover relevant material from correlated users, etc, we ran
into an N-P Complete (or at least N-P Hard) math-problem. These are classes
of problems that require a nearly infinite amount of calculation to solve.
When we projected the computational horsepower required to do real-time
correlation of say, two billion documents to, say, 100 million robots, we
realized that we'd need an ungodly amount of hardware -- way more than we
wanted to buy, and even then, it didn't look like it'd work all that well.

But with self-organizing, peer-to-peer networks, we can make horsemeat out
of N-P Complete. Think of a simple analogy:

You have a thousand people in a subway station, and you want to arrange them
by height. The server-side, top-down approach would be to take a tape-
measure and measure then entire crowd, sort them out with a spreadsheet,
then put them all in order. It's do-able, but just barely, and impossible if
everyone keeps growning, shrinking, coming and going.

A peer-to-peer approach might go like this: line everyone up, and give them
this instruction: if you're taller than the person on your right, switch
places. Keep repeating until everyone's in order. This is a really crude
algorithm, just a bubble-sort, the kind of thing you're exposed to in a
high-school computer-science course. But even with something as inefficient
as bubble-sorting, a self-organizing group will quickly find itself standing
in something like order. Further, with every "computation" (place-swap), the
system approaches order. In the top-down approach, order only happens once
the entire calculation has been executed.

So that's why distributed computing works so well for openCOLA.

As to what's cooking, give me a couple hours, I'll get back to you.
  

    
This reminds me of a clown/theatre exercise I teach.

People walk in a room at various speeds.  They don't walk in circles, but
they make sure to change their focus and their rhythm of walking, and
their destination.  They also make sure not to bump into anyone else.  I
ask everyone in the room as they are walking to pick out two people to
keep track of-- two people who they will know where they are at all times,
they are always aware of where their two people are. They are not to let
anyone know by word or movement who those two people are.  When every
person has picked two people, I ask them to form an equilateral triangle
with their two people.  An equilateral triangle in space, so that there is
equal distance between the two people you have picked and yourself.  If
one person moves, you must move yourself to keep the unity of the
triangle.  

The point is to hopefully find a balance.  If the group can do this
consistently (which is very hard) then the point becomes to have the
entire group move in space without losing the sense of the triangles.

It's a very complex exercise...
  

    
And impossible to manage from the top down.
  

    
Exactly, it requires setting the people in motion and relying on their
skill guided by their experience in order to make this fragile exercise
work.
  

    

Here's a question that I've been contemplating, ever since I read the
OpenCola web pages a few weeks ago:
Do you expect to have to overcome paranoia concerning the running of servers
on the client computers? I _know_ it's open source, and so I know that there
will be people checking to make sure there are no backdoors, trojans, etc
hiding in the code... but I would suspect that people would fear it anyway.
After all, there are security holes even in good open source projects- the
nice bit isn't that they aren't there but that they are found and fixed.

So, yeah. Do you expect to have to overcome a certain amount of user
paranoia, and do you have a plan to do so?

(I almost asked this question of slashdot, though in a more general form and
using OpenCOLA as an example...) (not that I expect they would have posted
it.)
  

    
The short answer is that yeah, oC is open source, yeah, we've retained the
services of @Stake (formerly 10pht Heavy Industries) to do code- and
architecture-level security reviews -- we're not relying on the good will of
the open source community to audit the thing for us -- but in the end,
people weigh security against benefits. All kinds of applications open
security vulnerabilities on one's computer, starting with connecting it to
the Internet at all! People run Outlook, Windows, Instant Messenger,
Napster, Gnutella, and a thousand other network apps, despite the well-
publicized risks inherent in each. But there are tremendous benefits to
running network-aware apps, and these benefits outweigh the risks. Of
course, we believe that we've embraced the critical success-factors for a
secure app: open code, security audit, eschewing perishable, crypto-based
copy-protection in favor of reliable network-audit based Digital Rights
Management.
  

    
I've got a thick book at home called *Distributed Algorithms* that
goes into things like how to get consensus among peers even when there
are network failures.  It's tricky stuff and tough to get right - heck,
even two-phase commit has a problem with in-doubt transactions if you
get a network glich at the wrong time.  Seems to me that any
distributed algorithm can be greatly simplified by adding a central
server, even if it doesn't do all that much work.  Given all this, what
problems are there to be solved that make a truly distributed network
worth all the trouble?
  

    
Oops, I should have said network outage rather than a network glich -
temporary network gliches are handled by TCP/IP.
  

    
Brian, that's a tough question. I don't want to shoot off my mouth about a
subject on which I'm not an expert, so I've passed this on to our research
scientists, Ethan Davis and Dan Moniz. I expect they'll get back to me
tomorrow some time, and I'll post the response tomorrow.
  

    
Trying to look prescient, here, but I'm betting the answer has something to
do along the lines of "that depends on how you define 'answer'".  To acquire
a truth, or universal consensus, probably would be greatly aided by and
might even require a central store to hold all of the info.

But oC (as I understand it so far) works more on the gossip model.  You
don't need all the info everyone has when you figure that most of the local
clusters have most of the information.  Since the 'answer' is "what is the
cool thing you know about right now" and not "what is the most maximally
cool thing I could ever hope to know", oC gives good answers right away and
better answers as the knowledge disseminates.  It seems pretty hiccup-free
to me.

(Cory is of course being responsible by deferring your answer to his
experts.  I am practicing for the role of media pundit by declaiming on a
subject without benefit of actual fact.)
  

    
O'Reilly has a recent (11/14) interview with Ian Clarke of Freenet fame
up at...

   <http://www.oreillynet.com/pub/a/p2p/2000/11/14/ian.html?page=1>

...from the 3rd page of which there's a link to a directory of
peer-to-peer projects.
  

    
Brian, here's an answer from Dan Moniz, one of our research scientists (who
aslo sternly reminds me that our centralized database problems with O^N, not
N-P Complete):

--

From my own perspective, calling what we intend to do, and indeed, what
everyone else in the space seems to be doing "p2p" is a limited, extremist
view. Napster isn't *really* p2p because it uses an index. I think what's
more powerful here is the idea of what what I like to call blank2blank.
Here's another buzzphrase I just made off the top of my head that fits
adequately: Dynamic hybrid network topologies. The whole point is more to
get away from the de facto centralized nature of the Internet and start
sharing more resources in an efficient manner, where approriate, which is
what openCOLA is about.

Mark Miller, who I talked with face-to-face for the first time while out in
California recently, and who I have been communicating with on various
projects longer than I've been with oC. Mark is behind interesting things
like agoric computing and distributed capabilities and he had an interesting
take on p2p. He likened it to an airport/airplane metaphor: technically, any
airplane can fly to any airport, but in the real world, because of
population density, the amount of business conducted in a given city,
remoteness of a location, etc. we find that some airports are naturally
busier than others. So it isn't so much c2s (client- to-server) v. p2p
necessarily, it's "hub & spoke". Hybrid models, constructed as necessary.

And sure, openCOLA is in the p2p space. We're in the c2s space too. We're in
the rabbit2ocelot space! We're trans-spatial, which is hard to label and
pigeonhole, but is pretty critical. The whole point is to make the network
layout change dynamically with the needs of users on the network -- witness
the grouping power of networklogically close peers via the relevance engine,
for instance.

So yes, servers, even that do very little, can un-complicate a lot, agreed.
But what's potentially more at issue is the ability for people to be able to
determine their own involvement by participation. *That's* new. So people
can set up a root clerver for their exclusive collection of Doctorow's Best
Fart Jokes if they like, all the while running other clervers that host
robots scouring for entirely different media, sharing nothing. It's an
agoric computing scheme with a market based around reputation, in turn based
on what you offer to the *community* (another key oC concept), so people
that provide will find themselves better off than those that don't (just
like people who maintain jobs v. homeless -- although there are obvious
social/political variables there).. And for what it's worth, we do use
servers, as an optimization, for instance, in _____'s [top secret project
name omitted] current design. They're not *necessary*, but they help.

As for why it's worthwhile, for me, it's a problem space with a lot of
interesting possible solutions and a lot of smart people working on those
solutions, with enough space for me to make an impact for the positive (I
hope). If no one ever did anything simply because it was difficult, well, we
wouldn't have gotten this far, that's for sure. Also, I think the fact that
so many people have adopted p2p and related technologies thus far says
something, even if all it is is "hey, we like this stuff!". We need more
software the world likes. So it may or may not be terribly efficient. I
think that c2s breaks down in a number of circumstances where p2p succeeds,
and vice versa. For me, it's not about being a fundamentalist on either side
of the coin, it's about evolving the state of the art.

--

Dan's at dan@opencola.com. He's a good guy to talk with if you're looking
for more detail.
  

    

If we get any more technical, we are all gonna have to sign NDAs!
  

    
Cory, can we back up a bit to what you said earlier:

> But peer-to-peer computing isn't a business model,
> it's a *technological* one. That's not to say that you can't make
> money with peer-to-peer technology. (You can. People will. Lots
> of it.)

If you already answered this, I missed it and I apologize. If not, just HOW
can people make money with this? "People" being the openCOLA operation,
in this case. Though I only have a very hazy comprehension of what you're
proposing, I assume there's equipment, labor and a physical site where
the equipment and the labor is housed. There's got to be some overhead,
right? 

How does openCOLA generate income off this?
  

    
AND...do you see individuals being able to sell their unused computing power
when they are on high speed connections 24/7 and only using for an hour or
two a day? Is it like taxes where the real burden is carried by the broad
middle class? Isn't the greatest computing resource all of this unused
potential?

ALSO...just reading about beowulf clustering in Wired isn't this a pretty
good idea for all those unwanted computers we're reading about having to pay
a charge to send to the landfills? www.beowulf-underground.org.
  

    
How does oC make money? That's actually pretty straightforward. oC finds
stuff, moves stuff, provides for digital rights management on that stuff,
and a transaction engine so that the authors of said stuff can charge for
it. oC makes the bulk of its revenue from fees taken out of the transaction.
This is the classic relationship between a sales-agent and a creator: the
sales-agent works on a commission, only making money when the creator does.

As to cycle-brokering, that is, farming large, computationally intensive
tasks out to consumer machines for crunching, that's an application that
*could* be built out of oC's kernel, but it isn't something that we're
working on at the moment. There are a number of reasons for this:

* A number of very good projects have already sprung up to fill this niche,
particularily distributed.net (my favorite), SETI@home, Cosm, and on the
commercial front, Popular Power.

* Not many jobs are that computationally intensive: when was the last time
you waited for a spreadsheet to recalculate? Jobs that *are* suited to such
distribution are either noncommerical (scientific research), or confidential
(insurance company reports, Postscript RIPs, rendering). As such, this just
doesn't seem like a huge opportunity to either make a bundle or change the
world.

I *love* Beowulf clustering. Some day, I'd like to get myself a few hundred
free 486s, put Linux on them, and use 'em to build the world's first
gigabyte floppy-disk array.
  

    
thanks for the pointers.
don't know how i avoided knowing about distributed.net so far.