-
‹ Home
Contents
-
Categories
-
Tags
-
Archives
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- July 2008
- June 2008
- May 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2007
- October 2007
- September 2007
- August 2007
- July 2007
- May 2007
- April 2007
- March 2007
- February 2007
Projects
-
RSS Feeds
-
Meta
Bitching about parallel computation
I Want A New Platform - at Union Square Adventures - can't handle a new platform!
The core of this issue is that we're in a **PARALLEL COMPUTING ENVIRONMENT**
That's really the issue. You're planning on using multiple CPUs to do the work, and the architecture bottlenecks at your ability to get the centralized components of the architecture to behave.
Gee... didn't the supercomputer guys have this problem about 30 years ago?
A single system, serving hundreds or tens of thousands of concurrent users is going to have these problems. The more interactive it becomes, the more **LIKE SOFTWARE** your web site is, the worse the problem will be.
You think you have problems? Check out:
http://disastr.org
and imagine this problem in the context of sixty million people hitting your system in a three hour window to try and figure out how to save themselves from a disaster.
Can it be done? Sure. But the STORM botnet is a your model, not your current web applications framework.
Start thinking of swarms of disposable entities serving your web sites. It's not about taking a single webserver app and scaling it no no no. It's about writing parallel supercomputer cluster software that happens to be user-interactive rather than number crunching.
Nodes that carry data. Migration over the network to spare compute resources. Local session data that's spooled to disk replicated on the fly rather than databases.
SQL *on the node* if you need it (think SQLite) rather than an Oracle install.
It's not easy: people in the supercomputer world have been worrying about this since the 1980s, maybe earlier. But that's what you're up against here: supercomputer sized computations with irregular datasets and complex interdependencies.
If you haven't thought about this stuff, you need to go back to the Transputer papers and the Occam programming language. Occam is really the best single source for clear thinking about parallel computation as a fundamental part of software design that I am aware of, and the Transputer is an idea who's time is not yet come - first we need parallel programmers, *then* parallel computers.
Once you've got your head around that - and things like the "Crazy Postman" algorithm and why they worked in a transputer environment - then you're ready to get down to writing code for these massive new generation applications.
Post a Comment