ASIDE: We face a firehose of content in our daily lives too. Definite tone of "how to cope" at this fowa. See also new Fav.or.it beta work.
Risk: hear only what you want to hear, Narrowing of experience.
Filters can still flood.
How do we define relevace? Interest to all, interest to those I know, importance to 1 person (personal)?
Topic clustering; comments, 'I like this'; Global importance.
reccomend by 2 or more friends - increase accuracy.
Realtime Friend Feed coming soon.
Scalability - Can't really change schema on multi-million row tables.
memcached to MySQL, stateless, replicated DBs
The comment problem - comment where and to whom?
API for getting comments *from* FF - but you still have to hunt & pull, and I'm sure it's not an open API.
"It's interesting that so many talks at #fowa end up being about collaborative filtering, recommendations and basically social science - robertgaal (Robert Gaal)"
I can' tblog this, if I tried to keep up with Andrew my pingers would combust.
"Scaling is a nice problem to have" - until you have it!
"Solving Sandy's Problem" - target audience is self - known audience.
Scaling reactively = ARGH!
"We're synchronous. That's hard."
Page persistence time: 2.5 hours.
*no* site content.
"We didn't have a DB server - we still don't need what we do have".
Start simple - use OSS tech, don't try and write it all yourself/at once.
Based on GAIM, but extracting /replacing the UI layer.
Turn on; 597 Diggs, servers melt.
Didn't scale before it was needed (known risk). You're probably guessing the wrong ones.
Effectively running 100's to 1000's of gaim instances.
Bottleneck: Not CPU, not memory: tools (Strace, gdb etc) showed gaim bug (double-free).