One of our read slave databases got itself in a bad way some time last night and we're working to get it back into shape. No data was lost; the remaining db is just getting crushed by the load, which is why a posting can take 10s of seconds to complete.
Thanks for bearing with us.
Commented and upvoted for an announcement regarding the lengthiness of commenting and voting.
15
KeyserSosaDec 10, 2009
+9
Your sense of irony knows no bounds.
EDIT: noseven7 wins this round. Clearly the bug is also manifesting itself with extra apostrophies when certain admins comment.
9
raldiDec 10, 2009
+5
'
5
KeyserSosaDec 10, 2009
+4
I't's g'e't't'i'n'g w'o'r's'e!'!'!
4
[deleted]Dec 10, 2009
+12
I am sense.
12
[deleted]Dec 10, 2009
+4
[deleted]
4
colkowalskiDec 11, 2009
+1
They're like apostrophes but cuter.
1
[deleted]Dec 10, 2009
+6
[removed]
6
KeyserSosaDec 10, 2009
+23
Yup. That means the machine is only virtually melting and virtually slow.
23
pavsDec 10, 2009
+3
I feel that ever since you guys moved to AWS listnook has been slower and been down longer. From my personal experience, EC virtual nodes are not comparable to dedicated servers in terms of raw power. I think they are over-rated.
Does anyone know any other online services as big or much bigger than listnook is hosted on EC?
**Edit:** on an unrelated note I use this [chrome extension](https://chrome.google.com/extensions/detail/geedigenhgnhbebebbjlidlalocdggjl) for listnook. Does it put any unnecessary load on the server?
3
KeyserSosaDec 10, 2009
+4
From our perspective, this isn't the case. We actually moved all of the core services in May, and most of the issues we've been having are scaling related (there are just too many of you and you're really persistent. ;)
4
jaxspiderDec 10, 2009
+5
> ...there are just too many of you and you're really persistent.
And here I am thinking that was a good thing.
before you go off on me saying... I clearly put a winking smilie... Well to rebuttal that... then you have an unclosed parenthesis.
Pick your poison.
5
KeyserSosaDec 10, 2009
+4
> And here I am thinking that was a good thing.
It is for most things, just not our hardware.
> then you have an unclosed parenthesis.
That was all part of my plan to include you in my parenthetical remark and you fell for my trap because --> )
4
jaxspiderDec 10, 2009
+2
I... I feel soo loved!
2
krispykrackersDec 11, 2009
+1
You guys are so freakin' sweet I think it's going to give me a cavity.
1
[deleted]Dec 10, 2009
+1
[deleted]
1
jedbergDec 11, 2009
+4
The servers scale linearly, but our software and budget do not.
4
pavsDec 11, 2009
+2
If I am not wrong, didn't you guys claim to have saved quite a bit of money since you guys moved to EC?
2
jedbergDec 11, 2009
+7
Yes, but that was 7 months ago. We've almost doubled in traffic since then. :)
7
pavsDec 11, 2009
+1
Wow, thats really impressive. Would you say it would have been much more expensive if you guys were still using dedicated servers like before? When I look at high-end dedicated server & bandwidth costs today, its really peanuts. But of-course I don't know the numbers you guys are going through.
Personally, I think optimizing servers to get the full potential can be an exciting and rewarding experience. Usually, if things works out as planned. :)
1
jedbergDec 11, 2009
+4
I think it would have been much more expensive to stick with physical servers. As an example, although our traffic has doubled since May and we're running 150% more "servers" than when we started (we've been able to optimize a bit), our costs have only gone up about 20% (we're running at about $18K/mo right now). Amazon keeps lowering their rates, so that helps too.
4
[deleted]Dec 11, 2009
+1
[deleted]
1
jedbergDec 11, 2009
+2
Having 10 servers costs exactly 10 times as much as one, but that's not the issue. The issue is that our software isn't written to take advantage of 10 servers.
More specifically, it is the data layer that is the problem. We can keep adding application servers all we want, but right now we have some bottlenecks in the data layer that need to be addressed before we can add more servers for the data layer.
2
KeyserSosaDec 10, 2009
+3
That works, and we do take advantage of it, but it doesn't work so well for databases since bringing up a new slave still requires shipping all of the data over to it, which can be really, really time consuming.
3
[deleted]Dec 10, 2009
+1
[deleted]
1
KeyserSosaDec 11, 2009
For more than 2 years actually. We currently have 4 write masters (each with a different primary task) and 5 read slaves (doled out by which tasks are most loaded).
Our current problem is that the voting read slave went down rather hard and re-replicating/rebuilding takes time as there is a lot of data and more coming in every moment.
0
Dan_FarinaDec 11, 2009
+1
Is there an interesting bug report for PostgreSQL to accompany this failure?
1
MozDec 11, 2009
+1
> **Edit:** on an unrelated note I use this chrome extension for listnook. Does it put any unnecessary load on the server?
It loads a small JSON file from the server for every comment (or maybe one file for each root comment—I'm not sure). It doesn't use a horribly large amount of bandwidth, but I wouldn't say that it's negligible.
1
pavsDec 11, 2009
+1
Thanks for the info. I think I will stop using it. It was mostly out of curiosity, trying out a lot of new extensions - it doesn't add any value to the listnook experience. Besides I don't care about karma points.
1
MozDec 11, 2009
+1
Try these three scripts:
* [Listnook Comment Live Preview](http://userscripts.org/scripts/show/37566)
* [Listnook Comment Boxes](http://userscripts.org/scripts/show/63628)
* [Listnook parent comment](http://userscripts.org/scripts/show/34362)
1
pavsDec 11, 2009
+1
Thanks! :)
1
diceyDec 10, 2009
>I feel that ever since you guys moved to AWS listnook has been slower and been down longer. From my personal experience, EC virtual nodes are not comparable to dedicated servers in terms of raw power. I think they are over-rated.
I have also anecdotally experienced this, however I imagine that the listnook staff has actual data from actual monitoring services.
0
jedbergDec 11, 2009
+1
We have not found that to be the case. We actually moved in May. It's really just scaling issues that have cropped up recently.
1
diceyDec 11, 2009
Do you have any references to documents describing sane resource planning policies with virtual servers? I think it would be difficult since the performance of a VM can vary depending on the load of other VMs which may be sharing the same physical hardware, as we saw in a recent posting comparing various cloud services. Do you just add in a fudge factor to account for this?
0
pavsDec 11, 2009
+1
This is a very good (though not exhausted) comparison of VPS performance that I recently read. http://journal.uggedal.com/vps-performance-comparison
Even before I saw that comparison, I did some personal number crunching benchmark with EC2 VPS and they are absolutely horrible.
1
diceyDec 11, 2009
Yes, thanks, that's the one I was thinking of. It was on the front page recently.
0
jedbergDec 11, 2009
+1
Actually, we have found the performance to be quite consistent. The only place that isn't 100% consistent is the disk, but that is usually within a small range.
The beauty of being able to summon new servers in just minutes is that we don't really need to do resource planning. We pretty much just guess, and if we're wrong, we do it again! :) (So far we've been pretty good with our guesses).
1
diceyDec 11, 2009
I see that you're on EC2, which was the only VPS provider that did show consistent performance in the [recent front page article](http://journal.uggedal.com/vps-performance-comparison). Good decision on your part, then ;)
0
PoltrasDec 11, 2009
+1
But the pain... It is REAL!!!
1
StevenDicksonDec 11, 2009
+1
I thought this stuff was in a clown somewhere...
1
[deleted]Dec 10, 2009
+3
[deleted]
3
KeyserSosaDec 10, 2009
We reorganized our request handling a little to try to deal with the load. Stalling on votes was jamming our webserver and slowing everything else down. Votes are still going through slowly, but everything else is now getting sent thru to a separate set of app servers.
tldr: no, but we're trying to work around it. :)
0
[deleted]Dec 10, 2009
+5
[deleted]
5
KeyserSosaDec 10, 2009
-1
Thanks for including the tldr there. I was having trouble finishing that sentence.
tldr: hi.
-1
[deleted]Dec 10, 2009
+2
I thought everything was in the cloud (amazons services)?
2
raldiDec 10, 2009
+17
Don't you get it? When your server dies in the cloud, it dies in real life!
17
KeyserSosaDec 10, 2009
+2
The servers are virtualized, but they still act like servers. There is still hardware behind them, and our postgres db can still get backed up waiting on I/O (as is the case right now).
2
jaxspiderDec 10, 2009
So basically... your virtual servers crapped out... So who was virtually shitting there pants trying to fix this?
0
KeyserSosaDec 10, 2009
+5
All I'm going to say is that the smell in virtual listnook HQ is virtually horrendous right now.
5
jaxspiderDec 10, 2009
+1
Just to let you know, I'm virtually laughing my ass off.
1
divshappyhourDec 10, 2009
+2
I commented somewhere, took a nap, woke up, and the comment still hadn't posted after the 45 minute wait. The second nap did it though. I propose napping while waiting for comments to post!
2
KeyserSosaDec 10, 2009
We also just updated the JS to spit back an error when the request times out. This should mitigate the need for naps. :)
0
divshappyhourDec 10, 2009
+3
Aww man, now I have no excuses for naps :( Thanks for all the work you guys are putting into this though, we really appreciate it :)
3
MulsanneDec 10, 2009
+28
as always I love the transparency.
Thanks for all of your awesome work
28
[deleted]Dec 10, 2009
+2
So that's not the cause of [this](http://www.listnook.com/r/pics/comments/acwhb/whats_wrong_with_my_listnook_envelope_pic/) continuing issue?
2
KeyserSosaDec 10, 2009
That is an unrelated rendering glitch in Safari that we're trying to figure out.
0
[deleted]Dec 10, 2009
+3
Gracias. I did submit a bug report to the "fixxit" place yesterday about it, before posting it to see if it was me.
3
fmissleDec 11, 2009
+2
I want a refund. These Service Levels are bullshit!
2
KeyserSosaDec 11, 2009
*\*\*Hands fmissle a sack of time and broken dreams.\*\**
0
[deleted]Dec 10, 2009
+35
10 seconds? Some of mine don't go through at all (I consider 5 min "at all").
35
andbrunoDec 10, 2009
+8
I tried posting a reply to one guy three times, and it sat on "Submitting..." for more than 5 minutes each attempt. I'd hit "save" and then go watch some MST3K on Hulu until it reached a commercial break, and I'd check on my comment: "Submitting..."
F***.
[Hell, I don't even anticipate this one will make it. I'm pretty much just typing for myself.]
Edit: I went to the bathroom, got a drink, and it actually submitted!
8
[deleted]Dec 10, 2009
+6
I know! I started just saving my comments in an excel file so I could put them up later, when listnook wasn't being slow.
6
DontNeglectTheBallsDec 10, 2009
+8
You're gonna need Powerpoint for that much winning style, I'm afraid.
8
[deleted]Dec 10, 2009
+3
I really wish I didn't read your name.
3
DontNeglectTheBallsDec 10, 2009
+4
[You and me both, lady. You and me both.](http://tcrc.acor.org)
4
shopcatDec 11, 2009
+2
Thanks for the reminder, I haven't done a little check in the shower for quite some time.
Edit: I mean BIG check, big, big, humongous check.
2
DontNeglectTheBallsDec 11, 2009
+2
Indeed, it's important. That, and the prostate check. Most people don't know that while one in eight women will have breast cancer during their lifetime, prostate cancer affects one in six men (and an astonishing one in four, for black men), with similar mortality rates as breast cancer. Yet, prostate and testicular cancers get a fraction of the attention, funding, and testing. It's important. *They're* important.
2
[deleted]Dec 10, 2009
+3
Whoa, MST3K is on Hulu?
3
andbrunoDec 10, 2009
+3
Yeah I know, it shocked me too. I first heard about it two days ago on Listnook.
3
kugellehrDec 11, 2009
+1
don t you have anything better to do??
1
andbrunoDec 11, 2009
+2
I could have been working, but I don't consider that "better".
2
asthehourglassturnsDec 10, 2009
+26
That would be tens-of-seconds my fellow commenter.
26
robotsongsDec 10, 2009
+1
Actually, that would be three-hundreds-of-seconds, my fellow commenter.
1
I0I0I0IDec 10, 2009
+16
Or thirty tens-of-seconds, my fellow commenter.
16
NotMarkusDec 10, 2009
+7
> ...which is why a posting can take 10**s of** seconds to complete.
7
[deleted]Dec 10, 2009
+5
CAN ANYONE HEAR ME!!!!!!
5
SyndromeDec 11, 2009
+8
Holy c*** i can hear text!
8
PoltrasDec 11, 2009
+3
Good news everyone!
3
StevenDicksonDec 11, 2009
**LOUD NOISES!!!**
0
BAMFDec 11, 2009
+2
STOP YELLING!!!!!!!!!!
2
Austin-GDec 11, 2009
+2
Loud and clear!
2
creator11Dec 11, 2009
+1
I can't hear text.
1
M0b1u5Dec 11, 2009
-5
And while "45 tens of seconds" is still technically correct - it's f****** disingenuous in the extreme.
Saying "submtting comments can take a f****** age, or just fail completely" would have been honest, and accurate.
Given that none of us are paying customers, why feed us bullshit, when the truth is right there, and easy to use?
-5
OduseiDec 11, 2009
+3
Paying customers are actually less likely to leave. Once you've paid for something, you feel "stuck" with it, and throwing it out or leaving means admitting that you've wasted your money.
3
I0I0I0IDec 11, 2009
+2
Bullshit man. I tried for a decade to get rid of AT&T but it just doesn't die.
2
[deleted]Dec 10, 2009
+2
Actually, that is 30 tens-of-seconds
2
ironiridisDec 10, 2009
+5
5 minutes is 30 tens of seconds.
5
xkillxDec 16, 2009
+3
is listnook still busted? because its giving me all sorts of c*** today.
3
KeyserSosaDec 16, 2009
-7
short version: yes.
Longer version: the db slave that was acting up earlier in the week died again, so we are rebuilding it from scratch, which is making parts of the site a little laggy.
We pay penance in sleep deprivation.
-7
xkillxDec 16, 2009
+3
thanks for the info! i'll try to be more patient.
3
pants6000Dec 10, 2009
+192
Happy Thanksgiving, everyone!
192
ketralnisDec 10, 2009
+44
If our bugs can cause temporal vortexes that allow you to comment on submissions before they are submitted, we should be able to get a nobel prize by writing more bugs
44
pants6000Dec 10, 2009
+35
You've probably earned one already--they're quite easy to get these days.
35
[deleted]Dec 10, 2009
+14
it was going to Listnook, but they gave it to Obama instead
14
[deleted]Dec 10, 2009
+20
[deleted]
20
GetOuttaMyOfficeDec 11, 2009
+3
but even *more* than karmanaut?
3
FunnyMan3595Dec 11, 2009
+3
He *is* karmanaut. His abilities just seem outrageous because the secret service and his speechwriters are both feeding him content.
3
Fat_Dumb_AmericansDec 11, 2009
He's started fewer wars.
0
tedivmDec 11, 2009
+2
It was actually a complex plot by the Higgs-Boson particle to distract a physicist browsing listnook at exactly the right time to miss it.
2
creator11Dec 11, 2009
+2
Yes somehow listnook is rehashing Primer.
2
[deleted]Dec 10, 2009
-5
[deleted]
-5
KrazyA1phaDec 10, 2009
+2
Last Thanksgiving, my friend.
> that allow you to comment on submissions before they are submitted
2
rq60Dec 10, 2009
+3
umm, no.
3
Element_22Dec 10, 2009
+10
I am outraged by how long it's taking Listnook to get back up! I demand my money and time back!
10
eclipse007Dec 10, 2009
+9
Anyone up for a class action lawsuit?
9
[deleted]Dec 11, 2009
+9
10s of seconds to not at all. I was recently out-trolled because I could not reply, this deeply saddens me.
9
raldiDec 10, 2009
+8
For related reasons, [awards](http://listnook.com/awards) are going to be suspended for a while. We'll be sure to back-calculate the lost ones once things are hunky-dory again.
8
[deleted]Dec 10, 2009
+15
Good. And consider not bringing them back. They make listnook feel like a 3rd grade graduation ceremony.
15
[deleted]Dec 10, 2009
+2
[removed]
2
[deleted]Dec 10, 2009
+5
It's like a gold sticker on a turd. You being the turd.
5
[deleted]Dec 10, 2009
-3
[removed]
-3
[deleted]Dec 11, 2009
+1
Hey, I graduated from 3rd grade, you insensitive clod.
1
f9tlsDec 11, 2009
+2
It was just today that I took notice of the awards and looked forward to checking back tomorrow.
2
personsaddressDec 11, 2009
+2
Hey you should come out with a setup diagram on how youse guys got listnook machines set up.
I run 5 machines myself, a web server(with dns server, and mailserver), 2 database servers, a gameserver.
Whats listnook setup, and ran on?
2
KeyserSosaDec 11, 2009
-4
Have I got the [post for you](http://www.listnook.com/r/IAmA/comments/a2zte/i_run_listnooks_servers_and_do_a_bunch_of_other/)!
tldr: currently 1 webserver (running haproxy), 15 8-core app servers running listnook instances, 9 databases (4 master, 5 slave), 5 caches (mostly memcached), 2 for cron jobs and miscellaneous.
-4
PissinChickenDec 10, 2009
+19
[In case you weren't aware](http://imgur.com/hzqrw)
19
willis77Dec 10, 2009
+11
Listnook has a half-life of 2 seconds? Rough crowd.
Personally, I give it at least 8 seconds before I delete my account in a furious rampage.
Edit: Not half-life
11
buckXDec 10, 2009
+20
That's not a half-life. Half-lives are exponential decay, so after 2 half-lives, you have should be at 25% of what you started with, as opposed to the 0% of the linear model.
20
[deleted]Dec 11, 2009
+3
[deleted]
3
buckXDec 11, 2009
+2
Well...I suppose that's valid. It's not what I would pick as a default, but I don't really have a reason to say it isn't.
2
willis77Dec 10, 2009
+10
Edit: I am wrong (and sleepy). But mostly wrong.
10
buckXDec 10, 2009
+4
"may apply to any quantity which follows a set-rate decay." Linear is not set rate decay. What you have is not a half-life, it's merely a length over which the y component halves by coincidence. Start measuring such a "half-life" at any other time, and you'll get a different result.
4
willis77Dec 10, 2009
+5
You are right. I retract my hubris and go to the corner to sulk.
5
NotMarkusDec 10, 2009
+2
How does what you quoted and linked to go against what buckX is saying?
2
jaxspiderDec 10, 2009
+10
Because **RAGE QUIT** always gets the message across.
10
PissinChickenDec 10, 2009
+5
she's a fickle w****
5
buckXDec 10, 2009
That's not a half-life. Half-lives are exponential decay, so after 2 half-lives, you have should be at 25% of what you started with, as opposed to the 0% of the linear model.
0
auraslipDec 11, 2009
+2
Can some on tell me what represents a standard unit of interest?
2
IcanbeviolentacreztoDec 10, 2009
+1
Really? Cause I'm pretty sure you're actually busy censoring innocent Listnookors. Clearly, this website codes itself while you guys sit around cackling about how you can ban people for no reason and send them arsenic lollipops instead of bobbleheads.
1
KeyserSosaDec 10, 2009
WHO TOLD YOU ABOUT OPERATION ARSENIOPOP™!?!?!!
0
HelcionelloidaDec 11, 2009
+2
[This lady](http://imgur.com/YBWRv.jpg"Lick him! Lick him Now!")
2
SaladForkDec 10, 2009
+12
Heh, I thought it was just me! Good to know it isn't, ;)
12
hamflaskDec 11, 2009
+5
I thought it was just you too.
5
tomparkerDec 11, 2009
+2
Q: What's the difference between a car salesman and a software salesman?
A: A car salesman KNOWS when he is lying.
2
KeyserSosaDec 11, 2009
-4
Seeing as we don't sell software, I don't know what to make of your analogy.
-4
tomparkerDec 11, 2009
T'was only a TJ (Tangential Jokeportunity...) No harm intended.
0
[deleted]Dec 10, 2009
+1
i think this has been going on for 3 days now. my submissions won't go through and the same stories have been dominating and holding position for a while. I was worried the site was getting stale and people just weren't posting much any more.
1
KeyserSosaDec 10, 2009
-2
It's definitely been building. The post mortem seems to indicate the proverbial shit hit the proverbial fan at about 5PM pacific yesterday evening. Prior to that the database in question was acting...odd.
It's getting some TLC from jedberg now so hopefully that will do the trick. Stay tuned.
-2
ketralnisDec 10, 2009
+1
> It's getting some TLC from jedberg
We need some Barry White on CD *stat*
1
[deleted]Dec 10, 2009
Regarding "database in question was acting...odd."
My suspicion is the database was becoming sentient and possibly neurotic:
"Maybe I don't want to index fields or store any new entries! Maybe I want to sing show tunes or play call of duty!"
Skynet is now in charge of the internetz.
0
[deleted]Dec 10, 2009
+1
[deleted]
1
[deleted]Dec 10, 2009
-2
Yep. Rather a CEO of a small company. meh :)
-2
darthsebDec 11, 2009
+2
Yeah, I'll work to fix YOU!
2
KeyserSosaDec 11, 2009
-5
Yeah? Well maybe that was out plan ALL ALONG!
-5
darthsebDec 11, 2009
+1
Yeah, I'll plan YOUR FACE!
1
j-marDec 10, 2009
+2
It's probably a good thing it's the end of the semester then, eh?
If karmanaut weren't busy with school, you guys would be screwed.
Question since I'm here. I know how comment karma works, but I submitted a pic and I'm only talking about a few points, but my karma up by my envelope (which does suck for being here a year. I'm not very interesting) hasn't changed. Am I misunderstanding how that works? Thanks for your help.
1
mcdeviantDec 11, 2009
+1
I want a free b******.
lI had to type this severar times to get it right because of Mr Johnny Walker. I like himw. He helps me to spell... clowqlyEDIT xlowlyEDIT slowlu ediy f*** this shit
1
DesCo83Dec 10, 2009
+1
Is that why all my comments for the last year have suddenly disappeared? I go to my overview and it's my last 10 comments, and then comments from a year ago...nothing in between.
1
rabiddachshundDec 11, 2009
+1
Can you fix the general slowness of the main page also? Whatever script highlights the previously viewed submission is not very netbook/atom friendly.
1
[deleted]Dec 11, 2009
+2
Listnook has slaves?
2
resephDec 11, 2009
+1
Is this not fixed yet? I'm getting "an error occurred while posting (status: 504)" when trying to submit posts.
1
jaxspiderDec 10, 2009
+1
Thank you so much for mentioning this. I seriously was thinking about suicide after seeing a 404 screen when I clicked on my inbox/username.
1
[deleted]Dec 10, 2009
+1
Ah lovely. I was wondering why some of my comments kept disappearing.
1
ddrtDec 11, 2009
+1
That's cool, I think every website is doing something like that today.
I'm going to time this comment...
**EDIT:** 6.74 seconds
1
FigsDec 10, 2009
+1
...and how long for the EDIT? :)
1
[deleted]Dec 10, 2009
+1
Timed reply.
Aw, only 1.88.
1
daysiDec 11, 2009
+1
Today? The site has been horribly slow for weeks.
1
oliver_higgenbottomDec 11, 2009
+1
I hate when slaves get out of line, dbs or not.
1
NueDumazDec 11, 2009
+1
Thank you. I thought my comp was getting sick.
1
eric22vhsDec 11, 2009
+1
Okay, but what about the slowness of today?
1
jon_kDec 11, 2009
So you use a MySQL replication cluster?
How do writes work?
I assume you have
MASTER
- SLAVE1
- SLAVE2
- SLAVE3
Then you use some load balancer to vary the queries. However, when a new user signs up, or posts a comment, how does the code make the right SQL connection?
0
steponcharlieDec 11, 2009
+1
Listnook's Slow
Listnook's Slow
Listnook's Slow
1
reddebDec 11, 2009
+1
Ok. So it's not me.
1
stakkarDec 11, 2009
Someone clogged the interweb pipes.. Please call al gore and see if this snow storm cooled off the planet long enough for him to come fix the pipes.
0
[deleted]Dec 10, 2009
Listnook has been slow as f*** for me recently; http://www.listnook.com/r/listnook.com/comments/acxhb/listnook_is_taking_from_3_to_20_seconds_to_do_page/
0
[deleted]Dec 11, 2009
+1
BETTER BE.
1
AerikDec 11, 2009
-5
You probably shouldn't call it a slave database.
-5
KeyserSosaDec 11, 2009
-4
You think we hurt its feelings?
-4
Dr-NoDec 11, 2009
+1
How about 'Mr. Slave database'?
1
ghaitchedDec 11, 2009
+1
tech support!!
1
j4k20Dec 10, 2009
+1
Thank you
1
TchockyDec 10, 2009
I realised my comment was a load of pigswill just as I clicked Submit. Thank you, IT God of Problems!
0
hoyfkdDec 10, 2009
-3
Today? I hadn't noticed since I experience the delays every day.
Edit: I'm sorry, let me forget that I experience this just about every day so I can sing the praises of the super fast comment system?
-3
[deleted]Dec 11, 2009
-1
I too apologize for not devoting my full 15 hours of uptime to voting and commenting. I'm working on it.
-1
[deleted]Dec 10, 2009
-8
Maybe you should work on that little "bug" that is keeping us from commenting on some of the sponsored links...
-8
raldiDec 10, 2009
+6
Sponsored links lead to revenue. Revenue leads to more backup databases.
6
[deleted]Dec 10, 2009
-3
Sponsored links are great. Disallowing comments on some sponsored links is not excusable.
-3
jedbergDec 10, 2009
+6
What bug is that? The one that lets the advertiser turn off comments?
185 Comments