· 199 comments · Save ·
Announcements Mar 24, 2010 at 2:38 PM

Sorry about that :(

Posted by jedberg

listnook was down from 12:30 UTC to 14:24 UTC. Initial indications are that the load balancer went down. We don't use Amazon's load balancer, we use haproxy on our own instance. Normally such a thing would cause our web monitoring software to freak out and wake us all up, but unfortunately it looks like that *also* went down at some point (that isn't on EC2 at all). Again, apologies for the inconvenience. **EDIT:** Here's the post mortem. It looks like we were hit by a bug similar to [this Ubuntu bug](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/276476). As you can see, the bug was fixed in a later version of the kernel. However, the way EC2 works, we can't upgrade the kernel ourselves. What I can do is switch to a new base image, but I hadn't, mostly because there were always higher priority things to do, and it was never really a problem before. As for the monitoring, it looks like our monitoring server experienced a power failure sometime yesterday, and the UPS failed, and then it failed to reboot cleanly. So what are we doing to fix the problem and prevent this type of failure in the future? First, I'm going to add a second monitor on a Slicehost machine that has been generously donated by a trusted listnook user, and then I'm going to have the two monitors watch each other. After that, I'm going build a new instance for our load balancer that is on the most recent version of Ubuntu, and does not have that nasty kernel bug.
Bug #276476 “Idle-priority scheduling bug blocks tasks” : Bugs : linux package : Ubuntu
Launchpad
Bug #276476 “Idle-priority scheduling bug blocks tasks” : Bugs : linux package : Ubuntu
"INFO: task blocked for more than 120 seconds" is a diagnostic message for when a process that wants to run is blocked from running for a long time. This bug report refers to a specific problem with idle-priority scheduling that has now been fixed. If you are up to date and are still getting the "task blocked" message, it is a different bug, so please file a new bug report.
🚩 Report this post

199 Comments

Sign in to comment — or just click the box below.
🔒 Your email is never shown publicly.
[deleted] Mar 24, 2010 +1619
[deleted]
1619
jedberg Mar 24, 2010 +524
I know, I'm a horrible person for having a need to sleep! I've been told that before, actually.
524
dakboy Mar 24, 2010 +592
You should spawn a child process, you'll be up every 2 hours anyway and can check the site yourself.
592
jedberg Mar 24, 2010 +298
Maybe next year. :) They do say that having a website is like having a child. You are always worrying about it and it keeps waking you in the middle of the night.
298
krispykrackers Mar 24, 2010 +24
Just wait until the right wing media finds out that this kid has four fathers...
24
jedberg Mar 24, 2010 +33
And no mother!
33
krispykrackers Mar 24, 2010 +17
If you guys get a divorce, I hope you win me in the custody battle. Daddy.
17
jedberg Mar 24, 2010 +48
Well, at least you are already potty trained. The website on the other hand keeps shitting itself.
48
krispykrackers Mar 24, 2010 +39
And you still love him more!!! I got all A's on my last report card! And none of you cared!! It's always "listnook, are you hungry? Are you thirsty? Did you make poopy for daddy? Does listnook want a num num?" I EXIST YOU KNOW. You can blame yourself for my future rejection of you and consequent drug addiction. *slams door*
39
jedberg Mar 24, 2010 +36
Is this what having a teenage daughter is like?
36
[deleted] Mar 24, 2010 +3
[deleted]
3
kurtu5 Mar 24, 2010 +2
You should try running a campus mail system. Sigh.
2
xlamplighter Mar 24, 2010 +303
Then it grows up, and wants an allowance, and to borrow the car, and it swears it'll be back home at Midnight, but you know it never comes back on time. And then it graduates, and goes to college. You were hoping it would make you proud, and become a doctor of neurosurgery, but NOOO, instead, it goes to some hippie school in Granolaville and studies Recreation Management.
303
[deleted] Mar 24, 2010 +18
Or it drops out of High School, goes to work with Jephron on the corner as a hustler, dodging strays and smoking Ls. Ala MySpace.
18
JephronOnTheCorner Mar 24, 2010 +13
Mothafucka, I ain't hirin' right now. Lookit you anyways...all pale like you spend all day behind a keyboard and shit. Gitchass off my corner 'fore I bust a cap, punkass.
13
modusponens66 Mar 24, 2010 +32
*And the cat's in the cradle with the silver spoon...*
32
Harry_Tuttle Mar 24, 2010 +8
[70s-narrative-rock upvote for bringing Harry Chapin to listnook.](http://www.harrychapinfoundation.org/)
8
gysterz Mar 24, 2010 +120
Haha that is my brother!
120
thatguydr Mar 24, 2010 +102
Your brother is a car?
102
dmd Mar 24, 2010 +6
Hey now. I went to [some hippie school in Granolaville](http://simons-rock.edu/) *and* became a doctor of neurosurgery. Well, a [doctor of neuroscience](http://www.cfn.upenn.edu/aguirre/wiki/lab_people#daniel_drucker_phd), close enough.
6
penguinsarecooool Mar 24, 2010 +15
except a website is always going to need you, and a baby grows up... until you need your baby to take care of you. Will a website take care of you when you're old and feeble? That's what I thought
15
gimeit Mar 24, 2010 +12
Except a website also provides you with income, whereas a baby is a constant money sinkhole.
12
Nessie Mar 24, 2010 +28
They got two kidneys, don't they?
28
[deleted] Mar 24, 2010 +13
I'd like to discuss your business plan in further detail as I also seem to have lots of superfluous organs.
13
elHuron Mar 24, 2010 +31
It can if you make money off of it.
31
temp9876 Mar 24, 2010 +8
Yeah but when babies get errors you get puke and shit all over you.
8
irenedakota Mar 24, 2010 +11
How is that different to a website?
11
temp9876 Mar 24, 2010 +10
uses more paper towel and sometimes it gets in your mouth.
10
PissinChicken Mar 24, 2010 +47
Give me your pager number, I'll backup the network monitors M-F 8AM-5PM EST.
47
jedberg Mar 24, 2010 +37
We got that time covered already, thanks. :)
37
PissinChicken Mar 24, 2010 +37
NP, honestly I was just hoping to get to call and wake you up. I know it's evil, but if I'm up, I enjoy waking others up.
37
[deleted] Mar 24, 2010 +12
I'll cover 2-5AM Saturday CST then.
12
simianfarmer Mar 24, 2010 +11
Only for nine hours a day? Amateur.
11
PissinChicken Mar 24, 2010 +15
I know. I was working 12 hour shifts but now I'm working the easy 9s. In my defense after work twice a week I go to class for 4 hours. I'll try and man up though.
15
Rubin0 Mar 24, 2010 +14
Do you have a beeper that wakes you up if the site crashes?
14
jedberg Mar 24, 2010 +32
Usually. But that also broke today.
32
pavs Mar 24, 2010 +33
Leave your number with us, so that we can call you next time the site goes down. 1000s of phone calls is the surest way to wake you up.
33
jedberg Mar 24, 2010 +34
415-klm-rdit
34
lazylion_ca Mar 24, 2010 +5
867-5309 ?
5
jedberg Mar 24, 2010 +15
That's the **secret** number. It goes to our secretary Jenny's desk.
15
ifatree Mar 24, 2010 +4
oh no. you just got me thinking of putting in easter eggs into our new avaya setup.. haha. and it's so close to april fools! just gotta keep telling myself: "with great power comes great responsibility"... "with great power comes great responsibility" ...
4
Rubin0 Mar 24, 2010 +2
If you guys had one more ad space on the right hand side, would you be able to afford a new one?
2
jedberg Mar 24, 2010 +9
Probably. But y'all would probably complain about it. ;)
9
kronholm Mar 24, 2010 +3
In my previous job with surveillance we had lots of systems (not computers per-se) that had a sort of dead-man switch mechanism (not the proper term, I know), so if all else failed, that one would sound an alarm. A system would poll various systems every x seconds and if there was no response, alarms were raised. Obviously the system was entirely independant of the other systems, so if the systems failed, this wouldn't fail.
3
[deleted] Mar 24, 2010 +6
At one of my companies, we simply had multiple monitoring boxes that in addition to monitoring all the servers, monitored each other. Management discovered that the on-call technician was *much* quicker to respond when 4 servers were texting him that a monitoring server was down instead of just 1, lol
6
mr_fusion Mar 24, 2010 +5
> Normally such a thing would cause our web monitoring software to freak out and wake us all up, but unfortunately it looks like that also went down at some point
5
countingspoons Mar 24, 2010 +7
it has to be asked...who told you that before?
7
Virtblue Mar 24, 2010 +4
You should just give up on your human body and go cyborg. The [Cymeks](http://dune.wikia.com/wiki/Cymek) did control the galaxy for quite a while.
4
kurtu5 Mar 24, 2010 +1
Problem, meet solution: http://en.wikipedia.org/wiki/Polyphasic_sleep
1
jedberg Mar 24, 2010 +5
Funny you mention that. There was an article about that on listnook years ago, when listnook was very new. The founders liked the idea, so Steve and Alexis actually tried it. I think they lasted 2 days.
5
HideAndSeek Mar 24, 2010 +169
No prob, you allowed me to reaffirm my opinion that digg sucks.
169
jedberg Mar 24, 2010 +140
That was the actual purpose of the downtime. :) Nah, I kid. I kid.
140
werfnort Mar 24, 2010 +40
So... Digg didn't do it? http://i.imgur.com/TlrNg.jpg
40
[deleted] Mar 24, 2010 +21
with the 1337 digg too. Nice pic.
21
[deleted] Mar 24, 2010 +30
oh jedberg, you stinker.
30
countingspoons Mar 24, 2010 +11
LOL same here! Because you know, turning off the computer and getting some work done...well thats just crazy talk.
11
[deleted] Mar 24, 2010 +9
[deleted]
9
[deleted] Mar 24, 2010 +39
[deleted]
39
mahlzeit Mar 24, 2010 +25
> to ensure the monitoring servers never gonna let you down. FTFY
25
DontNeglectTheBalls Mar 24, 2010 +11
> to ensure the monitoring servers never gonna run around and desert you. FTFY
11
jedberg Mar 24, 2010 +24
Yeah, but that costs money. Usually the site and the monitoring don't fail together. :)
24
romcabrera Mar 24, 2010 +1
Weren't both falls related? If not, why are you sure?
1
jedberg Mar 24, 2010 +9
No, one is the monitoring machine, which sits in our office, and the other was the load balancer, which is in the datacenter.
9
jon_k Mar 24, 2010 +1
So how does traffic flow here? Me>DATACENTER>LOAD BALANCER>OUT>AMAZON>return path? You've got a datacenter/colo just for load balancing? Why not put the LB software on its own amazon instance?
1
jedberg Mar 24, 2010 +4
No no. You send your data to Amazon, who sends it to our software load balancer on its own instance, which passes it to the appropriate webserver, and then it comes back the same way.
4
Pandalicious Mar 24, 2010 +11
Which is *exactly* what they want you to think.
11
kbedell Mar 24, 2010 +8
But who will monitor the monitors monitoring the monitors?
8
deusnefum Mar 24, 2010 +13
F*** you jedberg. Do you realize how much work I got done today because of your inattentiveness?
13
jedberg Mar 24, 2010 +17
Sorry. How about you spend a few hours on listnook to balance things out? I'll give you some free hours on your account.
17
deusnefum Mar 25, 2010 +1
Fine. But I'll accept no less than 6 free hours for this transgression.
1
[deleted] Mar 24, 2010 +25
Interesting that all Listnook problems henceforth must include a strong defense of Amazon's services. I didn't realize people hated Amazon that much. Personally, I use S3 and love it.
25
akatherder Mar 24, 2010 +8
I think people like Amazon, but there was a spate of speed issues and downtimes after listnook moved to Amazon (EC3 I think it's called?). I am content to attribute it to the growing user-base rather than the technology, but maybe I'm just a sucker.
8
jedberg Mar 24, 2010 +23
> but there was a spate of speed issues and downtimes after listnook moved to Amazon See, that's why I put in the defense of Amazon. That just simply isn't true. The site got faster when we moved to Amazon. The site happened to get slow after we *told* people we moved to Amazon, 6 months after the fact.
23
akatherder Mar 24, 2010 +6
I'm just saying "after", not "immediately after" or "due to". The point is that it happened while listnook was using Amazon and it didn't happen (much) prior to Amazon. There _is_ no correlation between Amazon and any issues. But the fact that they occurred while using Amazon's services could very easily cause some people to think that it's Amazon's fault.
6
jedberg Mar 24, 2010 +26
I only put them in to head off the inevitable "It was Amazon's fault!" because they run a good service and I hate to see their name besmirched like that.
26
[deleted] Mar 24, 2010 +12
[deleted]
12
jedberg Mar 24, 2010 +30
Software + hardware. The hardware is my iphone with an obnoxious ringer and a modem on a machine that checks listnook and calls us when it is down.
30
leemy Mar 24, 2010 +20
Does it keep closing listnook only to open it again, and again? Because now I'm no better than a modem.
20
jedberg Mar 24, 2010 +12
Yeah, pretty much. But that is they Python software, not the modem. So you are no better than a small Python program that I wrote.
12
cmcs Mar 24, 2010 +3
Does it sigh in annoyance and cluck its tongue? Does it wait that impatient second before erupting at you? Does it hang up in exasperation? Does it tell you that this better really be the very last time?
3
jedberg Mar 24, 2010 +9
Are you talking about your spouse or our website monitoring?
9
dfranke Mar 24, 2010 +1
A modem? Seriously? Can't you use Twilio or something?
1
jedberg Mar 24, 2010 +6
The modem itself is very reliable. The machine/dsl it is connected to appears not to be. I'll find out later when I get to where the monitoring machine is.
6
[deleted] Mar 24, 2010 +2
You should just give out your phone number and tell everyone to call you if listnook's down. Totally foolproof.
2
kobescoresagain Mar 24, 2010 +7
Most likely, I have that for my own site. It calls my cell phone.
7
[deleted] Mar 24, 2010 +18
[reads] *mumble mumble* down *mumble mumble mumble* Amazon EC2 *mumble* Yep, I knew it - Amazon's to blame. Hmph.
18
jedberg Mar 24, 2010 +14
You don't read good.
14
universl Mar 24, 2010 +20
I expect a full refund for my lost time this morning.
20
jedberg Mar 24, 2010 +30
Check your inbox.
30
BoonTobias Mar 24, 2010 +6
Guess who was on digg that whole time? That's right!
6
jedberg Mar 24, 2010 +15
And yet you came back. That is the best flattery, is it not?
15
BoonTobias Mar 24, 2010
Don't flatter yourself, beardboy, i will be keeping my eyes on you
0
jedberg Mar 24, 2010 +10
I shaved my beard 2 years ago. You must not be watching very closely.
10
[deleted] Mar 25, 2010 +1
[deleted]
1
spooniam Mar 24, 2010 +5
Just in the nick of time too. I was all "if this b**** is down for 5 more minutes, I swear to god I'm gonna rip off my testicle and feed it to my dog!" Then BAM! All is good and I still have testicle.
5
[deleted] Mar 24, 2010 +11
Most productive two hours all month! edit: My company owes you..
11
[deleted] Mar 24, 2010 +46
[deleted]
46
aamirica Mar 24, 2010 +9
Your technical jibber-jabber just makes me trust you more.
9
iamanogoodliar Mar 24, 2010 +10
I kept trying and trying to access the site and got so depressed when I couldn't that I committed suicide. Hope you're happy.
10
wickedcold Mar 24, 2010 +3
Am I the only one who has the message in the yellow box staying at the top of every screen even though I'm logged in? I'm referring to this one: >*listnook is a source for what's new and popular online. vote on links that you like or dislike and help decide what's popular, or submit your own!*
3
[deleted] Mar 24, 2010 +3
you're always so quick to ensure people think good of amazon... whats up with that?
3
g0zer Mar 24, 2010 +2
hey let us know what happened in detail. love your tech updates.
2
jedberg Mar 24, 2010 +2
I just edited the self text with the post mortem. Here is what is said: It looks like we were hit by a bug similar to this Ubuntu bug. As you can see, the bug was fixed in a later version of the kernel. However, the way EC2 works, we can't upgrade the kernel ourselves. What I can do is switch to a new base image, but I hadn't, mostly because there were always higher priority things to do, and it was never really a problem before. As for the monitoring, it looks like our monitoring server experienced a power failure sometime yesterday, and the UPS failed, and then it failed to reboot cleanly. So what are we doing to fix the problem and prevent this type of failure in the future? First, I'm going to add a second monitor on a Slicehost machine that has been generously donated by a trusted listnook user, and then I'm going to have the two monitors watch each other. After that, I'm going build a new instance for our load balancer that is on the most recent version of Ubuntu, and does not have that nasty kernel bug.
2
[deleted] Mar 24, 2010 +4
despite being logged in (which took forever) I am still seeing the not-logged-in yellow info bar along the top...
4
bjarkith Mar 24, 2010 +2
How could it be down until 15:24 UTC? It's 14:55 UTC when I type this.. What am I missing?
2
drpants Mar 24, 2010 +5
Have the admins thought about checking the oil filter? Sometimes, when it is clogged, the load balancers don't get enough air.
5
martinmarkovski Mar 25, 2010 +2
Please try not to sleep while there are users here :)
2
[deleted] Mar 24, 2010 +2
Does it seem pointless to anyone else to post a "is listnook down?" thread when listnook is actually down?
2
gtlogic Mar 24, 2010 +2
I pay wayyy too much to have listnook go down. This is ridiculous.
2
Shaper_pmp Mar 24, 2010 +139
In other news, [islistnookdown.com](http://www.islistnookdown.com) is completely f****** useless. I was checking it religiously for the first hour or so to make sure it wasn't just me (or our work firewall blocking access), and the whole time islistnookdown.com was sat there cheerfully lying to me telling me listnook was up and running. Much confusion was provoked.
139
countingspoons Mar 24, 2010 +42
a better resource....the listnook fan page on facebook. People start posting there in desperation whenever it truly goes down!
42
grt Mar 24, 2010 +69
An even better resource: http://twitter.com/#search?q=listnook%20down Nobody does complaining better than Twitter users.
69
albatroxx Mar 24, 2010 +82
[An even better resource](http://www.listnook.com)
82
libbrichus Mar 24, 2010 +32
I clicked what you linked there.
32
[deleted] Mar 24, 2010 +10
You should submit this story to Digg
10
[deleted] Mar 24, 2010 +10
Alternatively, if you don't use facebook, go to twitter and search for 'listnook'. If it's down you'll see a big 'ol list of people tweeting 'OMG Listnook is down, I'll have to actually do work today, lol'.
10
eetmorturkee Mar 24, 2010 +18
Made me want to stab my monitor when that "no" popped up. LIES
18
Shaper_pmp Mar 24, 2010 +38
Seriously - you set up a domain and website with *one purpose in life*, and it fails miserably when asked to perform it. Unbelievable.
38
[deleted] Mar 24, 2010 +10
That's what jedberg was saying this morning about the server monitoring... server. Edit: I like "*" next to my name.
10
[deleted] Mar 24, 2010 +8
I just searched Twitter for "listnook down" and many twitter whiners confirmed my worst suspicions.
8
lusophiliac Mar 24, 2010 +1
okay, now who's going to pull out the adblock card? I know you're out there...
1
threeminus Mar 24, 2010 +1
Here's a datapoint for your troubleshooting: during the last hour or so of the downtime I was trying to use the listnook iPod app, but it wouldn't load my profile or any pages. However, browsing the site through safari on my iPod worked no problem (if maybe a bit slow). TL;DR: iPhone app failed harder than regular site. (Also, thanks for being awesome!)
1
[deleted] Mar 24, 2010 +33
Man you guys know nothing about running a company, the standard fixing an error process goes as follows: 1. After receiving initial complaint treat users like morons and have them do something menial like plug and unplug their computer repeatedly then say it's someone else's problem, contact them. 2. After enough complaints are received spend weeks denying flagrant error. 3. Blame a third party component "the whole problem was that the server narwhal interface was slightly off because X Tech in China used mutant porpoises with antlers instead of actual narwhals, it was a minor mistake we are sorry" 4. When it is revealed that third party component was not the cause of error, promise everyone free candy to distract them from error. 5. Release buggy upgrade and tell people that they need to upgrade to Listnook Prime. 6. Repeat steps one through five 7. Hope a new much more devastating error comes along and overshadows previous error. 8. Blame the internet, then secretly fix error. You've skipped all those steps and went 1. find problem ----> 2. fix it You'll never compete with apple, adobe, or microsoft at this rate.
33
heega1 Mar 24, 2010 +2
I went to islistnookdown.com, and it said no... I went to listnook.com, and it was down... I went to islistnookdown.com, and it said no... I went to listnook.com, and it was down... ...
2
inserthandle Mar 25, 2010 +1
You should put some monitoring software on your home computer. If all the listnook admins did this there would be practically no chance of downtime going unnoticed plus there would be no extra cost.
1
[deleted] Mar 24, 2010 +1
Why the hell does this have down votes? O.o
1
[deleted] Mar 24, 2010 +1
I love finding out what causes the downtime, not sure why - but it always amazes me how many teeny little discrete parts go together to make this place work. Thanks for the update, now, go catch up on sleep.
1
c_megalodon Mar 25, 2010 +2
listnook admins should stop apologizing whenever these things happen, it's ok, it's not like were going to hunt you guys down, kidnap and then r*** you. :>
2
nmrk Mar 24, 2010 +1
>What I can do is switch to a new base image, but I hadn't, mostly because there were always higher priority things to do, and it was never really a problem before. Bad Programmer's Rules: "There is never enough time to do things right the first time, but there is always enough time to fix it after it fails."
1
[deleted] Mar 24, 2010 +1
[deleted]
1
VWSpeedRacer Mar 24, 2010 +1
If it makes you feel any better, we had A/C failure in our main server room today (15 HP DL360s, 2 DL580s, and a C-class blade chassis). It made for an exciting day.
1
Failcake Mar 24, 2010 +1
I want a refund for the time I couldn't use listnook. Or at least add it as credit towards my next monthly Listnook Gold fee.
1
[deleted] Mar 24, 2010 +1
I was wondering, are you all really bad at your jobs, or is it just one guy ruining things for the rest of you?
1
AmadeusZull Mar 24, 2010 +1
We need a copy of the post mortem.
1
rankun Mar 24, 2010
I heard this is the thread where you get free karma for making jedberg feel bad about himself.... Here goes: You (insert admin) suck for (random gripe about listnook)
0
dalore Mar 24, 2010 +2
Who watches the watchmen? (Or in this case who monitors the monitoring servers)
2
JiminiousBastiardius Mar 24, 2010 +84
"It's easier to go through a time with no money and listnook, than it is to go through a time with money and no listnook." --Winston ChurchPill
84
krazykipa- Mar 24, 2010 +62
"Shit breaks sometimes, and you just gotta fix it" -Edgar Allan Poe
62
yip_yip_yip_uh_huh Mar 24, 2010 +103
"Aw hell no, listnook's down like a muthafucka. Sheeeeeeeit." Leviticus 22:13
103
Pandalicious Mar 24, 2010 +11
The user wrests the whip from listnook and whips itself in order to become listnook, not knowing that this is only a fantasy produced by a new knot in listnook’s whiplash. -Franz Kafka
11
[deleted] Mar 24, 2010 +18
thought observation chief paint close offend makeshift disgusted squeeze mindless *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
18
[deleted] Mar 24, 2010 +10
You gots to diversify your websites, nigga. - Pliny the Elder
10
Fenris_uy Mar 24, 2010 +1
I want to see more graphs and layout diagrams, not only a 3 paragraphs self post
1
randomb0y Mar 24, 2010 +1
So how much is Amazon paying you to keep stressing that it's never their fault? :)
1
TrueDuality Mar 25, 2010 +1
Switch from Ubuntu. Otherwise good job
1
libbrichus Mar 24, 2010 +62
Listnook can never break. It can only become stairs.
62
Reference_Explainer Mar 24, 2010 +74
The comment is an alteration of the famous Mitch Hedberg joke: ***An escalator can never break: it can only become stairs. You should never see an Escalator Temporarily Out Of Order sign, just Escalator Temporarily Stairs. Sorry for the convenience.*** The alteration does not make much sense in Listnook's context, so it also pays homage to Mitch Hedberg's non-sequitur side.
74
guywithabike Mar 24, 2010 +15
I love you. Father my children. I mean, uh, nice novelty account.
15
stonedslacker Mar 24, 2010 +58
I used to love Listnook. I still do, but I used to too.
58
thatguydr Mar 24, 2010 +24
I'm against posting on listnook, but I don't know how to show it.
24
BennyG02 Mar 24, 2010 +16
At the beginning of a subject line in an email I like to put 'RE: This is what the word listnook would look like if the letters d, i, and t didn't exist'
16
RodneyPeete Mar 24, 2010 +9
l wrote an email to my dad, I was going to write "l really enjoyed using listnook", but l accidentally 'rarely' instead of 'really'. But I wanted to use it, and my backspace button was broken, so I wrote "l rarely visit Digg, Dad. There's a lot of shit you don't know about me. Quit trying to act like l'm some kind of Digger." l know this email took a harsh turn right away.
9
allhands Mar 24, 2010 +111
No problem. Shit happens. You guys work hard and it shows. It's not the end of the world if some minor downage happens.
111
[deleted] Mar 24, 2010 +1
Honestly, when I used to do digg, digg went down more often than listnook.... by a long shot. During those downtimes I'd come to listnook until eventually I could not handle digg's style and failures anymore and realized how much better listnook is. The clean and simple interface does it justice. :)
1
junkit33 Mar 24, 2010
I'll be the first to admit that nobody should complain about a free site, but Listnook *has* experienced an awful lot of slowness and/or complete downtime in the last couple of months. It does make me start to wonder how well the site has been architected, and if these problems are going to start occurring with more frequency as the site continues to grow. Like, for example - why would you *not* have a redundant load balancer setup for a site that gets this much monthly traffic?
0
patmools Mar 24, 2010 +12
I would argue that since so many people's worlds revolve around Listnook, it *is* the end of the world.
12
johninbigd Mar 24, 2010 +6
Speak for yourself! I'm stuck at home with a ton of freaking snow outside. I need listnook! I'm glad it's up and running smoothly again.
6
Takteek Mar 24, 2010 +1
Does this have anything to do with the "new user" box that decided to come back for me? > listnook is a source for what's new and popular online. vote on links that you like or dislike and help decide what's popular, or submit your own! I kept getting signed out and now I have that stuck at the top of all my pages.
1
dalore Mar 24, 2010 +1
So is the plan to transition to Amazon's load balancer and move the monitoring software to ec2 (or outsource it)?
1
[deleted] Mar 24, 2010 +51
[deleted]
51
TundraWolf_ Mar 24, 2010 +16
wow.... me too. I actually got a staggering amount of work done. Maybe I can toss jeddy a few $$ to bring down listnook when i'm on crunchtime
16
[deleted] Mar 24, 2010 +36
At first I was afraid, I was petrified Kept thinking I could never live without that listnook site But then I spent so many nights thinking how jedberg was a jerk and I bebrudgingly Got my ass back to work And so you're back from the land of nod I just walked in to find you here with that announcement on your blog I should have put you in the hosts file I should have blocked you anyway If I'd known for just one second you'd come back to derail my day -_that's quite enough of that. Ed_
36
JesusWuta40oz Mar 24, 2010 +36
*Turns on TV to the stock market watch channel* "Today the market reported an upswing in productivity in all sectors...analysts are unsure why.." *We know why*
36
schaffner Mar 24, 2010 +105
You have almost made me do some actual work.
105
ears Mar 24, 2010 +77
I..I..._alright, here goes_..I admit I kinda browsed Digg for a few minutes, got very confused..need a shower..
77
Kream1 Mar 24, 2010 +73
That's like boning your fat, annoying ex when your super hot girlfriend went overseas on vacation for a while.
73
quintios Mar 24, 2010 +10
I would love to see how much the traffic increased at Digg during the downtime. Would be interesting.
10
Bobinator2000 Mar 24, 2010 +11
I bet it felt like a time warp. No wonder you were confused.
11
[deleted] Mar 24, 2010 +9
I have that same complaint, it was terrifying and I would not recommend it to friends.
9
pillowplumper Mar 24, 2010 +7
I woke up, bleary-eyed, opened listnook, rubbed my eyes, refresh, refresh, *refresh*, ***refresh REFRESH*** , checked the ethernet cord, switched to chrome, SWITCHED TO IE, of all things, before deciding it was a sign from God that I should finally get around to writing my now-2-days-overdue paper. Guess not, though. Whew.
7
chmod777 Mar 24, 2010 +5
serious. i got sooo much done this morning. now i'm going to have to double slack this afternoon.
5
[deleted] Mar 25, 2010 +1
[removed]
1
AnubisX Mar 24, 2010 +1
My cat's breath smells like cat food.
1
lazylion_ca Mar 24, 2010 +1
Is there something some of us volunteers could run on our own servers that would ping the listnook servers occasionally and send an alert your way if the ping fails too many times?
1
Kryian Mar 24, 2010 +1
I missed school because of this! Normally I would browse listnook until it's time for me to get up and leave...but instead I was force, FORCED I SAY, to go to sleep and miss everything. My life may very well be ruined.
1
rajones85 Mar 24, 2010 +1
This is why it pays to use a hosted website monitoring service. 1-their business depends on keeping the monitoring service up, and 2-you have someone *else* to fire if monitoring goes down.
1
lickityskip Mar 24, 2010 -4
Why is this shit always on the front page? Who gives a flying f***
-4
wenowhavepower Mar 25, 2010 +1
Ubuntu as a server? Just stick with debian! or switch to FreeBSD!
1
[deleted] Mar 24, 2010 +9
Monitor the monitoring servers. Pingdom? **BTW, you guys rock. I don't really care.**
9
atrais Mar 24, 2010 +1
My browser said something about "no contact with proxy" when I went to listnook.com. I spent 20 minutes finding out if I had a hidden proxy somewhere. :)
1
[deleted] Mar 24, 2010 +1
you need QA. Someone should be testing the monitoring system, the backup power systems, the backup data systems, etc at least once a month.
1
[deleted] Mar 24, 2010
[deleted]
0
the_venerable_telix Mar 24, 2010 +1
Hint:Use pingdom or somthing as a backup for your monitoring
1
[deleted] Mar 24, 2010 +1
ha(proxy) ha(proxy)
1
eoin2000 Mar 24, 2010 +22
Nearly ended up on Digg. Please don't ever put me in that situation again.
22
[deleted] Mar 24, 2010 +10
I did end up on Digg and wrote a comment about them stealing articles from Listnook (that pregnant woman with the "loading" T-shirt). EDIT: I upset the Digg hivemind with my comment. They're so easy to troll.
10
YourFaceSucks Mar 24, 2010 +24
I'm guessing Digg will be a little light on content this morning.
24
jpjandrade Mar 24, 2010 +41
You mean digg will be a little light in content 3 days from now.
41
[deleted] Mar 24, 2010 +1
[deleted]
1
[deleted] Mar 24, 2010 +1
Why the f*** are you guys so concerned with defending Amazon?
1
malammik Mar 24, 2010 +1
why are you not using elastic load balancers?
1
baraqiyal Mar 24, 2010 +1
I don't understand. The HAProxy web site says that it is stable and reliable.
1
[deleted] Mar 24, 2010 +1
[deleted]
1
[deleted] Mar 24, 2010 +1
[deleted]
1
lazylion_ca Mar 24, 2010 +1
Asleep at noon? You must be a listnookor.
1
ibrokereddit Mar 24, 2010 +10
Ah yes, thank you! You don't even wanna know how many angry e-mails I got this time.
10
NueDumaz Mar 25, 2010 +1
If you give me your phone number I can just call you next time there is a problem.
1
jotate Mar 24, 2010 +11
I love how you tossed in "It isn't Amazon's fault" multiple times in anticipation of the "OMFGAMAZONSUX" blow back. You should probably clarify that it's also not the Search function's fault.
11
TundraWolf_ Mar 24, 2010 +7
I blame the search function for random mishaps throughout the day. "DAMN YOU SEARCH FUNCTION!!!!", yelled tundrawolf_ as he threw a chair across the meeting room and dashed out into the hallway.
7
← Back to Board