ListNook

One of listnook's greatest strengths is the huge collection of niche communities and categories of content that we have. One of our greatest weaknesses is that most of it never makes it to the front page. So many vast, undiscovered communities. I mean, just look at my own list of favourites: [programming](/r/programming), [technology](/r/technology), [comics](/r/comics), [math](/r/math), [Python](/r/Python), [coding](/r/coding), [linguistics](/r/linguistics), [haskell](/r/haskell), [robotics](/r/robotics), [answers](/r/answers), [electronics](/r/electronics), [StandUpComedy](/r/StandUpComedy), [ideasfortheadmins](/r/ideasfortheadmins), [ECE](/r/ECE), [emacs](/r/emacs), [listnookhax](/r/listnookhax), [Coffee](/r/Coffee), [sanfrancisco](/r/sanfrancisco), [erlang](/r/erlang), [bayarea](/r/bayarea), [chrome](/r/chrome), [listnookdev](/r/listnookdev), [systems](/r/systems), [artificial](/r/artificial), [compscipapers](/r/compscipapers), [algorithms](/r/algorithms), [macapps](/r/macapps), [horseporn](/r/random "good catch"), [arduino](/r/arduino), [operabrowser](/r/operabrowser), [SketchComedy](/r/SketchComedy), [golang](/r/golang), [kindle](/r/kindle), [smallprog](/r/smallprog), [robot](/r/robot), [Esperanto](/r/Esperanto), [avr](/r/avr), [hadoop](/r/hadoop), [cassandra](/r/cassandra), [colorblindness](/r/colorblindness), [android](/r/android), [england](/r/england), [BSD](/r/bsd) We have loads and loads of these communities, some very tiny, but they just aren't very discoverable. I think that helping people find this stuff is a problem worth solving, and so do plenty of researchers and grad students that have contacted us asking for this data (that we've historically had to turn away). There's lots of [research](http://en.wikipedia.org/wiki/Netflix_Prize) out there on this kind of problem that we'd like to participate in. There's our [JSON API](http://www.listnook.com/.json), but that's just not enough for the in-depth analysis that we'd like to do and allow researchers to do. We feel that opening up users' private data to researchers like that has to be done very carefully, and always with the permission of the users affected. So I'd like to announce that, from now on, we're going to share all your private data with DARPA. No, just kidding. Today we're adding a **new [preference](/prefs)** under "privacy options" called "allow my data to be used for research purposes". **By ticking that box you're agreeing to allow us to include certain data about you in big data dumps** like [this one](http://www.listnook.com/r/listnookdev/comments/bubhl/csv_dump_of_listnook_voting_data/). This is **optional and opt-in**. We want to make sure that everyone understands *exactly* what ticking that box will do. The data that you're giving us permission to reveal are: * Your [community subscriptions](/listnooks/mine) * Your list of [friends](/prefs/friends) **edit1** none of *their* data, just that you friended them **edit2** only friends that have *also* opted in would be listed * Non-content information about private listnooks that you post in (that is, we may share *that* you posted there, but not *what* you posted) * Your browser's user-agent * Information on spam reports that you've filed (the `report` button) On a separate tickbox, you can also share your voting history so that people can see your `liked` and `disliked` pages (this has been there since 2005). Either of these tickboxes will mean that you give us permission to share this voting data. Some items we're considering but want to talk to you about are: * The last time you visited listnook at the time of the data-dump (in general this can be approximated from your last vote) * The first two octets of your IP address (that is, if you're at 1.2.3.4, we may reveal that you're at 1.2.x.x) * A [one-way hash](http://en.wikipedia.org/wiki/Cryptographic_hash_function) of your email address **edit** looks like this one's out, lots of people seem uncomfortable with it **Please tell us if you think that any of these are going too far**, especially if you'd tick the box but for one or two of the data involved. If we ever change or add to this list, we'll reset everyone back to the default of `off` (and/or implement a more granular set of research-related preferences), so you don't have to worry about us sneaking things in there while you're asleep. **You're not agreeing to let us start telling everyone about every link you click or anything like that without your knowledge**. You are not agreeing to let us share the actual content of your private listnooks, and **if you do not tick the preference we will not share this data against your will**. This is for research dumps. We're not going to be fielding requests for data about individual users. We're not trying to share identifiable information and in the general case we'll try to keep you anonymous but we all know that [that doesn't always work](http://en.wikipedia.org/wiki/AOL_search_data_scandal) which is why this is **optional and opt-in**. Did I mention that this is **optional and opt-in**? Our goal isn't just to get a bunch of data out there, but to use this data to *make listnook better*. We want features like hyper-local communities and recommendations. And we want you guys to [help us](/r/listnookdev) shape those features, but to do so and attract interested researchers we need lots and lots of data for analysis. Also, if you don't tick the box, I'll kill [a kitten](http://www.flickr.com/photos/crumley/160490011/)

200 Comments

[deleted] Sep 15, 2010 +91

I would prefer to not share my list of friends. I feel that they should only be included in my list if they opt in as well. Otherwise, I would be totally happy to participate. I love data!

ketralnis Sep 15, 2010 +87

> I feel that they should only be included in my list if they opt in as well That's a really good point, I'll have to think about how that could work

burnblue Sep 15, 2010 +16

Not sure why anyone needs to know who the friends are at all. It's not like we use Digg's social model

[deleted] Sep 15, 2010 +44

Half my 'friends' are users I want to look out for, to avoid, argue against, , avoid being rickrolled, bel-aired or non-relvent tldr by.

smallfried Sep 15, 2010 +45

Listnook should have an 'enemies' list.

kleinbl00 Sep 15, 2010 +8

1) Download the [Listnook Enhancement Suite](http://listnook.honestbleeps.com/) 2) Adopt a system. Since RES gives you seventeen colors plus clear, you have leeway. I myself use clear for "notes to self" and the other 16 colors for "trolls of various magnitude" 3) Give yourself a note for each one - "wants enemies list" "doesn't understand irony" "needs to die in a fire" 4) Realize that after using it for over a month on a page with, say, 743 comments, only one name is tagged and that maybe, just maybe, it isn't worth it.

errerr Sep 15, 2010 +11

I vote for this. Make sure it is clear though, there is no 'ignore' list, just 'enemies'.

Nick4753 Sep 15, 2010 +3

The data isn't about judging you and who you friend - it is about finding out who the typical listnook user 'friends' and seeing if there is any link between why you would friend someone Too bad they don't have a staff of math grads to run stats on ALL the data and release it like OK Cupid does (where there is absolutely zero way for you to identify individual users in the data, only what they are statistically likely to act like) Plus that would give math grads actual math-related jobs :)

Wadsworth Sep 15, 2010 +14

Wait ... there are "friends" on listnook?

Glayden Sep 15, 2010 +8

[Yes.](http://www.listnook.com/prefs/friends/) - but, they don't get a message that you friended them or anything, it's relevant solely on your side... (At least this was the case before this whole opt-in list thing, now if you opt-in they could theoretically figure out who friends them)

TooSmugToFail Sep 15, 2010 +20

> they don't get a message that you friended them or anything It's like, they're your friends, but they don't know it. That's... That's sad man...

Zeulodin Sep 15, 2010 +14

High-school all over again. :(

iHelix150 Sep 15, 2010 +64

I'd be willing to participate, but only if it's truly anonymized. I don't mind showing up as a random number, but i'd prefer that my userID / email hash not be included. Take userid+email+salt (unique salt per data dump), hash that and you'll have a nice untraceable unique ID. Do that and I'm all in.

ketralnis Sep 15, 2010 +30

That's the idea but it's often possible to glean more from the semantic data itself, so you should assume that whatever method we use can be broken. We want it to be anonymous but we aren't perfect. This is why it's opt-in

tedivm Sep 15, 2010 +12

Even still, I would like it if people had to put a little bit of work into it. I like the idea of doing some randomization, especially if you're going to be including the friends list (which I also think should be a separate opt in- honestly it's the only reason I haven't checked the box yet).

[deleted] Sep 15, 2010 +326

[deleted]

326

ketralnis Sep 15, 2010 +166

Good idea, I should add a help wiki page for it

166

fazon Sep 15, 2010 +71

But who exactly is getting access to this info?

ketralnis Sep 15, 2010 +105

I'll release the dumps publicly

105

codygman Sep 15, 2010 +8

What will the dumps consist of?

ketralnis Sep 15, 2010 +19

Potentially all of the information I mention in the post. In practise, the dumps that my current version is generating consist in a CSV file of votes like user_hash,timestamp,direction,commmunity_id

codygman Sep 15, 2010 +27

Alright cool. As long as the user_hash is salted and peppered reasonably well I'll be checking that box for you!

snoobie Sep 15, 2010 +7

A salt would only help slightly, since if user_hash is derived directly from someones username it can be easily reversed if you have a list of all users.

mailor Sep 15, 2010 +5

I'd love to participate but I just don't feel like my privacy is safe here. My hash does not necessarily provide me with anonymity. Why a HASH of the user and not just an ID? Or is the hash very lovely salted?

wafflesburger Sep 15, 2010 +4

I'm confused. Everything you do here is done publicly, isn't it?

mailor Sep 15, 2010 +8

yes, it is. But since they actually release my data to the public, I have no more control on them. If I want to, I can delete anything I've written so far, or delete my account, or change something here on listnook. Once my data are out there, I can't control them anymore. I can be fine with that, but I'd prefer those data can not be linked to *this account*. It's not a huge technical issue to solve, and there would be an additional layer of anonymity between the user and the public.

superdug Sep 15, 2010 +3

Sure, but it's not aggregated into a easily digestible format. It's like red light cameras. Lots of people run red lights, but not all of them get ticketed for doing so. Anyone at an intersection can watch someone run a red light, but that person cannot easily see everyone who ran the red light for the last 12 months. (instantly)

kingnothing1 Sep 15, 2010 +6

Although you say this is for sub listnook discovery, how much of this will be geared to enhance properly placed advertising?

That's not the intention, but from a practical perspective I can't promise that nobody uses it that way since it's publicly available. To be quite honest I don't think any of our advertises have the ability to consume information like that. But I can tell you that that's not what I'm trying to accomplish.

superdug Sep 15, 2010 +17

... right now Forgive me, I have no doubts that you are pursuing this out of pure nerd joy that you'd get from consuming massive amounts of raw data. I don't think you want to "pull a fast one" on people here, but this really does stink like facebook when it comes to privacy concerns. I guess you just got screwed by coincidental timing. Digg is in a death spiral, thousands of users are coming to listnook, you're trying to make one of the biggest internet stunts in the world with Colbert and Comedy Central, and you just started taking subscription donations. I don't know how this data could be used for anything more than monetization of listnook. For instance, you could find out what stories that get over 1000 upvotes have for common words in a headline. I wouldn't have a problem if say, you did like okcupid with the stats on their blog, but opening it up into a one stop shop, just seems like a bad idea. Lastly, whats to stop people from taking the "scrubbed" data and using it to identify people through their listnook profile? I mean it's not hard to guess that USER_ID 98334 voted up a bunch of shit in /r/trees and then look and see which user hung out in /r/trees for the data set you're viewing. Before you know it, everyone finds out I smoke pot. The irony is not lost.

uep Sep 15, 2010 +16

I don't mind if it *is* for the monetization of listnook. It's opt-in. If this helps them keep the lights on, I don't have a problem.

averyv Sep 15, 2010 +2

> I don't know how this data could be used for anything more than monetization of listnook. For instance, you could find out what stories that get over 1000 upvotes have for common words in a headline. ...you could do that right now > I mean it's not hard to guess that USER_ID 98334 voted up a bunch of shit in /r/trees and then look and see which user hung out in /r/trees for the data set you're viewing. that is a pretty long walk. Also, our user names are right above all of our posts, so if you were to post anything in /r/trees, that would be just as good an indication that you were there. not that reading things from that listnook means anything at all, but still. > The irony is not lost. I am not seeing any irony anywhere here. as a final point, why is everyone so averse to having meaningful advertisement sent to them? I would much rather see ads that are relevant to my interests than ads that mean nothing to me. Right?

[deleted] Sep 15, 2010 +11

Are our usernames intact? I don't see why they can't be replaced with numbers. Usernames don't matter when it comes to stats. I'd tick the box if my username was changed to a number.

[deleted] Sep 15, 2010 +4

Can you also make this data accessible and real-time via your JSON/JSONP APIs?

ketralnis Sep 15, 2010 +22

That's not the intention, we're not trying to provide a stalking platform but rather develop offline processes for recommendations systems

tabber Sep 15, 2010 +3

I don't like this "publicly". you should release the information only to accredited academic institutions that are doing a research study monitored by an ethics board/committee and are overseen by a professional academic.

ketralnis Sep 15, 2010 +6

Since I don't have the resources to manage all that, you're probably better off not opting in if that's a requirement for you.

slf67 Sep 15, 2010 +8

How often will you take a dump?

ketralnis Sep 15, 2010 +26

Depends on my diet, I suppose

mean7gene Sep 15, 2010 +17

I couldn't quite tell if your're including full User Agent or not, but please don't, it's as good as an ID, EFF Paper on Tracking users by User Agent: http://isc.sans.edu/diary.html?storyid=8812

[deleted] Sep 15, 2010 +30

Holy c***. I just looked at mine: HTTP_CONNECTION:Keep-Alive HTTP_KEEP_ALIVE:115 HTTP_DWARF:YES HTTP_AND:AXE HTTP_VIA:1.1 AMARANTH HTTP_ACCEPT:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 HTTP_ACCEPT_CHARSET:ISO-8859-1,utf-8;Elven Runes;q=0.7,*;q=0.7 HTTP_DWARF_TOSS:false HTTP_ACCEPT_LANGUAGE:en-us,en;dwarvish;q=0.5 HTTP_REFERER:http://www.youtube.com/watch?v=enpWAuhvSjE HTTP_USER_AGENT:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.9) Gecko/20100824 Firefox/3.6.9 ( .NET CLR 3.5.30729; .NET4.0E) HTTP_WALK:NOT MORDOR

supaphly42 Sep 15, 2010 +449

Last time I did that, I got arrested. :(

449

IPoopedMyPants Sep 15, 2010 +44

I do it all the time. The trick to not getting arrested is to make sure you don't expose your genitalia.

willies_hat Sep 15, 2010 +23

I'm guessing that you personally achieve this by not removing your pants.

IPoopedMyPants Sep 15, 2010 +25

That's the trick.

willies_hat Sep 15, 2010 +20

I think you were sitting two rows behind me on the bus with me this morning.

IPoopedMyPants Sep 15, 2010 +17

I was meaning to compliment you on your hat.

wauter Sep 15, 2010 +5

Cool, you should do a netflix kinda contest to see who can predict preferred sublistnooks best for a set of users. Well I am sure it will take about 4 seconds after the data is available before some listnookor posts the idea. Just remember I said it first, boys!

racergr Sep 15, 2010 +7

I'm not ticking the box unless you clearly state that we would like to have the results posted here. Preliminary results are good as well.

If the data is released to the public I can't guarantee that everyone that downloads it releases any results of analysis that they do. That's what public means. But the idea is to get a project going in /r/listnookdev, so the process would be open

lemontrees Sep 15, 2010 +18

I am not very comfortable with public release. By making it public release, anyone can use the data for any purpose, good or bad. Currently people know(through a google search/comment history) what lemontrees has commented on. I can protect myself by deleting the account and reducing the research one can do to only the google cache. But now you are releasing csv files with my complete vote/comment history. Combined with the cache from google, one can create a complete profile about me. It is only a matter of time this is used for the whole set of wrong reasons. While I support usage of data for research and improvement, public release of this information does expose me a whole set of privacy issues.

I believe it is a username hash, but hopfully ketralnis will reply with confirmation.

yurigoul Sep 15, 2010 +2

With the use of memes and all, it might be of interest to some people to see a list of all the words being used in comments and being able track the number of appearances of those words over time. That way science people might be able to track the rise and fall of certain memes over time. For me the development of language is of great interest, but I am not a scientist (well technically I am but do not do much with it anymore). Recently a listnookor had to explain what someone meant when calling something corny in a movie review and the review was not that old. With the rise of the internet, the existence of memes and the millions and millions of none-native speaker having to write/speak english, the english language will change. The english language also changed during the times of immigration to the USA, and my guess is it will change and develop at a much faster pace now as during those times. Might be an interesting theme for a thesis in 20 or 40 years or so.

ketralnis Sep 15, 2010 +3

> With the use of memes and all, it might be of interest to some people to see a list of all the words being used in comments and being able track the number of appearances of those words over time. That way science people might be able to track the rise and fall of certain memes over time. This data is already public through our JSON API

Gravity13 Sep 15, 2010 +10

Hey, I don't know if you're aware of this sublistnook: http://www.listnook.com/r/TheoryOfListnook/ - but if I weren't so damn busy lately I'd be posting more as I have tons of ideas in the works for some listnook research stuff. For example, I made these pretty graphs from some data I took in August: http://www.listnook.com/r/TheoryOfListnook/comments/d48qa/highkarma_equilibration_why_does_64_always_like/ - I intended on dissecting the data some more, giving it a real data analysis and not the half-assed one I gave it, and coming up with a more formal social explanation of why the sublistnooks had different equilibrations (I plan on showing the lifetime of a submission by plotting karma vs time too, and then maybe matching that up with the approval rating). If other people are into that sort of thing, this is also a great place to get in on it. Right now it's by no means completely academic but I know after my physics GREs this november and finals I'll have much much more time to pick up a few projects.

IPoopedMyPants Sep 15, 2010 +6

I'd just like to thank you for having it selected off by default. I decided before going that whatever the box was checked, I'd do the opposite. If the box was already checked allowing you guys to use my data, then you have already decided to use it and you're only giving me an option to opt out of something you've already signed me up for. That's something that facebook does with every one of their new features and it is an incredibly sneaky and shitty practice. If the box was left unchecked, then you actually respected my right to choose to help the community. Showing that kind of respect for my privacy is rare among admins of *any* website. The box was unchecked, listnook respects all of its users, I checked the box and now you guys have earned the ability to use my data for research. I hope the data helps in finding a cure for getting asparagus poop stains out.

Millss Sep 15, 2010 +4

Yeah I agree, its a great idea to release this data because listnook is interesting from a lot of different perspectives... but we need a place where people can go to find/post/discuss the results of research which gets done on this data or we'll lose a lot of the potential benefits. I've made [this new sublistnook](http://www.listnook.com/r/Listnookresearch/) for exactly this reason, and I've put [a bunch of graphs](http://www.listnook.com/r/Listnookresearch/comments/de4re/basic_frequency_plots_for_a_months_worth_of/) in there to demonstrate the kind of things which can easily be done with listnook data... if a group of people with a variety of skill-sets were to start conducting research on this kind of data I think there'd be a lot of potential to produce some interesting findings...

cronin1024 Sep 15, 2010 +59

This stuff is OK > * Your community subscriptions > * Your list of friends > * Non-content information about private listnooks that you post in (that is, we may share that you posted there, but not what you posted) > * Your browser's user-agent > * Information on spam reports that you've filed (the report button) > * The last time you visited listnook at the time of the data-dump (in general this can be approximated from your last vote) But I think this is a little TMI: > * The first two octets of your IP address (that is, if you're at 1.2.3.4, we may reveal that you're at 1.2.x.x) > * A one-way hash of your email address The IP one I can understand, it helps with geolocation which could be interesting, but it's something I'd rather not have preserved for all eternity in a data dump. And what is the purpose behind the email hash if the information above is already tied to our usernames? I honestly can't think of any way it would be useful.

Noted. You're not the only one to complain about the email address (which is a surprise to me), we'll definitely think harder about that one

tyrryt Sep 15, 2010 +12

It's a surprise to you that people would not want their email addresses associated with their reading and voting activities and then provided to third parties? (yes, I got the part about the hash, but it's offensive in principle, and in any event unnecessary - usernames are unique, and if you're worried about multiple accounts corrupting your advertisers' data, disallow multiple accounts using the same email address)

ketralnis Sep 15, 2010 +18

This isnt intended for advertisers, although strictly speaking they would have access to the public dumps like everyone else

ItsOkImRussian Sep 15, 2010 +2

Where are the public dumps located?

[deleted] Sep 15, 2010 +2

In the original post they linked to a previous dump of data done a while ago. That data dump is hosted on their Amazon S3 server as a t******.

ketralnis Sep 15, 2010 +7

The discussion will be in /r/listnookdev. But I haven't made them yet because I didn't have anyone's permission to share their data

ItsOkImRussian Sep 15, 2010 +1

What hashing algorithm will be used to encode the emails?

jaxtapose Sep 15, 2010 +4

> The road to hell is paved with good intentions It doesn't matter what your intentions are. It is your responsibility to consider who else might use that data. You can't just fob it off into public domain and dust your hands of it. If you release a tool that someone can abuse, then your at fault.

burnblue Sep 15, 2010 +1

I don't remember ever giving listnook my email address. Do you have my email address?

cwm44 Sep 15, 2010 +32

It'd be cool if we could opt in without it being tied to our usernames too. I'd be happy to have you use any & all data besides the contents of my comments grouped together which the username gives, doesn't it?.

[deleted] Sep 15, 2010 +23

s2upid Sep 15, 2010 +12

Seconded. Why does the data have to be tied with the username?

jartek Sep 15, 2010 +2

I'm not meaning to harp on this subject, but once again, wouldn't it make more sense to hash the usernames? If for no other reason, emails are only optional and will limit your (or their) analysis. Presumably people with novelty accounts don't verify their emails, so I can't imagine using email addresses will consolidate user behavior. In fact, I'm willing to bet against that. Most users probably use their primary account for the grand majority of their listnooking, and only use novelties for occasional humor/karma. Unless you're ProbablyHittingOnYou or NonSensicalAnalogy.

[deleted] Sep 15, 2010 +12

ketralnis Sep 15, 2010 +16

It's intended for researchers but we'll release the data publicly as part of that process. We'll try to keep your username out of it but sometimes that's not possible

[deleted] Sep 15, 2010 +3

>We'll try to keep your username out of it but sometimes that's not possible Can you explain this a bit better? I've opted in, I just want to know what bits of my information might wind up public-facing and associated with my username. Thank you for already doing the right thing in not only asking for permission, but being mostly clear about what it means.

I mean that we'll try to keep it anonymous, but we aren't perfect, and the nature of the data is such that it may be gleanable. For instance, if someone watched you behind your back while you were surfing listnook and wrote down some of your votes along with timestamps, they could find you in the dump by looking for those timestamps and then learn the rest of your votes. It's the nature of the data so you should assume that it may be broken

cigerect Sep 15, 2010 +2

I would be fine with allowing my votes to be used for research if I could also keep them private, i.e., not viewable from my profile page (and I'm [not the only one](http://www.listnook.com/r/announcements/comments/ddz0s/listnook_wants_your_permission_to_use_your_data_for/c0zi6jl)). Of course, I'm not sure how easy that would be to implement... Bethatasitmay, I did check the *allow my data to be used for research purposes* box; and I hope you guys will report some of the researchers' findings once they start coming in.

Ok, that's good enough for me. I didn't assume, but wanted to make sure, that it wasn't going to be something where it'd be a list of my activity preceded by my username. Not to say that it's not easy enough to track me down regardless.

kleinbl00 Sep 15, 2010 +29

I debated sending this privately. Maybe that's what I should have done. But I think your community should hear this. **If you don't take the friends list off your list I'm deleting my posts. Then I'm going to wait long enough for everything to disappear off of Backtype and I'm deleting my account.** This isn't about me "opting in." I'll tell you this much - I'm never f****** doing that. It's not that I don't trust *you guys* - it's that I don't trust anyone you'd give or sell my information to. We've plenty of reason to believe that you guys go off half-cocked all the time. Most of the time, it's no harm, no foul. But even if I opt the hell out, everyone who friended *me* (and there are 172 people that I know of) has a finger pointing at me. It takes one "finger" per dimension of equation space to suss me out from my shadow. And I have no interest in that, thank you. You have no disclosure whatsoever of who gets to see my data. You did not have "we will share your friends' data if they let us, whether or not you opt in or not, and there's no way of knowing who has friended you and no way of removing yourself ever" when I agreed to your terms of service. When I gave you my email address, you didn't say "by the way, we may be whoring out our entire cloud in the future." When I gave you my paypal info, it wasn't so I could be a f****** data point for you to rent out "for science." The *least* you owe us is a preview of what, exactly, the data you're sharing on us looks like. Your Terms of Service need to be rewritten. And should you do this, I'm f****** **GONE.** No offense, but you guys are a bunch of punk-ass idealistic dreamers. You're offering up a massive cloud of information to *anybody* who wants to use it for "research purposes" and if there's one phrase that has been used to excuse the most horrors in the history of mankind besides "religion" it's "research purposes." Sorry if this comes off as tinfoil hat, but F*** You Guys. I skinned my Facebook to nothing for exactly this reason and participate in no other communities that Google can even skim besides this one. If you change it this much, I'm leaving. And I'm not looking back.

ketralnis Sep 15, 2010 +13

There's a thread above discussing making the friends list require that both sides opt in, please chime in there. For the disclosure, it would be entirely public. The goal is to get an open source community around a recommendations feature. We're not trying to bind your data to your username, just warning you that it might be possible from the data available if you opt in

IJCQYR Sep 15, 2010 +9

Thank you for posting this publicly. There is quite a bit of idealism going on here. Listnook should certainly not allow one user to opt in another user for information disclosure, and being on someone's friend list is a piece of information. Also, I think that a lot more people would be on board with this in general if the usernames themselves were not published.

dudehasgotnomercy Sep 15, 2010 +16

I think your concerns are valid, if somewhat prematurely paranoid. But geez, why so hostile?

The data dump you linked to apparently lists usernames. I don't mind my data being shared for these purposes, but it really should be anonymous. Give all the usernames a one way hash so you can keep track of which user is which, but that way theres nothing personally identifiable about the information.

[deleted] Sep 15, 2010 +5

With enough data on someone you can identify them. The concern about identifying friends is because even with just that piece of data is could be possible to figure out the friends of an "opted out" user. So in a way that bit is forcing an opt in. Of course that is assuming the hash is hacked on the usernames...

calis Sep 15, 2010 +35

I'm not ticking the box. Send proof of the dead kitten.

damontoo Sep 15, 2010 +8

This sounds okay as long as everyone has access to all the data. No special treatment for universities etc. Let us use our own data.

frickindeal Sep 15, 2010 +22

God I love this f****** site, and the people who run it. *This* is how you do things. You simply ask. Thank you.

fireburt Sep 15, 2010 +4

Sounds fine, but I'm not really down with my e-mail going anywhere outside of your hands. Until you implement that, count me in. Also, if you should ever change what we are opting into I assume you will make sure we need to then opt-in to release those new features. Thanks for letting us know and trying to make listnook even more awesomer.

> I'm not really down with my e-mail going anywhere outside of your hands Yeah, you're the second to mention this one. It would be a one-way hash, can you talk about why you'd be uncomfortable with it?

fireburt Sep 15, 2010 +10

Mostly because I know don't know much about one way hashes though I've heard that some turn out to be breakable. Can I ask why a researcher would be interested in my e-mail anyways?

It was more of a hypothetical. I didn't expect it to be controversial

isendra3 Sep 15, 2010 +2

As a grad student studying research right now, one of the mail issues looked at in an IRB is the confidentiality of data. If there is no reason to give this, then why give this?

I thought it might be helpful for correlating accounts owned by the same person. My theory is that you do as much voting on each account so if we can link them for the purposes of analysis we can keep listnook's culture of novelty accounts from making the data less useable. In any case I've removed it from the list of potentially disclosed data in the post description

It could be brute forced or found in a rainbow table. It's a bad idea, there's no good reason to do it. There are more secure ways to do what you want. Like associate an e-mail address with a unique number (without hashing, keep a table or something and dont make it public.)

Moridyn Sep 15, 2010 +6

Can you elaborate on how you plan to make these communities more "discoverable"? Even if it's just some random speculation. I, personally, don't want listnook to be a reflection of myself; I like the fact that I'm exposed to a broad sampling of information and viewpoints, hivemind aside. I guess what I'm thinking of are targeted ads based on demographics and history, which I've never been a fan of. Obviously, sublistnooks are very different from advertisements, but it's still a case of "based on this data, our computer algorithm thinks you might like *this!*".

ketralnis Sep 15, 2010 +8

By allowing us to disclose voting histories and subscription information to an open source community that could help us build a recommendations system from it

ares_god_not_sign Sep 15, 2010 +7

Please do 'em all, but give us the option to opt out of them.

ketralnis Sep 15, 2010 +44

> Did I mention that this is **optional and opt-in**?

darkfarmer Sep 15, 2010 +17

I think he means to allow for options to select which data to be sent, after opting in the program.

klavin1 Sep 15, 2010 +3

Does this give Conde Nast access to said info or just listnook?

Conde doesn't have access to any of our data atm, but this would be publicly available dumps

kleinbl00 Sep 15, 2010 +11

[Your privacy policy says otherwise.](http://www.listnook.com/help/privacypolicy)

I read his post as a more technical thing, as in "Conde has not set up a method of accessing this data atm". But I could be wrong.

phuzion Sep 15, 2010 +1

Can we get a sample of what the data actually looks like?

Take a look at the comments CSV dump that I linked to http://www.listnook.com/r/listnookdev/comments/bubhl/csv_dump_of_listnook_voting_data/

[deleted] Sep 15, 2010 -1

My crystal ball is showing me something ... the listnook frontpage .... getting clearer ... all the links go to ... wtf ... 4chan? Seriously, can we unlink our email addresses from our accounts. I knew I shouldn't have done that in the first place. And truly, this sounds more like a play for advertising than any research. And to bring up such a controversial topic on what was a fairly anonymous site when all this Truthiness stuff is going on reeks too (like the government delivering bad news on a Friday). Sites need to make money and the other sites we visit are already doing c*** like this, but this doesn't feel right at all.

-1

> can we unlink our email addresses from our accounts Yeah, it can just be removed in your prefs. Right now we just use it for password resets (if you lose your password and don't have an email address on your account, it's gone forever). And I've removed this from the list of data that could end up in dumps (but it was going to be a one-way hash anyway, it's not like we were going to go around giving out your email address).

new2u Sep 15, 2010 +2

>macapps, horseporn, arduino Ah yes! The little known community centralized on horse p***, who could have forgotten.

supaphly42 Sep 15, 2010 +2

I don't get the email part. Can you explain better to us what a one way email hash is, why info it gives out, and why it's needed in this?

tsdguy Sep 15, 2010 +1

I HAVE CHECKED THE BOX. REPEAT, I HAVE CHECKED THE BOX. Does this negate all the bad stuff I've posted on Listnook? Seriously, I do have a comment. Will you be informing Listnookors that a data dump has been sent to a particular user/group? I don't have any idea how many times you have been asked/plan to respond to requests for data. Perhaps not automatically but you might make it an option for organization to permit notification of Listnook that data has been sent. I'm sure you have a form to be filled out, include a question like "Will you permit Listnook to notify users that have opted in that we have sent data to your organization?" Another question - do you plan to charge for this data? It might be a consideration for people on deciding to opt in or not. BTW: Would you be willing to rub the Alien on the CEOs of all of the other companies that have no problem whoring our private data without notice or recourse. Might help...

[deleted] Sep 15, 2010

jooes Sep 15, 2010 +293

Question: Will this information be anonymous? Will my username be beside all of this information? > * Your list of friends > * A one-way hash of your email address I don't like these. **EDIT:** I think it's quite odd how this question hasn't been answered yet :/

293

noodhoog Sep 15, 2010 +61

I'm surprised this doesn't have more upboats. I love Listnook, but I've seen too much data collection turn evil, even when started with the best intentions. I'd be happy to provide anonymized data though - the list, minus my username, friends, and email hash. Edit to add: Also, thank you for such a transparent and honest announcement, and huge kudos for promising to default settings to off if you change anything :)

Ferwerda Sep 15, 2010 +13

Completely agreed. I wouldn't consider opting in if this data is easily traceable to my username. Not that it matters that much.

[deleted] Sep 15, 2010 +7

Yes, I don't see a problem (except what the OP brought up) except for the fact that when the Listnook team or Conde Nast figures out we're giving you our data voluntarily, they are going to start thinking about how they can make money off of it. It's not Listnook's fault, it's the nature of the beast.

[deleted] Sep 15, 2010 +10

I do not agree to be signed up for anything that tracks anything about me. I surf with private browsing mode and use noscript/flashblock simply because i don't like things intruding on me. This kind of thing seems like a listnook killer to me. If this were another site, listnook people would be up in arms setting a rally against it for intrusion of privacy.

joetromboni Sep 15, 2010 +2

Also...I'll click the box if you can do something about r/golf not having a moderator :) I've tried and tried to get something going, but have not got a response yet:(

BrowsOfSteel Sep 15, 2010 +440

>… [artificial](/r/artificial), [compscipapers](/r/compscipapers), [algorithms](/r/algorithms), [macapps](/r/macapps), **[horseporn](/r/random "good catch")**, [arduino](/r/arduino), [operabrowser](/r/operabrowser), [SketchComedy](/r/SketchComedy), [golang](/r/golang) … ಠ_ಠ

440

reseph Sep 15, 2010 +137

/r/horseporn is forbidden :( [EDIT] robotjox opened it for us. Let's do this!

137

esoomyzark Sep 15, 2010 +57

The admins are just keeping all the precious horse p*** to themselves.

ketralnis Sep 15, 2010 +293

Yes. Yes it is.

SquareWheel Sep 15, 2010 +270

Forbidden love, that is.

270

XoYo Sep 15, 2010 +53

The love that dare not neigh its name.

DEADB33F Sep 15, 2010 +47

Try /r/equineporn

Saan Sep 15, 2010 +105

No, I'm fine thanks.

locodoso Sep 15, 2010 +12

I'm glad I'm not the only one that tried

slothoholic Sep 15, 2010 +72

Only after you realized it was r/random right?

[deleted] Sep 15, 2010 +53

atomicthumbs Sep 15, 2010 +20

I clicked and ended up on [/r/kitchenfire](http://www.listnook.com/r/Kitchenfire/). What the f***?

Dead_Rooster Sep 15, 2010 +11

Holy shit, what an awesome sublistnook! I'm glad you found it.

SoBoredAtWork Sep 15, 2010 +43

You accidentally a word.

I'm a little bit worried that as soon as I saw that list of sublistnooks, my eyes were instinctively and immediately drawn to "horseporn". I didn't even look through the rest of the list and happen to notice it. *Horseporn* was the first entry I saw. I shall only use these powers for good!

zarley_zalapski Sep 15, 2010 +12

Looks like he slipped a big one in there.

Jank1 Sep 15, 2010 +11

That's what she said.

doctorwaffle Sep 15, 2010 +9

I clicked horseporn, and /r/Japan came up. Coincidence???

one_time Sep 15, 2010 +3

Wow if you move your mouse over 'horseporn' a pop up shows 'good catch'. Apologies if pointed in this thread somewhere. Too many comments.

kleinbl00 Sep 15, 2010 +2

Can anyone else get the sample dump to load? I want to take a look at it, and the server keeps resetting.

coderanger Sep 15, 2010 +1

What information about friends is in the dump? You mentioned that our account name itself is anonymized-ish, but are friends given as real names or some kind of ID?

[deleted] Sep 15, 2010 +1

how about anatomizing the data instead of giving actual user ids and community names ??? the researchers dont need the actual data but an equivalent transformed data.

avnerd Sep 15, 2010 +1

>We want features like hyper-local communities and recommendations. What is a hyper-local community?

fooey Sep 15, 2010 +1

any consideration of trying the Google Prediction API rather than rolling your own? http://code.google.com/apis/predict/ http://googlecode.blogspot.com/2010/09/prediction-api-make-smart-apps-even.html

ModernRonin Sep 15, 2010 +25

> A one-way hash of your email address Too far. Allows spammers to verify my address if they have a short list of candidate addresses. I'm fine with everything else.

minghua Sep 15, 2010 +1

I still don't quite understand the scope of this data. The voting data dump example you gave includes the username. Would these data also include the username? And if the answer is yes, my next question is: Would these information be accessible to the public like the "liked" and "disliked" pages are? And if they are not, what would happen if some of these data leaked from the researchers you gave the data to?

smallfried Sep 15, 2010 +1

The list is not very formal. For instance: How is everyone id'd? How are friends id'd? What information on spam reports will be included? What kind of hash function? Are you using a salt? Is the salt known? Also, if you share your voting data *and* have this in the anonymized list, user id's can be quickly obtained. I'm in an area where there are probably not more than 5 or so other listnookors: the first two octets will make me identifiable. You will be sharing this with people who can join in on the algorithm development in listnookdev. This means that if I would have bad intentions, I could easily join. Having a third party that is hired under a contract would have a lesser risk in my opinion.

first_danger_last Sep 15, 2010 +36

"preferences updated" What would be the purpose of providing the one-way hash on email addresses? I don't like that idea, but I'm cool with the rest.

jpfed Sep 15, 2010 +1

Regarding anonymizing data: Can someone point out to me what the problem is with the following? Say you want to anonymize email addresses. That is, you have email addresses in table Source, and you want to end up with a table Destination that contains identifiers in 1:1 correspondence with email addresses, but does not allow the recovery of those email addresses. So... Why not create a new table Mapping that has a row for every distinct email address in Source. Mapping has two columns: EmailID (GUID or autoincrementing integer) and Email (string). Now, create a new table Destination, which has all the columns of Source except Email, and instead of Email, uses the corresponding EmailID from Mapping. Finally, drop tables Source and Mapping. What is the issue with using this approach? It's so obvious (but people treat anonymizing data as a big unsolved problem) that something must be wrong with it.

It's because from the semantic nature of the data you can often glean who they are, it's not that anonymisation is technically difficult. For instance, if you know the most active users on listnook, you could look at the top-N voters in the dump. If you know their timezones, you could guess who is who based on what time of day they are voting. It's these sorts of end-runs around the anonymisation that are the problem, and why you have to treat anonymisation as broken even if you make a best-attempt at maintaining it

jeba Sep 15, 2010 +23

Perhaps to group users who use multiple accounts.

Bjartr Sep 15, 2010 +7

unique id that can be used to cross-reference study results?

Scurry Sep 15, 2010 +2

I'm kind of confused about how, exactly, revealing this information will help small communities become more discoverable.

meatsack Sep 15, 2010 +3

I'm guessing they could compare your subscriptions with other peoples subscriptions and find how similar you are to other listnookors based on this. Then they could provide a 'listnook you may like' feature somewhere that lists listnooks listnookors like you use that you're not already subscribed to.

Are you planning to give this information away (to everyone, or just people who deserve it?) or sell it?

ketralnis Sep 15, 2010 +21

On a related note, I'm looking to build a group that wants to help develop a recommender based on the next vote dump that I'm able to do based on the people that opt in here. Subscribe to [listnookdev](/r/listnookdev) if you're interested :)

LostChild1 Sep 15, 2010 +115

I'll opt-in, but only because you guys were so upfront and mature about it. I appreciate that more than anything else. :)

115

Funkyduffy Sep 15, 2010 +12

This. Recently, Listnook has treated me with more respect than my university administration.

lolbacon Sep 15, 2010 +4

In their defense, creating a Jabob's Ladder from your pubic hair in the student rec center isn't the best way to gain their respect. Unless you're in art school.

slothoholic Sep 15, 2010 +20

Don't lie, you only did it to save a kitten!

LostChild1 Sep 15, 2010 +20

Not really, as I just finished killing one by uhm... other means.

peaceisoverrated Sep 15, 2010 +36

ATM's stopped taking kittens years ago.

kobie Sep 15, 2010 +1

May I ask three questions? Who are these researchers? colleges, ad wizards, google?! What is in it for me? Not to sound self centered, but I came here looking for what will actually come out of this. Like, long term, maybe get suggested sublistnooks to subscribe to. Short term, you approve my idea of a Dynomighty Wallet with an orangered logo border.

Paul-ish Sep 15, 2010 +8

I would be happy to let researchers have my votes (anonymously), but I still wouldn't want anyone to be able to go to my profile page and see my votes.

internetsuperstar Sep 15, 2010 +160

Thanks for making it optional. I have checked the box.

160

relic2279 Sep 15, 2010 +45

I too have opted in. I've always thought listnooks greatest strength was the niche communities but they can be hard to find. Sure, you can search for what you're interested in, but sometimes it's fun to browse. And it's tough to browse 50k+ sublistnooks.

americanhipster Sep 15, 2010 +70

I've opted-in as well. In the past 24 hours I've now [donated to charity](http://www.listnook.com/r/blog/comments/ddc0w/update_in_less_than_eight_hours_the_colbertrally/), helped listnook grow with research, AND saved a kitten from the hands of ketralnis. I will sleep well tonight.

In the past 24 minutes I have eaten 3 Ambien. I will sleep well tonight.

[deleted] Sep 15, 2010 +22

Facebook should learn from Listnook how to make privacy settings...

tjragon Sep 15, 2010 +70

I want to opt in but I hate kittens... not sure what to do :(

schoule2008 Sep 15, 2010 +59

Opt in and kill one of the little devils yourself?

pdinc Sep 15, 2010 +60

*Everything went better than expected.*

[deleted] Sep 15, 2010 +25

Wow, don't know why but have read that in a demonic voice.

ZPrime Sep 15, 2010 +1

> Non-content information about private listnooks that you post in (that is, we may share that you posted there, but not what you posted) Ummm, why not include the contents of the post? They are already public.

gregK Sep 15, 2010 +60

let me unsubscribe to /r/jailbait first

TundraWolf_ Sep 15, 2010 +26

********TLDR;******** Today we're adding a new preference under "privacy options" called "allow my data to be used for research purposes"

lurkergirl Sep 15, 2010 +7

It would be nice to be able to specify certain sub-listnooks as off-limits for data mining. Take the "horseporn" sublistnook mentioned in the original post as an example...

addishero Sep 15, 2010 +9

Thank you very much for asking for our permission. Seriously.

twinkletits Sep 15, 2010 +15

Make a trophy for opting in and I bet you'll double the number of people who do so.

scaredsquee Sep 15, 2010 +5

My trophy case looks totally lame with the verified email thing sitting in there. My only trophy :(

drainX Sep 15, 2010 +4

>Coffee, sanfrancisco, **erlang**, bayarea, chrome Wow. I didn't even think about checking if there was an Erlang sublistnook. I'm doing a large project in Erlang at the moment and it's the first time I'm using the language. Loving it so far. This sublistnook will be my new home :)

jsnef6171985 Sep 15, 2010 +3

I just want to say that I love you & please don't ever sell listnook out. This is one of the most beautiful things I've ever seen on the internet, & believe me, I've seen a lot of beautiful horseporn. I'm at a loss for words for how proud this post makes me to be a listnookor. My only problem with this is that there's no way for me to post embarrassing photos of other people & attach their name to it so that if anyone googles their name that picture of them taking bodyshots off a male prostitute midgit will show up. You must fix this bug.

wtmh Sep 15, 2010 +12

See? All you had to do was ask like adults. Checked. *(Also, pay no mind the niche p********** I search for.)*

terminusest Sep 15, 2010 +2

Does not want to share * friends * on-way hash of an email (I know, probably not reversible, but still - there are cracks for most hashes, and using an incrementing or otherwise poorly randomized salt or a weak crypto for the hash is always a risk) * IP's first two octets General/Other concern with 'browser user agent': Will it be solely the user agent string, or will it be the full [Panopticlick style browser fingerprint](https://panopticlick.eff.org/)? User agent basic string, cool. Browser fingerprint or extended info? Umm, yesplez NO. Additionally, will the data collected and dumped be able to be referenced as 'one set'? IE: Identifiable in the dump as the friend list, subscriptions, private listnooks that are posted to, user agent, spam reports, and possibly ip octet/email hash/last vote given of a single unique user, which allows a lot more chance of identifying than just, say, separate blocks of people friended, spam reported by sub-listnook, user agent by bulk numbers, etc? Generally, some of the data I don't mind sharing, but as you preemptively mention this is opt-in and *should* not be identifiable, and doesn't sound like it *will* be identifiable without a lot of effort on another party's part. That said, and given that Google has a shit ton more information about me than you do, I'm not **absolutely** against any of it. And thank you for making us aware of the changes. And most importantly: For making it **opt in**. F*** yeah.

Rentiak Sep 15, 2010 +6

I'm fine with all of that, except the octets of my IP. If you made that optional, I'd be down.

Noexit Sep 15, 2010 +5

If the username wasn't included I'd participate. If you can modify it so that my data passes, but the username is excluded I'll tick the box. Otherwise, you know, [Goodbye Kitty](http://goodbyekitty.net)

WindySin Sep 15, 2010 +9

Does this mean that they'll develop some kind of algorithm that could potentially in the future create a perfect AI Listnookor who would get karma faster than that ProbablyHittingOnYou guy? Because if so, I opt in.

RedType Sep 15, 2010 +19

> Also, if you don't tick the box, I'll kill a kitten The ole hard sell, eh?

Time for some one-upmanship then. If you tick the box I'll kill a really cute kitten.

"Non-content information about private listnooks that you post in (that is, we may share that you posted there, but not what you posted)" Little to creepy for me.

theborgs Sep 15, 2010 +3

I think you are wasting time and money on this one... Don't get me wrong, it is not a bad idea, but I believe they are more important things to do to improve the site. **tl;dr Can we have a Klingon translation of Listnook ?** (my comment was not serious; I really don't see any problem with this idea and I enabled it in my profile)

endtime Sep 15, 2010 +3

I don't mind you using my voting data as an anonymous data point, but I don't want it associated with my account/username/etc. A one-way hash of my email address isn't that anonymous, because the space of all realistic email addresses is significantly smaller than the string space. Just assign a random number instead.

dymaxion_angrily Sep 15, 2010 +3

That's cute. It's kind of like asking people for legal permission to use copyrighted images on a different website. They always respond back with something like "uh yeah sure, but you know the other 99% of the internet just takes them without asking right?"

[removed]

kittenbrutality Sep 15, 2010 +1

What is the JSON API?

NotYourMothersDildo Sep 15, 2010 +26

Clearest. Privacy. Disclosure. Ever.

[deleted] Sep 15, 2010 +18

Lets be honest - the community would have reacted badly to anything less.

Just out of curiosity, why release this update now? Is 7pm PST (or so) a peak time for Listnook?

listnook wants your permission to use your data for research to build some new features!

💬 Send a Message

200 Comments