Reddit reviews Survey Methodology

Sexy basketweaving? Kinky.

> And likely the survey is poorly designed (I haven't seen it yet, but most first attempts are not so great)

Almost certainly, and the sampling frame is nonexistent. You can suggest that this might make useful pilot data for something more planned (that had also been through IRB review; human subjects and all).

I'd direct them to some survey methodology literature (Groves et al is pretty fantastic. Groves used to run the Census.) You ought to also have them go get CITI trained if your school does that, or have them look at the IRB submission process if not.

Then, as they're working on this, introduce them to some key pieces of literature and the idea that they need to enter into the conversation that's already happening.

If they're interested in the sexy question, then it shouldn't be too hard to press them to think about some sort of research question type elaboration - who answers it this way? who that way? How can we trust the sample to be representative of the larger population?

What many don't realize about surveys is there is a ton of research that has gone into finding out how to get the best data. This research spans incentives, wording, timing, visual design, etc. Here are some resources that are must reads for anyone trying to legitimately gather survey data: Internet, Phone, Mail, and Mixed-Mode Surveys, Survey Methodology, & Survey Research Methods.