Comment by abakker | Hacker News Reader

Comment by abakker | Hacker News Reader

Longtime SurveyMonkey customer here, but, I have largely switched to SurveyGizmo and Qualtrics. 2 primary reasons:

1. I work with Panel Providers, and their Quota tools and panel integration tools are just a bit clumsier. They don't really have Data Quality checks, they don't have cross-logic quotas, they don't have a kind of survey flow tools.

2. Analysis stack: They are currently chasing analysis in a rather frustrating way. (e.g. Online). SurveyGizmo and Qualtrics both have stronger features to export to SPSS files, which is the survey industry standard. SPSS files are read by many tools that specialize in survey analysis - R, Q, Wincross, etc. As an actual enterprise survey user, I will never use the web reporting tools for anything more that casual sample monitoring. I can't build my final products on those, pretty much no matter what they do. I'd rather they invested in better downstream exports.

Things I wish that they, or anyone in the industry would do:

1. Export to some kind of database, where data attributes could be stored. I don't know whether this is just a bundle of CSV + Pandas code to get the survey into a data frame, or a sqlite file where the data connections are preserved (e.g. logical groupings of multiple choice, multi select questions). SPSS files are a _good start_ they get almost everything right, but, they are proprietary-ish, and they leave a lot to be desired in workability.

2. Would be nice to see more up front indications of the types of analysis that various survey questions can provide. i.e. indicate to users/survey authors what kinds of actual relevance the answers to their questions have.

I like survey monkey a lot, and, if you need something lightweight, low touch, and long-running, they are great. but, if you need to do market research there are more full-featured tools to use that will make the whole process easier.

I like that survey monkey finds an audience and induces them to answer your questions.

I also know roughly how much it will cost.

I can't tell pricing from SurveyGizmo or Qualtrics after several seconds of clicking around.

This is their pricing tab: https://www.surveygizmo.com/team-enterprise/

come on!

SM's audience product is good if you are consumer facing, not B2B. The costs are nice, and the revenue model for the people helps avoid garbage data. (SM donates to charity on behalf of the survey taker, rather than paying them directly). In B2B surveys that I run, I end up paying a lot more per complete (for, say, a VP of infrastructure at a company over 1bn) and I still have a high disqual rate and low incidence.

SM is a good product, don't get me wrong. The team is great, their support is great. The reasons I outlined above are the _only_ reasons I am moving away from them.

But the audience product is the product. They don't charge for anything else.

Is there something comparable that finds an audience for you?

Or, to put it another way, for B2B I take it you have to find your own product "by hand" anyway, right?

SurveyGizmo's pricing is hard to find but I eventually found this page https://forms.surveygizmo.com/plans-pricing/

Thanks. Good find.

I was looking to see if they find an audience for you and put them in front of your survey. I can't tell if they do that.

I wasn't sure if you were saying SurveyMonkey is missing SPSS export - it supports exporting to SPSS AFAIK. Is it that the export is missing data or features for your needs?

It has them - the features for naming every individual choice, specifying labels, and otherwise tweaking that output in SurveyGizmo are much stronger. This isn't a dealbreaker, but makes archiving past analysis, reports, and data files easier and more descriptive than e.g. "Q1_2". It also makes it a bit nicer if you don't have to do the renaming downstream, since it just avoids another step where you can introduce errors. SurveyGizmo also has some features specifically for R, though I haven't used R yet.

Long term, the whole survey industry should probably move away from SPSS to something fully open source, with a file type that is truly standard that can be read into all the different systems.

PS - I know that there are APIs for individual questions, data, etc. But, as an analyst first and programmer much much later on my list, I find them too complex to really be worth it. A lot of survey analysis is one-off.

I'd love a pandas/numpy/python call that just looked like:

get_survey_as_dataframe (surveyID, API_key, any other stuff).

Aha! Got it, that’s good feedback.

Hiyo. Cofounder of Statwing (YC S12) here.

We make an easy-to-use statistical analysis tool, and we were acquired by Qualtrics in 2016 (and rebranded as "Stats iQ").[1]

There's a lot of situations where I could imagine preferring to analyze data in SPSS or Python or something (e.g., complicated transformations, exact same analyses run routinely, advanced analyses that we can't do yet). But, to my mind, ideally you'd not need to export so you can just stay in one tool and work on the one canonical version of your data.

I'm speculating here, but is it possible that your complaint isn't with the fact that the analysis is online per se, but rather that no online analysis tool you've used is anywhere close to as good as the standard offline ones? Or are there really features that you couldn't imagine existing online that you really rely on?

Asking partially b/c it makes my shilling more informative and interesting, but partially because I'm genuinely curious. (Apologies for the shilling, folks, I know it's not great form, but startup habits die hard.)

[1] Go to the bottom of this page to watch a demo: https://www.qualtrics.com/iq/stats-iq/

So, it is interesting, but the big issue isn’t just the analysis, it is the presentation. If a company is doing analysis for its own sake, then I could see the process working well using StatsIQ (I have a license, BTW) but, in the end, my reports end up in PowerPoint with custom style sheets and templates.

Until my customers are ready to consume a web interface, and the tools produce a set of web-ready charts (maybe with something like plotly?) it doesn’t do me much good.

Another problem is that I work with plenty of analysts that are not going to get qualtrics licenses, or access to my surveys. I also have a team of people that codes surveys in qualtrics but doesn’t really do analysis. Right now, the licensing doesn’t favor me giving a bunch of users access to the survey tool, and doesn’t have fine grained enough control to manage them if I did.

Long story short, for me, analysis and publication are tightly coupled, without an end to end solution that changes my current workflow, having portable files and PowerPoint charts remains the most effective solution that still enables some specialization of skills in my team.

Been out of that realm a few years, but if using panel providers anyway, Sawtooth was pretty much only software that didn't make me want to smash my computer. Especially any online builder tool, yuck. Never did run into anything we couldn't make it do.

Another lesser known survey analytics tool that just came out recently is Surveyking.com. They have a lot of really advanced features that some of the smaller companies don’t have and it was all coded by one guy apparently. From the pricing, it looks like they are only $19 a month for everything. https://www.surveyking.com/pricing

What is a panel provider?

Companies like Research Now, Mindforce Research, SSI, etc. They allow you to buy completes for specific research surveys. The completes are anonymous, but the respondents wive categorical information like Role, Title, Company Industry, Size, etc. Respondents are paid for taking the survey.

As an aside, panel data quality has gotten a lot more iffy in the last year or two. Seemingly, bots are getting much more sophisticated at "faking" a valid looking response. I now spend more than half of my actual surveying time on response validation. For those of you interested, you need to have logic that cross-validates, (i.e you said you were using AWS in one question, but don't use public cloud in another) and you need to read open ended responses. Even if you don't want open ended responses, I would not do any survey without at least 2 open ended response questions.

Reply on news.ycombinator.com