Hacker News Reader

Comment by anonu | Hacker News Reader

One thing I've had to remind myself of - coming from a CS/Engineering background similar to most folks on HN (I'm guessing) - is that there are 2 types of people: Those who program and those who don't.

To me, Salesforce looks like a big shared Excel file with a bunch of sheets. Tableau... well I can do the same thing with some scripts or spin up a web server.

To others, this tech is just magical. Pay the money, do the integration and it just works... And clearly people will pay a lot of money for things that "just work".

View on HN

>well I can do the same thing with some scripts or spin up a web server.

That's a massive waste of your time and effort. The maintenance costs become massive as well should you choose to create your own system. At the VERY least you should leverage existing free open-source tools such as Metabase or Superset or Dash, or free tools such as Google Data Studio or Mode Analytics, if you're not going to spend cash to get a tool like Periscope Data/Looker/Tableau. I mean this gently, but you likely underestimate the complexity of a reliable reporting/analytics infrastructure. Think about it this way - these tools are either collaborated on by a large open-source talent pool, or are created by teams of dedicated software engineers just as talented as you.

I've worked with quite a few companies in an analytics consulting type role, and your "I can do the same thing with scripts" statement is one I've heard countless times. The long-term maintenance costs and technical debt (and "rigidity cost") of rolling-your-own analytics far outweighs the cost of a true analytics platform.

If you decide to roll-your-own anyway, look at tools like DBT and Airflow to reduce long-term maintenance costs.

> that's a massive waste of your time and effort.

Yeah I work at a company in the analytics space and see that all the time. It peaks the curiosity of people who are software developers (yet their core competency at their job is something else). They think its a fun work side-project and go after it. Write some python scripts to do ETL and process the data...make a backend with pg, a web server, then do charts in d3.js.

A year later they have a bunch of nice demos to their bosses but nothing that they can actually use in production because it crashes, there's no UI for interactive queries, no reports for people in the business groups, no user management etc. Then they drop it because they're busy with their actual job. So the cost of that engineers time to do something that didn't work was about $20-30k over a year. While the product they could actually use in production was around the same price.

just an FYI, the word you're looking for is 'piques' not 'peaks'

This sounds like it should be included in my yet-to-be-written biography. I fell into exactly this trap at one point.

How does Tableau make your reporting more reliable? My understanding is that Tableau can hook into different data sources -- like your warehouse and SalesForce, for example. Then you can write some kind of SQL to generate charts.

The auto-chart generation is nice. But what about Tableau makes it more likely to be accurate? Aren't you just as likely to make an error on the SQL in Tableau than if you didn't use Tableau?

I never said that about tableau in particular, which is why I listed half a dozen analytics platform solutions. Using any of those makes for a more powerful, flexible, sharable, usable tool than one engineer's self-rolled internal webpage with "analytics". Nothing special about Tableau in particular, in fact, I categorically prefer SQL-based visualization tools over Tableau.

The only exception to "using any of these is better than creating your own" is large companies like Google and Facebook, where they have entire teams of engineers who are dedicated to creating an in-house SQL+Visualization tools. It is absolute hubris for one engineer to think they can make a robust analytics platform!

Tableau is a GUI first. It's not designed for SQL based access and barely supports it other than as a type of custom datasource.

Since people aren't typing code, it can be more accurate to use, and it provides visual results beyond just a table that can be useful in detecting anomalies in your data.

Out of genuine interest and perhaps even a need for such a solution, is there anything that does what Metabase/Superset/Dash/GDS/ModeAnalytics does but for realtime datastreams? For instance, parsing and recording and visualizing the events coming from a websockets or some event queue/bus?

I believe you'd want a tool such as Grafana (which I believe is free/open source), which I have seen eng teams implement for realtime streaming. There is also Kibana which I am less familiar with.

Those all work for that already, because they're just frontends to whatever backend system you're using. You a database like BigQuery with real-time streaming inserts and rerun your queries for results.

If you want analytics directly on the stream, then there are plugins available to support reading the query results of something like Kinesis Analytics or Confluent's KSQL.

Ah yes like the other user mentioned AWS Kinesis Firehose and GCP Pub/Sub may fit the bill as well

Grafana/Kibana are the actual charting tools

Yes perspective is built for streaming data - https://perspective.finos.org. Open source streaming pivoting engine that operates in the browser (using wasm in a webworker, with integration with Apache arrow for ingest of binary data strrams off websockets)

You might want to check Striim - it wraps kafka + streaming transformers with a scripting language, and a visual pipeline design tool. It then offers some really nice dashboards with a real-time “feel”. Not affiliated in any way but I took it for a spin and was happy to see how much could get done with it in just a few hours and without prior knowledge.

If you familiar with Druid then Check out Metatron Discovery( https://metatron.app) Druid is strong for data streams.

Na. Roll your own and munge text how ever you want. Don't cede control to bad GUI/web tools.

If you are good, you aren't "scripting", you are making a rad MVC system.

Salesforce puts all the MVC together in nasty, nasty ways.

This... is not good advice. Do you tell people to roll their own databases and networking protocols as well?

As an engineer, this has really fascinated me. Tableau, Salesforce, Excel, etc. were things that never made a significant amount of sense to me. I thought mostly of Salesforce as a CRUD app (which it is!), Tableau as d3 with nicer ergonomics, and Excel as... well... something I never understood.

If this is something you guys are interested in, I started a company called Retool (https://tryretool.com) that is essentially Excel for developers. Imagine if every Excel cell — instead of being a cell — were instead a React component. So you drag and drop these components around, and you can connect them to any back-end datasource (postgres, APIs, etc.). So you could drag on a table and have it pull data from `select * from users` from postgres, and then drag on a button and have it `POST` the selected row back to your API, in order to ban a particular user. The goal is to let end users build CRUD apps (like Salesforce) around their existing datasources quickly.

If you guys have any feedback... I'd really appreciate it. We're just starting out, and really curious to get any feedback from developers. Thanks!

so you're rebuilding Visual Basic in react.

Do you mean that as a good or a bad thing? VB was fantastic to many people!

absolutely a good thing.

It was a light hearted jab at the description that made it sound like something completely original and never tried before :)

I hope he does well, those tools are super useful.

I would have to delve deeper to give actually useful feedback, but I wanted to say from a presentational point of view, it looks great, well done for putting something like this together

Uh this is super super cool, well done.

When I looked into this a few months ago, your pricing didn’t make sense to me. You’re saving developer time, but charging on user time. So I’ve got 800 people in this org and I could either pay a shit ton to use retool or just build it from scratch. The difference between both of these is my time so it makes sense I’d pay you for that.

That made it not viable for me. Building from scratch was way cheaper with Upwork. Anyway, Product was cool.

Thanks for the feedback — that's really helpful. I agree that our pricing isn't good for a large org that is looking to build a tool or two (instead of say, a few hundred).

Do you mind if I email you? (I can't see your email in your profile, but my email is david@retool.in if you're interested in reaching out.) I'm really curious — just to learn — what kind of pricing plan might work for you. Would a per-app model work for you, for example? Thanks!

I've since left the company but one of my friends working there is the one who suggested we try the product anyway. I'll tell him to contact you.

Unrelated to the tool, your homepage is absolutely gorgeous. I imagine the rest of your execution is along those lines, which is to say - excellent. Well done!

To put a finer point on it, they'll pay a lot of money for tools that require less-skilled labor to operate. They will gladly spend a lot of money on Tableau to avoid paying the salary premium to have "those who program" build them an analytics system. It costs much less to hire business analysts who don't program (aside from SQL).

Now if they just taught business analysts how statistics work.

If I had a dollar for every time someone showed me a wrongheaded graph they were using to make business decisions, I could retire.

There are varying degrees of skill and competence in business analytics just like there are in software development.

>is that there are 2 types of people: Those who program and those who don't.

Is that fully accurate, though?

Where does that leave data scientists / data analysts? I know SQL very well, and I know python's data stack (numpy, pandas, matplotlib, plotly, seaborn, various stats toolkits). I have a strong understanding of the "programming ecosystem" i.e. concepts, terms, definitions, and so on. I understand (basic) computer architecture, I've used and am familiar with (basic) shell/terminal, and services like Docker/Heroku on the command line, and can certainly use GUI cloud tools for AWS, GCP, etc. I can read and understand code and how systems fit together. I've worked alongside engineers of all types.

But I'm not a software engineer. I don't tell people I "program" because my strongest skill is SQL and generally people do not refer to that as "programming".

I’d instead use the distinction “those who can sling code when necessary” versus “those who cannot/will not sling code.”

you are a programmer. At least in my book. You may not be experienced in architecting software projects, or writing highly maintainable code, or dealing with niche frameworks and tools (gui frameworks, network systems). But that is just a lack of training, interest, opportunity, etc...

This is nice to hear. Truth be told, I'd like to get some professional software engineering experience. The problem for me is:

1. It's hard to get considered to be a SWE in the first place when your job titles are more in the realm of data analyst. They'll toss out my resume for a fresh grad, much less someone with experience, without a second thought.

2. If I were to make the switch I'd likely have to start at level 0 on the scale ladder. I've already career changed once into tech, and at this point I do not wish to "reset" my experience another time

"Data Analyst" is often programmer lite. If you've been doing that work for a few years you could easily transition into a full time SWE role. If you really need a confidence boost and/or some on-trend training, consider doing a full stack bootcamp.

And those who can program are split into those who can estimate work and those who think they could replace a product like tableau with a few scripts.

Yea no this is wayyyy underestimating the costs of data engineering. I can tell you from my very limited experience some queries I've seen, on some systems and some datasets, have estimated TTLs in YEARS -- and that's the difference between companies that invest in tooling and infrastructure (and big O analysis during system design) and those that don't. And you will invest a good deal of effort and time and money before you realize it, it's like the perfect ambush.

Never start a land war in Russia, and never neglect your data infrastructure if data is in any way a key business differentiator/fundamental in your market.

And this attitude right here perfectly exemplifies “NIH syndrome”. Beware if you work somewhere and a colleague suggests that some homegrown hacked-together non-feature-complete solution would be competitive and “cost less” than the professional product.

Wait until a manager asks you to do something that Salesforce was not built for and IT tells you a back office resource can do it. None of those of the shelves enterprise solutions just work, they all need some degree of customization and integration. I am struggling sometimes to justify the cost of a vendor solution + customization versus just rolling out my own or an open source one that can be more easily built on top of.

Everything you write, I cheer.

Massive customization, and then you are bound by this broke ass Object model in which to get it all done, between Apex and VisualForce nauseating crap.

I don't want to pay hundreds of thousands of dollars for the right to do write database driven web pages.

> I can do the same thing with some scripts...

I thought that the point of Tableau was to provide a tool that end-users (who can't program) could use to interactively explore their data. That's not something you can replicate with a bunch of scripts.

I can see how Tableau would be great if you are in a situation where you can have business guys creating their dashboards and exploring their data.

The issue in practice is almost always getting the data in a workable state so that you can manipulate it easily in Tableau. In my experience in smaller and mid-sized places, Tableau tasks get punted to analytics and data science, because they are needed anyway to get and transform the data in the first place. And these people usually prefer and are capable of using more technical tools than Tableau. I know I would rather use Shiny or Dash.

Maybe that's not a difficult problem in larger corps.

If you just want a chart based on a DB query, then yes, it's a waste of time and money. The value of a lot of these products is that they have query/charting builders, which takes a lot more effort to implement. These days a lot of business aren't willing to accept analytics being controlled by a single person or team, as they're used to anyone being able to do their own custom thing.

Yep. UX is by far the most underrated aspect of product development by engineers.

Are you sure your scripts and web server costs less than Tableau?

Probably not. From his comment I doubt he's even used Tableau (and I say this as a programmer who worked on something similar - https://perspective.finos.org - and yet I still find myself turning to tableau pretty often).

Yeah, Tableau is very expensive. Way more than a few weeks of dev time.

> Way more than a few weeks of dev time.

Except it's not just a few weeks of dev time that makes up the overall cost. Consider infrastructure, maintenance, updates, support, training, etc,. Those things start to add up and you don't get the benefit of scale/community if you do it on your own. There's also the opportunity cost of building your own system when you could buy something existing and use that time to work on other things.

True I'll give you it depends on the situation and might indeed be the better option. For a startup I wouldn't recommend it at least

Tableau is $850/yr per seat. Very expensive might be overstating things a little.

So how many seats are needed to hire somebody who creates a custom shiny + R app that does exactly what you want? How big is the average company that makes use of tableau etc.? Let's do the math ...

It’s 2019. Are we seriously still trying to argue that homegrown solutions are overall more cost effective than SaaS? Are we still of the opinion that developing tooling outside of your company’s core competencies is somehow adding value?

Absolutely. There are tons of people I encounter every single day that seem to think it is cheaper to build everything inhouse. I think it comes from a complete misunderstanding of how business delivers value coupled with RMS-like distrust of anything they cannot completely crack the lid open and mess around with. Mostly the former, some of the later.

Well it's probably a question of perspective and problem domain. Also my main concern is precision and correctness. Quite often you can choose between (1) customized tools that achieve almost what you want them to do, but often the people who implemented or customized them didn't fully understand the problem and did things wrong without anyone noticing for a long time, or (2) custom tools that do exactly what you want, that were tested by somebody who cares.

Ah of course. Just a few few weeks of dev time to build the same thing a multi-billion dollar company sells. Then after a few weeks of dev time you find out you're missing about 80% of the features that makes tableau useful. After that few weeks your manager asks you how the project is going and you just mutter something about how you should probably just buy looker or tableau.

Nah, these sorts of folks are far too obdurate and prideful to ever willingly admit defeat. More likely they would keep telling the manager that they just need a few more weeks time to finish it while they frantically keep rolling that Sisyphean boulder up the hill.

We were getting ready to do a tableau api/integration with our software and the licensing came out to about $100k for two server dev licences and six client licenses.

> Those who program and those who don't.

And you should know, that the ratio between those two is something like 1:1,000,000

So, for the most of the world, tech is essentially "magic" to them.

I actually felt the same way about Salesforce until about a year ago when my company bought it. It certainly has a duplo blocks side that let's non technical users do complex things (aka Excel) but their backed is extensible through a language that is more or less Java for custom handlers/queries/etc and they are switching their front end to W3C compliant web components for developer maintainability after getting burned by rolling their own front end framework in the early angular 1 era. They have super tight integration with heroku postgres dbs (because they own them) and have some other best technical tricks up their sleeve to the point that I my opinion has changed quite a bit.

Reply on news.ycombinator.com