thingsilearned

+1. Historically we’ve had to really bring order to nature in order to efficiently make food. In the future there’s an opportunity (with drones, ai, robots, etc) to instead harvest from the wild. It’d never be near as efficient but would definitely keep a balance.

This is a great group already doing this with bison, and he’s got a great book as well

https://wildideabuffalo.com/pages/regenerative-ranching

would love to participate!

We've definitely gone much deeper in this version with data modelling.

This book though is to be clear (and that's in the first about sections) does not have much on product/event based analytics nor giant scale data as judged as 100+ TB data management.

Thanks! It was definitely a long time in the works!

100% here. We need to drop old terms and methods that are now obsolete.

We don't work with rest calls - just directly with databases and SQL, but do take a look at chartio.com! We aim to make it as fast and agile as possible to make and collaborate on charts and dashboards with your team.

Great writeup here but I have a question on whether Product Analytics is at all appropriate for Startups. I'm incredibly biased (this problem is why I built Chartio), but I think these tools end up being quite bulky, expensive and a non-agile way to do analytics on your product. I feel these products are best suited for later stage companies trying with great tracking detail to improve a funnel or engagement metric, not simply monitor usage.

The majority of the important things for a startup are usually already being tracked with timestamps in the database (new signups, new users, churned users, new todo items, etc). If a key metric is not directly tracked, there's usually a good enough proxy available somewhere in the database. A Busineess Intelligence product, or a SQL to chart tool is much more applicable and affordable in the startup stage.

There are likely going to be responses here "But what about..." and I'll premptively respond to those with the question: Could that be answered/solved almost just as well with a query to the database, without sacrificing the agility, extra setup and data collection time, and expense overhead? Those are all very costly things for a startup.

Dave from Chartio here: we've just launched a complete re-write and extension of our visualization library with many more settings, maps, customizations and updated themes.

We're now iterating quite quickly on expanding and improving the options and would love any feedback!

I've had similar thoughts to the title, but my ideas were more around improving the ease of use/writing.

1. A cleaner universal more natural syntax for analytics: I love writing python as it is to me such a cleaner syntax than C or Java. We could do the same for SQL and make something that feels more natural. Turning a common query like

> SELECT count(*), TO_CHAR(created_at, 'YYYY-MM-DD') FROM Accounts GROUP BY TO_CHAR(created_at, 'YYYY-MM-DD') ORDER BY TO_CHAR(created_at, 'YYYY-MM-DD');

into something much more natural like

> count by Day(Accounts.created_at)

2. A Visual SQL: for analytics it's so much faster to query and explore visually. Building queries visually means you don't make common typo or syntax or structure errors, joins happen smoothly, you can browse the data as you build, you don't need to google for syntax (what's that date function again?), and it works across dialects and databases. We've built and launched this a few months ago at Chartio https://chartio.com/blog/why-we-made-sql-visual-and-how-we-f...

I've been fairly productive especially in the first few months of lockdown.

1. Chart based commenting system for Chartio https://chartio.com/blog/charts-worth-commenting-on/

2. Helping the https://howwefeel.org team with their data and dashboards https://how-we-feel-chartio.herokuapp.com/the-how-we-feel-pr...

3. A number of hydroponic experiments including with lettuce https://img.chartio.com/nOueDAy2 and even trying corn https://img.chartio.com/NQugK2G0

4. Finishing a book on Data Management that'll have a dead tree edition published later this year https://chartio.com/blog/cloud-data-management-book-launch/

5. A few small wood projects like a planter bed and a projector mount

+1 here. Figma's collaboration features are also just incredible so many people can participate simultaneously, and make copies w/ their own versions.

We've got ours also well setup with a big library of assets like buttons, and illustrations we've had made, and different example charts, so if what you're discussing is a new feature or landing page it's really easy to drag in a bunch of ready-made components to express your idea.

Dave, founder of Chartio here. With so many of our customers working remote we realized that we needed to prioritize our collaboration features and worked long hours to get this out.

We're really excited about how this came out - like comments on google docs, it makes for an entirely different and more collaborative experience. I hope it helps teams create better dashboards together and more easily discuss their data.

Thanks, the user-research-first part was definitely enlightening and important to do. Our first many iterations were very wrong.

As for large scale datasets - we do cache those sample tables and we only grab the first 10 rows. Also, BigQuery actually has a great API for fetching a set of sample data that we utilize heavily. They made it because of exactly the potential concerns you outlined around columnar stores.

I wish I could say that we had that insight from the beginning and that it worked right away, but we ended up failing into that realization after a lot of designs and prototypes.

So what you describe is somewhat built in to what we have now. Users still have to choose what columns they want to look at (there's no real way for us to guess that) and then we do apply some knowledge on what type of data they're looking at to help them get to what they're likely looking for.

We've also tried at times to make default dashboards for data sources when people connect. We can do this to some extent with known Schemas like connecting GA, SalesForce, Hubspot, etc, but for databases - that's proven to be a largely impossible task so far. Everyone's data is so different, and have such odd conditions to consider filtering by, that the auto dashboards end up being quite useless.

We do compete very directly with Tableau and are often used along side it as well. In the article I explain a lot of what's different about it from other BI products like Tableau. The main things are - it's way more flexible as a visual language, and through a lot of prototypes and user tests has shown 80% of business users can write significant queries on their first try.

As for an established service - Chartio's been around for almost 10 years, we're profitable, and are the main data interface behind some really great companies and brands https://chartio.com/customers/

Yes we do require a work email to signup as our target audiences are businesses. We've discussed launching a public version which I'd love to have sometime soon.

It's free to download on dataschool.com. I agree on the gap and that's why we wrote this book. We launched that on HN here 5 months ago

https://chartio.com/blog/cloud-data-management-book-launch/

Yes it does

Thanks! Hadn't noticed that the icon was mirrored, thanks for pointing it out!

Yes, right now it's part of Chartio. I'd love to eventually open it outside of that, but there are no immediate plans.

No, sorry that's a marketing/sales team decision. I'll talk with them about it.

We do have a reverse SSH tunnel connection https://chartio.com/docs/data-sources/tunnel-connection/

So the agile world of data we're looking to bring about is one where your DAO or marts, are still regular schemas - not just flat tables or cubes. So the developer teams would be creating these mart schemas, and then business users could be doing much of their own self-serve exploration and visualization with Visual SQL. I wrote about some of these modern architecture approaches here https://chartio.com/blog/cloud-data-management-book-launch/

Oh awesome! I haven't heard of SQL bucket but I'll check that out as I love anything that encourages SQL based modeling. I was writing my own and then DBT came out and we push people there primarily. Send me a note - dave-at-chartio and I'd love to chat sometime.

Ah yeah, you do have to signup to try it out (though it does start you off with demo data). We wanted to offer a non-login version but it makes things more complicated for a company that obviously needs to prioritize security highly.

This video is maybe the closest you can get without trying yourself - https://www.youtube.com/watch?v=YBXMTipHGfQ

Ha, sorry about that! Are you talking about the interface, the idea, or the content?!

I fully agree with your experience here. We have a super hard mission at Chartio - it's not just about the interface but also how the data is setup. The interface, being as flexible as it is and also enabling full schema (instead of dataset) browsing is a pretty big part though in also allowing a more agile version of data modeling. It had that very much in mind and we've written a book (soon to be published with Wiley) on proper modern data governance techniques.

https://dataschool.com/data-governance/

Our next phase is to help people get to that cleaner source of truth much more quickly than traditional dimensional modeling approaches. Tools like Visual SQL and DBT (https://www.getdbt.com) are really changing the complexities here.

No idea. Thanks for reporting. We'll look into it

Totally - it's not a solution to everything. We can though do as much as possible making the UI intuitive, flexible and fast. Those gains are not just for business users but for everyone.

Every product needs some education (excel is still of the most popular online courses) and we knew ours would be no exception which is why we've also last year launched DataSchool - our free online community driven courses on data

https://dataschool.com

Yes we are! No plans to go on-prem yet but security is obviously a giant focus and necessity

https://chartio.com/product/security/