malisper

So I understand the rationale for writing your own storage layer and think this is an awesome project, but there's something missing for me. One of the issues Peter brings up is they've come across a number of serious bugs in RocksDB. My question is, why would Pebble have less bugs. In fact, I would expect it to have significantly more bugs because Coackroach is the only company using Pebble.

They mention briefly how they are going about randomized crash testing:

> The random series of operations also includes a “restart” operation. When a “restart” operation is encountered, any data that has been written to the OS but not “synced” is discarded. Achieving this discard behavior was relatively straightforward because all filesystem operations in Pebble are performed through a filesystem interface. We merely had to add a new implementation of this interface which buffered unsynced data and discarded this buffered data when a “restart” occurred.

but this seems to only scratch the surface of possibilities that can come up with a crash. For example, it's possible the filesystem had synced some of the buffered data to disk, but not all of it. There's no guarantee about what buffered data was synced to disk. All you know is that some, all, or none of it made it to disk.

Bugs in this area are still regularly found in e.g. Postgres, so I'm having a hard time seeing how Coackroach is making sure Pebble doesn't have similar problems.

I think some seed rounds work this way. Some seed rounds have a small number of large checks that fill up the entire round.

For managing chemistry, the extent of our interaction with most of the investors is they are on our investor mailing list. Besides that, we ask them when we need help with something.

> A $1.5m is barely enough to invest in one early stage startup these days.

I just raised a $1.5m seed round for my startup and can tell you this isn't true. We've published all the numbers on our raise so you can take a look for yourself[0]. The average check an angel wrote was $22k and we even had a check as low as $5k. In fact, the largest check in our round was only $200k.

[0] https://www.freshpaint.io/blog/anatomy-of-a-seed-round-durin...

I've had a number of chats with Dan Luu about this. If you want the high risk, high reward of a startup, while earning the total comp of a FAANG, you can work for FAANG and angel invest on the side. If out of college, you work for FAANG for ~5 years, you'll be able to meet the requirement for an accredited investor by making over $200k a year. At that point, you can invest any disposable income into early stage startups.

In addition to getting the benefits of both working at a startup and working at FAANG, you also get the benefits of an investor! You are able to diversify your portfolio across several different startups and you are able to get liquidation preference in any startup you invest in.

IANAL - I don't believe acquiring another company and the PII along with it is necessary a GDPR violation. Article 14 of GDPR[0] lays out the requirements for what a company needs to do when a company acquires personal data, not directly from the data subject. I believe this is what applies when a company acquires another company. At first glance, it appears the email Scribd sent does comply with most of article 14 such as:

    - They notified you they were acquiring your personal data.
    - They notified you are able to opt out.

The email is missing a few things such as the legal basis for processing. The email links to the privacy policy and an FAQ, which might be enough to provide that information. I'm not sure.

Since Scribd is not processing the data for a purpose other than what it was collected for (they mention this in the email), continuing to process it for the original purpose is ok. Presumably SlideShare complied with article 6 of GDPR[1] and either asked for consent or used one of the other bases for processing the data before the acquisition.

[0] https://gdpr-info.eu/art-14-gdpr/

[1] https://gdpr-info.eu/art-6-gdpr/

Even though it says "initial back-end language(s)" it looks like the information is often pulled from the startup's current career pages. Mixpanel is listed as using Go as one of their initial back-end languages, even though Mixpanel was founded in June 2009, several months before the first version of Go was released.

I've worked in the space for over four years now and I'm currently building a startup in the space. Google Analytics is the gold standard. It's the tool everyone is familiar with and is often the thing someone does after setting up a new website is install GA.

The main startup alternatives are Mixpanel, Amplitude, and Heap. All three are unicorns or are close to being a unicorn. The main issue with GA is it's focused on analyzing pageviews. While you are able to get a lot of information from what pages people are viewing, very little of it is actionable. Mixpanel pioneered the idea of "event-based tracking" where you instrument the actions you care about and run reports over them. This let's you more complex reports like, of after a user sign up, after N months, how many of them are still signing in. Or, how does the conversion rate from users on the free trial differ based on location of the user.

The main enterprise alternative is Adobe Analytics, formerly known as Omniture. Pretty much every really large business that uses an analytics product uses Adobe. IIRC, the minimum price you will be quoted for Adobe is around $100k.

Besides using an end-to-end analytics solution, the other alternative is to use multiple services together. For this approach people will use a tool for data collection such as Segment or Freshpaint (disclaimer, I'm the founder of Freshpaint) to instrument their website. They will send the data from their data collection tool to a data warehouse such as Snowflake, BigQuery, or Redshift. Then they will use a tool such as Looker or Tableau to visualize the data in their data warehouse.

Both Fathom and plausible generate a unique salt every day. By getting rid of the old salts, they've anonymized any data older than a day. From [0]:

> We do not attempt to generate a device-persistent identifier because they are considered personal data under GDPR.

> Instead, we generate a daily changing identifier using the visitor’s IP address and User Agent. To anonymize these datapoints, we run them through a hash function with a rotating salt.

[0] https://plausible.io/data-policy

One of the claims of Umami is that it's GDPR compliant:

> Umami does not collect any personally identifiable information so it is GDPR and CCPA compliant. No cookie notices are needed because Umami does not use cookies.

From auditing the source code, this doesn't seem to be the case. First, it claims it doesn't use cookies, but it clearly uses localStorage to store a "sessionKey"[0].

The other claim, that Umami is GDPR and CCPA compliant because it does not collect any personally identifiable information is only half true. While the data collected isn't PII (because you can't use it on it's own to identify a user), it's still "personal data". This is because the "sessionKey" stored alongside all events is actually a pseudonymous user identifier. It's really just a hash of the user's IP along with a few other properties[1]. Because the data Umami collects, when combined with some other data, can be attributed back to the user, the data is still considered "personal data". That means you're still subject to most of GDPR such as GDPR deletion requests[2].

[0] https://github.com/mikecao/umami/blob/f4ca353b5c68750bf391e5...

[1] https://github.com/mikecao/umami/blob/master/lib/session.js#...

[2] https://gdpr-info.eu/art-17-gdpr/

Summary: The Recurse Center is a six week retreat by programmers. The first day of RC is mandatory. That's the day they go over how the community works, and their community guidelines which they consider to be one of the most important parts of RC. Since OP missed the first day, RC told them they would have to defer to the next batch.

IMO, it does seem like the consequences of missing the first day are a bit harsh. To be fair to RC, they do mention this is the only person to miss the first day since they've been doing RC remotely. This person is the only person out of hundreds to not know they were supposed to be there the first day.

When I went through the Recurse Center several years ago, I recall them mandating you are there for the first day. My high school graduation was the same week as the first day of Recurse Center. I wound up flying from Chicago to New York over the weekend to attend the first day of RC on Monday. Flying back to Chicago on Wednesday to attend high school graduation on Thursday. Then flying back to New York over the weekend to finish RC.

For more context, the RC team is understaffed. It's ran full time by 7 people. Notably the staff are doing this largely out of good will. RC makes enough money to sustain the community, but does not make a lot. I know they are having a hard time during COVID. I can understand why they wouldn't want to have one of their staff go through the orientation again as it would create a large burden on them.

tl;dr The penalty RC gave is harsh although they likely did communicate the fact the first day is mandatory properly and performing orientation a second time would create a burden on them.

I've done migrations of several databases with ~20TB of data before. Assuming you're data is immutable, A much easier approach than what they did is to restore a backup of the database onto a new machine. Since you don't have to worry about serving production traffic you can have the machine go full throttle on performing the migration. When the migration has finished you have to copy the new data that came in during the migration into the new database which isn't too bad.

When I've done this with 20TB DBs, it usually took several days to complete. That's much better than the 11 months mentioned in the blog post.

> You can (usually accidentally) end up with a render blocking script tag. In this article example it was client-side optimizely.

This makes sense for a tool like Optimizely which has to block rendering.

> The parsing and execution of third-party javascript is definitely non-trivial if you profile it, especially on lower end devices.

Given that this is supposed to happen after the page has already loaded, does this matter? If the thing you are optimizing for is page load time, and it takes a few 100ms after the page has already loaded to run the JS, does that actually negatively impact the user experience?

> Finally, browser download priority requires async and defer attributes on scripts (usually), or other clever ways of deferring loading.

When you add third party JS, shouldn't this be taken care of for you? Scripts for adding third party JS should already make use of async and defer as needed. What's a case where you would need to actually think about async and defer when making use of third party JS?

Can someone explain to me why third party JS has such a big impact on load time? I thought browsers deprioritize third party JS and load it after all first party assests have been downloaded and the site has been rendered. If that's the case why does third party JS still have such a big impact on page performance?

> OTOH the standard library sort can be improved if you know the distribution of the numbers you're sorting (e.g., small numbers can bucket sort, small lists can use a sorting network, etc). If this thing can efficiently almost-correctly-sort because it's better at these types of pattern matching, we can just run a final pass of insertion sort to make it useful!

I haven't read this paper, but that strategy was exactly the strategy behind "The Case for Learned Index Structures"[0]. The idea was you could train a model to predict the location of a row based on a primary key. After computing the model, you could then go through every row and compute a bound on how far off the model is from the actual row. To perform a lookup, you start off at the model's guess, then look around based on the bound on the model for a row with the given primary key.

[0] https://arxiv.org/abs/1712.01208

A CTE probably would have fixed it, but we weren't able to use them. We were using Citus which at the time didn't support CTEs.

Using a CTE to do this would also be suboptimial since it materializes the entire result of the subquery in memory. This is as opposed to the OFFSET 0 which would only materialize one row at a time.

They do wrap it in a transaction, they just omit the code for it:

> This is mostly the real code, the only difference is we also rollback the transaction on failure. It's extremely important that this happens in a transaction and both the messages and merkle trie are updated atomically.

I've personally experienced cases where making a slight change to a query in a non-obvious way resulted in a 10x or more speedup.

My personal favorite is I was generating a query that pulled out a dozen different fields from a large JSONb column. Naturally, you would think Postgres would read the JSONb field once, then pull out the individual fields from it. Instead, Postgres was reading the JSONb column once per each field. I figured out this was the case because the number of blocks read from EXPLAIN (ANALYZE, BUFFERS) went up proportionally to the number of fields I extracted from the JSONb column. Reading the JSONb column was especially expensive because Postgres needed to deTOAST[0] the JSONb column.

The obvious fix is to write a subquery to read the entire JSONb column and then have the outer query extract the individual fields, but that doesn't work! Postgres will inline the subquery, basically undoing your attempt to prevent the unnecessary accesses. In the end, the solution wound up being to add OFFSET 0 to the end of the subquery. That doesn't change the semantics of the query, but it does prevent Postgres from inlining the JSONb column access.

[0] https://malisper.me/postgres-toast/

I am Michael Malis :)

There's two issues with GA. First the functionality is somewhat bare bones. It does a really good job out of the box, but it's pretty hard to use beyond that. Take the funnel feature. You need to define your funnels ahead of time before you can start analyzing them. Compare this to any of the other product analytics tool where as long as you were collecting the events up front, you can compute any funnel you want.

Second, once you cross the free tier, you will have to pay at least $150k/year. The free tier is 10 million hits per month[0], which is high enough that most businesses never hit it. Other analytics tools often come in at a better price point at that kind of scale.

[0] https://marketingplatform.google.com/about/analytics/terms/u...

> We felt the people building the thing should have the context of usage data and this shouldn't only sit in another team - which was a behavior we saw happening quite frequently when we did user interviews early on.

What is preventing the engineers from getting access to that data? Existing product analytics tools usually don't charge by seats, so if they wanted to look at the data, they should be able to get access to Mixpanel, Amplitude, etc.

Compare that to Posthog's pricing of $25/user/month which incentives you to minimize the number of people at your company that have access to Posthog.

> We had no idea if that hypothesis was correct - that engineers would care. We did a launch HN to find out and got quite a lot of good feedback and growth.

Heap also found a lot of success on HN when they launched: https://news.ycombinator.com/item?id=5424206. While there was a lot of excitement early on, the kind of crowd it attracted wasn't a good demographic to sell to long term.

> We have users doing 5 million events/day on Heroku's cheapest standard tier dyno.

The challenge isn't ingesting the data. Especially with an analytics tool where you can turn off synchronous_commit so ingesting an event doesn't require writing to disk. The bigger challenge is around processing queries that span years of data. Queries get proportionality slower the more data you've collected. A query over 12 months of data is going to be 4x slower than a query over 3 months.

For those not familiar with Posthog, it's open an open source product analytics tool. That means you install some javascript on your website. It then collects information on what features people are using on your website and let's people run charts and graphs to ask questions about feature engagement and product usage.

Product analytics is a pretty large space. The three main existing product analytics tools are Mixpanel[0], Amplitude[1], and Heap[2] (I used to work at Heap). Notably, Mixpanel was valued at $865MM as of 2014 and Amplitude was valued at $1B as of two weeks ago.

Posthog is an open source competitor to these tools that you are able to self host. This is interesting for several reasons. If you don't feel comfortable sending data to third party tools, you control the data yourself. In addition, existing product analytics tools can get expensive. After you get off the public pricing, you will likely find yourself paying >$12k/year for one of these tools.

There are also challenges I see for them going forward. In my experience, the person who makes the decision to buy a product analytics tool isn't in engineering. It's usually a product manager or sometimes a marketer. I imagine the fact that it's open source is less interesting to one of the traditional product analytics purchasers. Compare this to a company like Gitlab where the end users are engineers.

Scaling these tools is also a huge challenge. My job at Heap was to basically scale Heap. If you want to self-host a product analytics tool, you will basically be responsible for scaling the system. On top of that, because each Posthog instance is isolated from each other, Posthog won't be able to take advantage of the shared compute power. Imagine a product analytics tool that has 100 customers and 100 servers. If the compute power was shared, each query would make use of all 100 servers. If the servers were isolated from each other, each query would only be able to make use of 1 server. That's basically a 100x slow down in performance. Posthog does sell a managed version which in theory shouldn't have this problem. It will be interesting to see long term whether that's a big driver of revenue for them.

[0] https://mixpanel.com/

[1] https://amplitude.com/

[2] https://heap.io/

Doesn't this only hand the useState hook? How would this handle something like the useEffect hook where you may want to rerun code on every render?

I've been thinking about the need for easier to use databases for a long time. I previously started a company based on selling database software, so I've seen a lot of problems in the space. I honestly most databases are too hard to use and there's been no major improvements here in the last few decades.

Take Postgres. You write code in SQL, a programming language unlike any other mainstream programming language. Instead of writing code with for loops and if statements, you write it with joins and where clauses. On top of that, Postgres has a "magical query optimizer" that takes your SQL and figures out how to execute your query. Unless you have a good understanding of indexes and how they impact the query optimizer, you'll have a hard time getting Postgres to be fast. I still regularly say WTF when optimizing Postgres queries even though I've been doing it for years. That's not to mention there's tons of database specific terminology like tables, rows, schema, etc, that you have to learn before you can become an effect user of Postgres.

As much as HN likes to bash it, I think Mongo has done the best job of creating an easy to use database. With Mongo, you can store JSON and you can pull JSON out. Of course, I would never use Mongo personally.

I'm hoping that Supabase is able to bring about the next-phase of databases by not just making it possible to make a database fast, but by making it easy to do so.

> We went a step further, and tried to define this more precisely about what it means to "offer" TimescaleDB

I don't understand how the bit you posted helped make things more concrete? Section 3.21, the section you referenced lists three conditions, all of which have to be true for your product to be considered "Value Added". I agree the third condition, the one you quoted, is pretty clear. But the second condition, the one I quoted seems really vague so the definition of "Value Added" as a whole becomes really vague.

> What that means is that if you've defined the Heap schema, you have built the indexes and tables, and then are offering a SaaS product on this, you're fine.

FWIW, Heap would automatically create new tables for customers as they sign up and would also automatically create new indexes for customers as needed. For that reason alone, I'm pretty sure Heap would violate the Timescale license.

I agree that it's pretty difficult to be specific about what "value added" means. I'm not sure what the right solution is. I would still want to go over with the Timescale License with an IP lawyer pretty thoroughly before I were to use TimescaleDB.

> All of these capabilities are being released under the Timescale License, our source-available license that permits broad usage, except for where organizations are providing TimescaleDB-as-a-service.

Maybe someone can give clarification on this, but the line between using TimescaleDB to build a product and providing TimescaleDB-as-a-service seems incredibly blurry. If I have a product that in some way let's you query time series data, and that product is powered by a TimescaleDB, would that count as providing TimescaleDB-as-a-service?

I used to work for Heap which is an analytics tool. In a way you can view Heap as just a wrapper around Postgres. We stored event data in Postgres and provided a UI that allowed you to express queries (e.g. count the number of logins over the past month). We would take the query in the UI, compile it into a SQL query, and run the SQL against Postgres. If Heap was powered by TimescaleDB, would that violate the Timescale License? In fact, you could technically view any dashboarding product that queries TimescaleDB as providing "TimescaleDB-as-a-service".

I looked at the actual license[0] to see what it says, and it seems really unclear. The license gives you permission to use TimescaleDB to develop "Value Added Products or Services" which it defines as a product that uses TimescaleDB as part of a larger offering. One of the requirements for a product or service to be considered "Value Added" is:

> (ii) such value-added products or services add substantial value of a different nature to the time-series database storage and operations afforded by the Timescale Software and are the key functions upon which such products or services are offered and marketed

This seems incredibly vague. What exactly does "substantial value of a different nature" mean? In the end, tons of products are just wrappers around DBs. If products like Heap or Datadog were to be backed by TimescaleDB, would they add "substantial value of a different nature" on top of it? In the end, Heap and Datadog are products designed for querying time series data. I could definitely make a case that they don't provide value of a different nature from TimescaleDB. This vagueness seems like a huge risk and without further clarification, makes me want to stay far away from TimescaleDB.

[0] https://github.com/timescale/timescaledb/blob/master/tsl/LIC...

Sure. I'm the founder of freshpaint.io.

The premise of segment.io is that there are lots of tools that take user behavior data from your site and it's a lot of work to integrate them all. For example, when a user signs up, you may tell multiple different tools that a user signed up:

  - You tell Mixpanel so you can create graphs of how many people signed up.
  - You tell Google Ads so Google knows a specific ad just resulted in a conversion.
  - You tell Optimizely so it knows a specific page from an A/B test just converted.

Before Segment, you would need to write code for each tool separately. This doesn't sound so bad, but it becomes a pain when you have dozens of different tools and dozens of different events you want to track. With Segment, you only need to tell Segment that someone logged in. Segment will then send that event to all your other tools. You can think of it as like a multiplexer for user behavior data. Instead of integrating 10 tools, you just integrate Segment.

The challenge with Segment is you need to write custom code for every action you want to send into Segment. This is bad for two reasons. Usually the end user of Mixpanel/Google Ads/Optimizely is a non-technical person that doesn't know how to write code. What they have to do is file a Jira ticket for an engineer to add a new bit of tracking to the website. Depending on the size of the organization, that person can end up waiting two weeks or more in order to start tracking a new bit of data from the website.

The other challenge is people often don't know what to track ahead of time or forget to track something important. For example, if you launched a new feature two weeks ago and forgot to setup tracking on it, there's no way to get that data back.

Freshpaint solves these problems by automatically collecting every user action upfront. Anytime someone clicks a button on your site, that fires an event in Freshpaint that someone clicked that button. You can then use Freshpaint's point and click UI to say that whenever someone clicks that button that is a "login" event. Then you can send that event into different tools. This is great because the point and click UI allows a non-technical user to send data into different tools and because we track everything up front, even if you forgot to track something, Freshpaint will still have recorded every instance of that action. That way, even if you decided you want to start tracking some action today, you can use our "time travel" functionality and recover every instance of that action since you installed Freshpaint.

If you take a look at some of Segment's open source code, it isn't hard to see why they wound up struggling with microservices. It looks like they subscribe to the "left-pad" style of software development. They have tons of repositories that have less than 10 lines of code. They have a two line repository for calling preventDefault[0], a four line repository for getting the url of a page[1], and a eight line repository for clearing the browser state that calls into eight different packages[2].

Disclaimer: I run a Segment competitor. I'm pretty biased, but still...

[0] https://github.com/segmentio/prevent-default/blob/master/lib...

[1] https://github.com/segmentio/canonical/blob/master/lib/index...

[2] https://github.com/segmentio/clear-env/blob/master/lib/index...

I've programmed Lisp code for several years and personally I do find Lisp code harder to read, especially when doing arithmetic.

The direct benefit of Lisp syntax is that everything is consistent. Arithmetic operators have the same syntax as regular functions. Because all Lisp code has a consistent structure, that makes it easy to do some really neat things. For example, it isn't hard to write a program that takes Lisp code as input and processes it in some way.

This is the reason why you see so many people write a toy Lisp interpreter. It's really easy to write code to evaluate Lisp code because all Lisp code has an identical structure.

As a Lisp programmer, this benefit emerges in the form of macros. Macros run at compile time, take some Lisp code as input, and spit out new Lisp code as output. The new lisp code is then substituted for the old one.You can think of macros as a very small user-defined compiler. These let you build all sorts of language constructs that you couldn't build in other programming languages. For example, you can add pattern matching to a Lisp with macros. The macro would take a list of patterns and corresponding expressions as input and spit out some code that checks for the patterns and runs the corresponding expressions if the pattern matches.

I understand this as phenomenon as the new grad and the senior developer are optimizing for different things. The new grad is focused solely on the asymptotic complexity of the code. It doesn't matter how slow or how complicated it is in practice, they are solely focused on using the fastest data structure asymptotically.

The senior developer optimizes a different set of criteria:

  1) How hard is it to understand the code and make sure it's correct.
  2) How fast the algorithm in practice.

There are several different reasons why the performance of the algorithm in practice is different than the performance in theory. The most obvious reason is big-O notation does not capture lots of details that matter in practice. An L1 cache read and a disk IOP are both treated the same in theory.

A second reason is the implementation of a complex algorithm is more likely to be incorrect. In some cases this leads to bugs which you can find with good testing. In other cases, it leads to a performance degradation that you'll only find if you run a profiler.

I one time saw a case where a function for finding the right shard for a given id was too slow. The code needed to find from a list of id ranges, which one a given id fell into. The implementation would sort the id ranges once ahead of time and then run a binary search of the ranges to find the right shard for the id. One engineer took a look at this, realized that we were doing the shard lookups sequentially, and decided to perform the shard lookups in parallel. This made the code faster, but we still would have needed to double the size of our servers in order to provide enough additional CPU to make the code fast enough.

Another engineer hooked the code up into a profiler and made a surprising discovery. It turns out the implementation of the function was subtlety incorrect and it was sorting the id ranges on every call. This happened because the code sorted the id ranges inside of a Scala mapValues function. It turns out that mapValues does not actually map a function over the values of a hash table. It instead returns an object that when you look up a key, it will look up the value in the original hash table, then apply the function[0]. This results in the function being called on every read.

The solution was to replace mapValues with map. This dramatically improved the performance of the system and basically brought the CPU usage of the system down to zero. Notably, it would have been impossible to discover this issue without either knowing the difference between map and mapValues, or by using a profiler.

[0] https://blog.bruchez.name/2013/02/mapmap-vs-mapmapvalues.htm...

Fun fact - a lot of this is kind of stuff is used in SQL databases! Before you finish a transaction (i.e. a sequence of commands), you can rollback the transaction. This will undo the effects of every command ran as part of the transaction. It's interesting to take a look at how different production databases implement undo:

MySQL and Oracle use the "reconstructing previous states from the current state" strategy. As commands are ran, they keep track of how to undo those commands in the "undo log". If you rollback a transaction, all they do is run the commands in the undo log.

Postgres uses a different strategy. When Postgres modifies a row, it marks the old row as deleted and inserts a new row. Furthermore these changes are associated with the current transaction and considered to be pending until the current transaction completes. If the current transaction fails, Postgres will use the original version of the row because the transaction that modified it never actually completed.

SQL databases have a completely separate form of undo used for point in time backups. This is the ability to restore a copy of your database from any point in time in the past. As far as I know, use a combination of the "recording each state" and "reconstructing previous state from initial state". There's two parts to your typical database backup. There's the base backup which captures the state of your DB at a specific point in time. Separately there's a redo log that captures every change made to the data after the base backup was created. By replaying the changes made before a certain time, you can restore the DB to any state after the base backup was created.