iknownothow

I would consider the following optimizations first before attempting to rewrite an HTTP API since you already did the hard part:

1. For multiples processes use `gunicorn` [1]. Runs your app across multiple processes without you having to touch your code much. It's the same as having the n instances of the same backend app where n being the number of CPU cores you're willing to throw at it. One backend process per core, full isolation.

2. For multiple threads use `gunicorn` + `gevent` workers [2]. Provides multiprocessing + multithreaded functionality out of the box if you have IO intensive. It's not perfect but works very well in some situations.

3. Lastly, if CPU is where you have a bottleneck, that means you have some memory to spare (even if it's not much). Throw some LRU cache or cachetools [3] over functions that return the same result or functions that do expensive I/O.

[1]: https://www.joelsleppy.com/blog/gunicorn-sync-workers/

[2]: https://www.joelsleppy.com/blog/gunicorn-async-workers-with-...

[3]: https://pypi.org/project/cachetools/

It makes me think of "cute aggression"

The problem is, which "database"? If you know the data is always going to be in the range of ~1TB, use Postgres or some other ACID database. But I don't know of any petabyte scale ACID compliant database.

Also I'm not sure if ACID is sufficient or not for banking systems.

I'm very interested in such products. It appears to be similar to Materialize DB.

I use dbt and Postgres for managing a data warehouse. It appears Risingwave doesn't have support for dbt yet, there is an open issue [1]. Materialize supports dbt. Materialize also offers Postgres wire compatibility i.e. I can connect to the Materialize using Postgres drivers. I wasn't able to find this information on Risingwave yet.

[1] https://github.com/risingwavelabs/risingwave/issues/7577

Question for OP - How big is the company you work for?

I work for a company that has around 50 technical employees and I'd say use of LLMs is putting pressure on the company against hiring more employess rather than actually laying people off.

My armchair estimate is LLMs make the employees ~5% more efficient "on average" (not everyone is using it or using it effectively), which is 15-30 minutes more efficient per day. That would mean you'd start thinking about laying off 5 people if you're a company with 100 technical employees. If you're a smaller company, it would be premature to lay off employees based solely off of the impact of LLMs.

I don't know much about him apart from him being an amazing story teller. I've listened to this story told by him about 25 times. I hope y'all enjoy it too. It's called the LA speed check story.

https://www.youtube.com/watch?v=Lg73GKm7GgI

If this is you, then take a moment and empathize with yourself. If you realize that your situation can't be helped then embrace opening packets and not feeling guilty about being unable to cook. Once you let go of the guilt, your inner engineer takes over and you can optimize which packets of food to buy.

You'd be surprised that there are many healthy options for food from a packet out there such as Soylent, Huel etc. I for one have embraced Huel's hot and savory suite of meals and I've optimized/replaced at least one meal a day. It's healthier than what I'd cook, cheaper than what I'd cook, much much faster than what I'd cook and I've become leaner for free.

I had investigated if this was possible to do but I couldn't find a way to do it myself. It's possible that pd.Timestamp might be intricately tied interally to vectorization optimizations using numpy or others. Therefore separating out just the timestamp stuff from pandas would be a lot of work.

I know some of the HN crowd are gonna wag their finger at me but the moment I have to deal with timestamps in Python, even for the most simplest of cases, I parachute in pandas. `pd.Timestamp` (and its friends like `pd.Timedelta`) in my opinion are the most powerful timestamp toolkit available anywhere.

To me it's very often easily worth the extra 50-150MB of install size that just works. I've used pandas for around 7 years now and it was inconsistent and buggy at the beginning. But these days, it's practically flawless. Also I'm aware that pd.Timestamp is built on top of many awesome libraries.

Does anyone know how many Falcon 9s were required to make it to 200 flights? I assume they didn't actually make 200 rockets.

We use dbt at work. The entire analytics lineage (not the data) is laid bare for all to see. There is no analytics done outside of dbt. If PII were to enter into the lineage accidentally, it's easy to spot and remove. You also get to know which downstream models have used this PII.

My opinion, comparing Americans with Germans I see and I find two obvious differences:

  * On average Americans are likely to be obese and very likely to be overweight.
  * Fit & muscular Americans are much bulkier than fit Germans.

My conclusions

  * On average, Americans consume way more sugar.
  * On average, Americans consume way more fast food.
  * I think American meat has more steroids.

No thanks.

Here's why I dislike it. There's a lot of libraries out there in Python that appear to be DSLs or have custom syntaxes that heavily use decorators etc e.g. Flask or FastAPI. In exchange for learning these custom syntaxes they offer tangible value in terms of eliminating boilerplate or solving a particular use case really well. In case of Flask or FastAPI, they make writing http/rest servers a piece of cake.

This library however offers not much benefit over the following example from Effective Python and in return I have to learn a whole new Python.

  def add_ints(a, b) -> Tuple[str, bool]:
      if isinstance(a, int) and isinstance(b, int):
          return a + b, True
      else:
          return None, False

  result, status = add_ints(1, 1)
 

Basically it's is a lot of syntactic noise for very little.

If you're a beginner absolutely give it a try!

I implemented all of the book almost 10 years ago in Python and pygame. I was starting to code professionally back then and it is responsible for much of the confidence I needed. It definitely is more about nature and math than code. If you're the kind of hands on learner like I am, then I'd recommend using this as a resource to learn to code first before learning about theorectical data structures. Thanks to the author, I am now in a position to buy it :)

I'm a data engineer and I only know Python. It appears Golang hits a sweetspot on many metrics such as performance, parallelism, ease of use etc and since 2016 there's been a lot of new data products and tools written in Golang. So it makes sense to me that the book would use a popular language for the domain.

That's interesting! I don't have much experience with Clickhouse, especially not in the last two years. I'll have to try this out myself. That's a pretty incredible if it can handle batching internally.

The total dataset size doesn't seem much at ~1TB. So you can use Clickhouse for fast analytics like others have suggested. Consider hosting on Hetzner since they have cheap Nvme disks.

The biggest problem you're going to face is ingestion of these events during peaks at 500k events per minute. You can't ingest them individually into Clickhouse or most other databases. So unfortunately you will have to add one additional streaming layer to cache these events so you can create batches of events once every few seconds and ingest a big batch of 1k-10k events into Clickhouse. AWS API Gateway + Kinesis is operationally easy to set up and quite cheap and should be able to handle your peak load. Afterwards use a Lamda to batch >1000 events from Kinesis and insert into Clickhouse. I've never tested this last part so I'm not sure how it will work out.

It'd be nice to know what you eventually go with. Please send me a message if you can of what you've finally chosen.

If it's a database or similar product, then a docker image. If it's a utility that I can test against my databases, then a binary :)

Many many demo gifs or videos for as many use cases you can imagine right in your home page. Preferably these demos should progress from the simplest use cases to more complex ones.

Show me the code snippet of the hairiest aggregation.

Show me what needs to be done on the terminal, what needs to be done on the IDE, what needs to be done on the browser etc.

Finally good realistic benchmarks. Several gigabytes of data that all start on an SSD (not nvme) then finally output aggregations and time taken to do so.

If you've been intellectually honest in your demos and your product is good for my use case, no reason I wouldn't try it.

You can watch the full interview here [1].

Jon Stewart knew he was just grilling a person of power with no real significance. The real underlying problem was laid out in an example in Jon Stewart's closing comments from 01:05:00 to 01:09:00.

[1] https://www.c-span.org/video/?527213-1/comedianjon-stewart-d...

I'm a visual learner and you have wired my intuitions about this topic permanently. Amazing work!

1. How many insertions do you expect per second or per minute?

2. What's the size of each insert?

3. At the end of one year, what's the total size of your dataset?

4. How long can your largest and most complex analytical query take to finish? Should it finish in a minute? Is it okay if it takes an hour? Is it okay if it takes upto 24 hours?

We use Postgres as a Datawarehouse, once AlloyDB Omni is stable there is virtually no reason for us not to use it.

Obviously, we use Postgres not for its performance but for its ergonomics, first class dbt support and unbeatable extension ecosystem. Now if you're telling, I get all that with no compromises whatsoever and with a 10-100x analytical query performance increase, I'd be crazy not to use it.

From my perspective, Postgres just keeps on giving.

Cleaning my cats' litter boxes.

I turn the vacuum cleaner on and put the hose into the box so it sucks all the bad air out. It also sucks all of the dust out when I'm scooping and when I put in new litter. I also set the air purifier at home to max. I put on gloves and I put on a mask before cleaning. I also buy the best unscented litter money can buy at my local pet store. Over time I've found out which cat food leads to less stinky poos (on going process).

I'm open to some good suggestions.

Almost all European companies will be inclined to use Azure in EU regions to get data privacy guarantees than use OpenAI's API directly.

How much memory would a spark worker need to process a single JSON file that is 25GB?

To clarify, this is not JSONL or NDJSON file. Just a single JSON object.

I'm not a prompt engineer but prompt engineering is a real skill* and you can specialize for different models. For example, a good and well informed prompt engineer will have intuitions that GPT like LLMs are bad at arithmetic tasks even before they ever made a single prompt because they understand or at least have a good idea of how these models actually work. Sure, they won't know for real, unless they've tried to make the models to arithmetic.

I think the very best prompt engineers for GPT3/GPT4 are working at OpenAi. I would be very surprised if no "guardrails" put around ChatGPT are implemented using embeddings. It makes perfect sense to use embeddings to put up guardrails and makes perfect sense as to why there are jail breaks.

* I wouldn't call it a real discipline yet.

edit: rephrase

It looks incredible and I truly hope your project takes off for my sake since I have to work with Kafka from time to time!

I had a quick look through the code snippets and docs so maybe I missed it, what happens if the message received on the topic doesn't match the pydantic model? Is there an error handler or callback?

What you're describing is the common symptom among many programmers which is their unfamiliarity with disk limitations. It's not the database's fault for not being able to write 10M rows to the disk quickly. Disks, especially the cloud SSDs are quite slow defaulting very often to just 3000 write IOPS.

On the other hand, programmers are very familiar with CPU and memory limitations. Very rarely will a programmer try to hold 10M or 1B rows in memory because they intuitively understand memory limitations which is usually around 4GB per app/process.

Sorry but in my opinion you can't beat free styling with Excalidraw and a bunch of nerds leaning back in their own basements and home offices thinking deeply and being critical of ideas.

The overhead of having such meetings is next to nothing in remote compared to the office. No one has to get up to go to a meeting or book a meeting room.

Did you just have a eureka moment 5 minutes after a meeting ended? No problem! Ping the group chat and get on a call in less than 15 seconds while the idea is still hot in memory.

Oh you want to pick up where you last left off? No problem, upload last meetings Excalidraw file and start editing.