addisonj

This is pretty interesting, but for anyone who is on semaglutide maybe not a huge surprise? At least in my experience with semaglutide, everything I consume I need less of and I feel way less desire for things.

I am not a big sweet eater, but usually enjoy a decent sized chunk of dark chocolate about every other day. I still have roughly the same cadence of wanting a piece, but the difference now a single square is enough and I don't feel the need for more. It isn't like the desire for things like sugar, fat, etc is gone, it is more like I am just wayyy better at knowing how much of that I need.

I have been on semaglutide for ~2 months and it has honestly been wonderful for me. I have described it as being the drug that helps my brain stop acting like we live in times of famine and has massively improved my relationship with food.

It really is just a shame that it is so cost prohibitive as I genuinely think that it probably could do a ton to reduce healthcare and impact of food overproduction (I eat so much less meat and I have heard some from others), but I don't think that will happen unless insurance starts covering it

This game holds such a special place for me, not only for just fun childhood memories, but also so much of what got me into the excitement of how cool it was to see the rapid progress of tech.

Certainly, it isn't atypical for software industry folks of my age to have games be a gateway into tech, but I think I was a bit different in that I rarely considered wanting to make games, rather I enjoyed the tinkering with my computer and the rapid pace at what games could do as much as the game (which continues today with my spending more money on playing with the hardware and toys rather than the actual games) and half-life just

I first played Half-Life on a family PC with no graphics accelerator and I loved it, but I remember the jank from ~15 FPS at a few hundred lines of resolution. This is what drove me to build my first computer and after debugging my way through all the issues that a 12 a year old would make when building a computer before the age of YouTube, I remember being absolutely blown away by the lighting and speed of what this little piece of hardware added to the experience.

That wasn't the first time I felt the rush of getting a computer to do something I wanted (that would probably be getting doom running in windows 3.1 after dealing figuring out the mystical "command line") but it certainly was one of the most drastic in just how fast tech changed.

A small addendum to this... I really want to figure out how to bring similiar experiences to my kids, because it was that loop of problem -> learning -> breakthrough that I think was hugely transformative to not only my career trajectory, but also just in learning to love learning.

So thanks Valve and Half-Life team for this happy memories... but maybe I won't feel the same when I try and play HL DM this weekend and inevitably realize how slow my reaction times have gotten in the last ~25 years ;)

I don't want to say I disagree with the author, what he says here is entirely valid, but I do have one suggestion to anyone facing this issue:

Make a practice out of README's having more context about who is behind this work (individual, team, etc), the status of it, and the desired outcomes. I think this helps not just to make it clear that users shouldn't have an expectation of support, but also if a solo project it will end and flow.

I think that this kind of becoming a requirement because open -source is in a really peculiar place and these sorts of issues are likely going to get worse before they get better.

It isn't just the number of open-source projects that is growing, but the variety in models and desired outcomes for open source that is diversifying, and in many cases, more towards where any project with traction is less likely to be a fun side thing for a smart dev and more seen as a 'product'

I am not sure how I feel about this trend overall, but I do know I have appreciated this sort of context in projects and it does seem to at least make it easier to close issues by pointing at those statements.

I have been following oxide for a bit, and really don't have to add to the tech conversation, but do want to say:

Congrats to the team on reaching this big milestone and (in my eyes at least) just as much congratulations on doing it in a way that has been unique and sticking to values that seems to drive a strong positive culture (at least from the outside looking in).

Shipping products is hard, and only getting harder. IMHO, one of the big drivers of that is just how complex every market has became. Building and selling software alone is so much more multi-disciplinary than it was 10 years ago and adding hardware to the mix is upping that by a huge factor. As I look around, I see so many companies struggle to build teams that can handle the huge range of required tasks. To see a company like Oxide that (once again, from the outside a least) seems to have things together on so many fronts, especially while doing it while sticking strongly to some core values, is pretty inspiring.

Not to get overly cynical, but I don't think it is an extreme opinion to say that current start-up culture feels like you have to make big compromises in what you believe in order to be successful. Whether that be open-source, how you value and pay employees, or even just rushing things to deliver that aren't ready.

While I acknowledge Oxide has some well-connected, experienced founders that I am certain enabled them to get the resources and trust to do things their way, I really hope they kill it so that other founders and builders can learn that you still can build not just financially successful products, but great organizations that truly care about their values.

Congrats on the launch and interesting project! I have a lot of questions :)

* I am by no means a crdt expert, but from my tinkering I have come to the opinion that the upside of the consistency does come at the cost of ease of understanding how state ends up how it does, which can bleed into user facing issues if how a crdt converges conflicts with intuition. A lot of that seems like an education and design problem and I am curious how you are thinking of talking that? Do you plan on some support for server-first writes for situations where crdt might not fit?

* from the docs it seems like adding electric to a table does a migration to add new columns / new tables, I assume, to support the crdt, but my other knowledge of crdts is that it can be expensive in terms of size, especially for preserving history. I will have to poke at it to learn more, but I do wonder where you think you shouldn't enable electric?

* I am curious at the commercial model? Are you going full cloud service like Supabase/neon? Or just host the elixir component?

I will keep poking, but really really interesting!

Thanks for the answers Mike!

One more follow-up on the scale side (which I mentioned with sibling comment), it isn't so much about clickhouse itself, but about scaling up ingest. From my own experience and from talking with quite a few APM players (I previously worked in streaming space), a Kafka / durable log storage kind of becomes a requirement, so I was curious if you think at some point you need a log to further scale ingest.

For enterprise side, I was previously in data streaming space and had quite a few conversations with APM players and companies building their own observability platforms, happy to chat and share more if that would be useful!

Generally, any sort of async/batch inserts will get you decently far, but still will have limitations well before you get to million rows a second, mostly because it is really difficult to get your batch size large enough from individual producers without some sort of aggregation, which that aggregation is a challenge if you care about durability.

So often that means you need something like a Kafka to get the bulk ingest to really perform to get batch sizes large enough.

That kind of gets into one of the challenges of OSS observabilility systems, you don't want to make the dependencies insane for someone who only has a few thousand logs a second, but generally at some point of scale you do need more.

Wow, there is a lot here and what here is to a pretty impressive level of polish for how far along this is.

The background of someone with a DX background comes through! I will be looking into this a lot more.

Here are a few comments, notes, and questions:

* I like the focus on DX (especially compared to other OSS solutions) in your messaging here, and I think your hero messaging tells that story, but it isn't reinforced as much through the features/benefits section

* It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Or is the cloud backend a bit different where you can offer that scale without making open source so complex?

* I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features? Or are you considering functional things? Especially with recent HashiCorp "fun" I think more and more it is useful to be open about what you think the split will be. Obviously that will evolve, but I think that sort of transparency is useful if you really want to grow the OSS side

* on OSS, I was surprised to see MIT license. This is full featured enough and stand alone enough that AGPL (for server components) seems like a good middle ground. This also gives some options for potentially a license for an "enterprise" edition, as I am certain there is a market for a modern APM that can run all in a customer environment

* On that note, I am curious what your target persona and GTM plan is looking like? This space is a a bit tricky IMHO, because small teams have so many options at okay price points, but the enterprise is such a difficult beast in switching costs. This looks pretty PLG focused atm, and I think for a first release it is impressive, but I am curious to know if you have more you are thinking to differentiate yourself in a pretty crowded space.

Once again, really impressive what you have here and I will be checking it out more. If you have any more questions, happy to answer in thread or my email is in profile.

This will be... a "fun" comment section ;)

I will put my conclusion first: as an industry, there is so much room for real research and understanding in what does and doesn't work in reducing defects. However, this itself is a pretty difficult problem because of the difficultly in tracking, quantifying, or even defining what a "bug" is.

Until we have that data/research, any conversations about type systems vs testing vs process vs monitoring and saying which is "better" I think very rarely changes opinion and is more likely just to reinforce opinions held by different groups.

That isn't to say this article isn't useful, it is a good real-world perspective of things that do happen. I think the problem is that something about most of our brains (either in the writing or in the reading) causes a huge portion of us to immediately look the issue with such absolutes that most of the learning/discussion that could be had are drowned out through disagreement about the basics.

So in an attempt to try and hopefully start a discussion that goes beyond the argument about the value of types, I would love to learn about any research/products/ideas/success stories of how we as an industry can make software more resilient?

In my (limited) research over the years, it seems like a lot of the research is often limited to a single company (MS did a lot of measurement in the past) or looks more at the people aspect and less at the tools/technology.

I would think that LLMs might make some analysis, like measuring the impact of bugs, more tractable as I think an LLM could probably do an okay job of assessing the real impact of a bug (assuming there is a post mortem doc) better than just having a label like "major" that determines impact. That, combined with some okay understanding of the code that fixed a bug, could maybe help give a more complete understanding of how different tech/process/team structure impacts defect rates?

This is a pretty interesting article... and I generally agree with the pain points... but I don't really like the conclusion

* While the author states that not many apps "need" high concurrency in userspace... I would invert that and say that we may be missing so much performance, new potential applications, etc because highly concurrent code is so hard to get right. One bit of evidence of this (to me at least) is how often in my career I have had to scale things up due to memory or other resource limitations and not CPU. And when it is CPU, so often looking into it more finds bugs with concurrency that are the root cause or at least exacerbate the issue

* While I completely agree that rust is not easy with async and have myself poked around at which magical type things I need to do each time I have touched async rust code, I don't really like the suggestion being to "go use a different language", first, because if you are picking up rust, you (IMHO) should have a very good reason to already have chosen it. Rust is not easy enough or ubiquitous enough that you should be choosing it "just for fun" and your reason for using Rust should be compelling enough that you (right now) are willing to put in the effort to learn async when you need it

* What the other mentions in the body of the article, but I think is more of what my suggestion would be: don't use async unless you need it!. While I would love to see Rust (and think it should) evolve to the point where async is "easy", maybe we instead just need to get more pragmatic in what is taught and written about. I think when people start Rust they want to use all the fanciness, which includes async, and while some of that is just programmers, I think it is also how tutorials, docs, and general communication about a programming language happens where we show the breadth of capability, rather than the more realistic learning path, which leads people to feel like if they don't use async, they aren't doing it right

Finally, I do really hope Rust keeps working on the promise of these zero cost abstractions that can really simplify things... but if that doesn't work, I am at least hopeful of what people can build on top of the rust featureset/toolchain to help make things like async more realistic to be the default without the need for a complex VM/runtime.

Heh, it seems like this article could easy be called "hourly billing considered harmful" and I would basically have the same critiques of it as most "<popular tech> consider harmful" posts.

I won't pretend I have years of experience in the freelancing world... I am fairly new to it myself. But I do know as engineer who also has done some stints in product, sales, and business side, that, just like in software, there is no one "right way" and that one of the most consistent challenges business face is choosing the best of many reasonable options, especially when one of those choices is viewed as a "best practice" but the team lacks the direct experience to answer the question of "best practice for who and when?"

I fully agree with the author that there are some real market forces that can make hourly work not the ideal for many situations... but I also think it massively over simplifies the different types of work that are done as "freelancing".

Whole new products, new features in existing products, taking over projects, rescuing projects, consulting with less experienced teams, embedding in teams, integrations of external tools, is probably only 25% of the type of engagements a single consultant may come across and probably only 1% of the type of engagements that exist. Combine that with the huge range of attitudes of companies and very quickly we are in a space where no single solution exists.

It seems like the author's true point is in the conclusion, that you should focus on driving value and results for your customer and you should structure work to align with that, which is absolutely true. The general take of "hey engineer guy who isn't as business savvy, that value might not be best delivered via hourly billing!" is not bad advice. If you do hourly billing because it is the most straight forward, you should probably think about it... but you also shouldn't move to project based billing because of some people calling it a best practice either.

My advice to any engineer looking to freelance (but really to almost anyone everywhere in this industry) is to not take the easy way out when facing decisions outside your domain. I think so often work seen outside of code is seen as "lesser value", and because of that, shortcuts are made by just taking the easiest or "best practice" answer to a problem and running with it, and not gaining any understanding of the problem space. No one can be an expert in everything, but when you can't make this decision someone else's job, like when a business doesn't have some expertise (which in freelance is always the case) then you can't afford to not put in the time to learn enough to at least be informed. If you then do decide to do the easiest / best-practice thing, at least you know why you choose that option and can later evaluate how it is working for you.

In summary, there is no simple answer, be thoughtful and keep trying new things. What works for one gig may not work for another.

Re: Supabase and timescaledb.

Just want to make a bit more clear, Supabase has the ability to do some distribution via replication, but it isn't a true multi-master DB.

Timescaledb does support a multi-node config (https://docs.timescale.com/self-hosted/latest/multinode-time...) on top of postgres but that isn't in the open-source apache-licensed version, instead it is only in Timescales's community BSL version which isn't license compatible with supabase

And yeah, please don't hesitate to reach out in regards to OTel... lots of opportunity but also not as simple ;)

Congrats on the launch!

I have quite a few years of observability experience behind me and hand't really considered some of the unique aspects that LLMs bring into the picture. Here are a few thoughts, responses to your questions, and feedback items

* Generally, I think you do a good job of having a clear, concise story and value proposition that is fairly early in a market where the number of people hitting these problems is rapidly growing, which is a pretty nice place to be! But, I do think that can be a challenge in that you have to help people recognize the problem, which often means lots of content and lots of outreach.

* I think going open-source and following a PLG model of cloud/managed services is pretty reasonable way to go and certainly can be a leg up over the existing players, but I noticed in your pricing a note about enterprise support of self-hosting in customer VPC and dedicated instances. There is lots of money there... but it also can just be extremely big time sink for early stage teams, so I would be careful, or at least make sure you price it such that it supports hiring.

* Also on pricing, I wonder if doing this based on storage is how people would think about? Generally, I think about observability data in terms of events/sec first and then retention period. If you can make it work with a single usage based metric of storage, than that is great! but I would be concerned that 1) you aren't telling the user which plan can support throughput and 2) you could end up with some large variance in cost based on different usage patterns

* The biggest question I have is how much did you explore opentelemetry? Obviously, it is not as simple as just going and building your own API and SDK... but when I look at the capabilities, I could see opentelemetry being the underlying protocol with some thinner convenience wrappers on top. From your other comments, I understand that you see some ways in which this data is different than typical trace/observability data, but I do wonder if that choice will 1) scare off some companies that are already "all in" on otel and 2) you don't get any opportunity to use all of the stuff around otel, for example, Kafka integration if you someday need that.

* As far as your question about OLAP, I wouldn't rush it... In general, once you are big enough that the cost/scalability limitations of PG are looming, you will be a different company and know a lot more about the real requirements. I will also say that in all likelihood, ClickHouse is probably the right choice, but even knowing that, there are lots of different ways to tackle that problem (like using hosted vs self-managed) and the right way to do it will depend on usage patterns, cost structure, where you end up with enterprise dedicated / self-hosted, etc. I will mention though that timescaledb is not a bad way to maybe buy you a bit of headroom, but it is important to note that the timescaledb offered by supabase shouldn't be compared to timescaledb community / cloud. The supabase version isn't bad, it just isn't quite the same thing (i.e. no horizontal scalability)

Anyways, congrats again! It looks like you are off to a good start.

If you have any other questions for me, my email is in my profile.

Congrats on the initial launch!

Here a few thoughts, feedback, and questions:

* you do a good job in this post of describing why you need more than just ChatGPT to get acceptable quality, but much less of that is in the readme. I wouldn't be afraid to sell the project a bit harder, even in early days

* likewise, I think a small visual of the architecture would be helpful, just to make clear what the relationship is to chatGPT, and the additional features of a context store, etc

* Between the notes here and your product pages, it seems a strategy for commercialization with this repo being a lower level tool, with your commercial service having UI, simplified integration, and a hosted version to make this easier for semi-techical teams? If that is the case, I wouldn't be afraid to make that more explicit. GitHub is more and more a place for discovery, even to semi-techical people, but to do that well, I think focusing on a readme that makes it clear who the open source is for is important

* How are you planning on solving the data access problem? Is this a full SaaS service that will need access to the customer data directly somehow? Do you deploy this in the customer's environment? In thinking beyond this open source release and to the commercial side, that would be what I would want to know

* This point has been hit by others in this thread, but I would be curious to know what your plans are to help protect against valid but incorrect queries? I am not as convinced that this problem is insurmountable, as it does seem like you could build features to remove ambiguity by asking questions, show alternatives, etc that most semi-techical people could reason through, but ultimately it seems like thinking about how to involve the teams that own the data might be an important part of the problem.

Anyways, congrats again! This is an area I am really excited to see how it evolves (and will be exploring more!). My email is in my profile if you are interested in chatting more :)

Congrats on the launch!

Interesting project in a space that I am pretty certain is going to change a lot in the coming years. Here is a bit of random feedback and questions.

* Some of your messaging related to python vs yaml is a bit confusing, which results in me not being immediately clear on the value prop. After digging through docs and code I now understand that the yaml is a declarative pipeline calling the underlying python code that can include user defined transformations. Nifty! As someone who has led data platform teams, I understand that this would be a big win for any data platform team to better support data eng/scientists. But you don't tell me any of that. I would look at trying to give more context to what this is and adding more of these use cases and values in your marketing (even if they are pretty nascent at this stage)

* From the loom, the play you are doing is clear and makes a lot of sense to build a cloud service to easily run these jobs... but that makes me wonder if your licensing choice is maybe a bit too restrictive? IMHO, the most important thing to do when building dev tools is to be very deliberate in your end-to-end user -> customer journey and designing your open source and commercial strategies to nicely dovetail. For a product like this, I would think the faster and bigger I can build a community, the better, and that may mean "giving away" a lot of the initial core innovation, but with a clear plan on the innovation I can drive through integrated services, which would imply as open as a license as possible. As is, I think you might find it much harder to get people to take it serious, as, unlike other source available companies (Elastic, Cockroach, etc) you aren't yet proven to be worth the effort to get this approved vs a full open source alternative

* On a similar note, what is in the repo right now seems to be a relatively thin wrapper around spark. That isn't a criticism. Many technologies and communities have started based on a "remix" of a lower level tool that offers simplified UX/DX or big workflow improvements. What sets those apart though, imho, is to drastically lower the barrier to entry to using the underlying technology and to be seen as leaders and experts in the space you operate. I am guessing you probably have lots of features planned, but I would also give a soft suggestion to look as much into thinking of learnability as a feature (via features, interactive docs, etc) as I would almost anything else, as that is really where a lot of the value of a higher level interface like this comes in

* My past experience with really large and complex ETL jobs that essentially required dropping into spark to represent them has me wonder how much actual complexity can be represented by the transformers? I would be curious to know what your most complex pipeline is? It doesn't seem there is an API limitation why these pipelines couldn't get quite a bit larger and represent many sql statements, other than big long spark pipelines getting kind of ugly, and in some cases, could even remove the need for quite a few airflow jobs. I am curious to know if and how you see Serra addressing those sorts of problems like those types of ETL jobs.

Once again, congrats on launching! Happy to give more context/thoughts in a thread or reach out to me via in profile

Here are some questions, suggestions, and thoughts roughly categorized as I went through and understood a bit more about refine (the open source) and what I can glean from enterprise.

Landing Page / Initial impressions

* You mention "enterprise" a lot on the landing page, along with being open-source, but then you also call your commercial version "enterprise". It is a tad confusing delineating what is what. Having built and sold developer facing tools, *I* have enough context to understand that you mean is "this isn't a toy, you can actually use it in for-real production apps with for-real big company requirements even in open source", but I don't think you are doing yourself favors by calling your commercial tier enterprise, as it really confuses the message.

* The above is made a bit worse by the fact that you don't have any mention of a commercial product on your landing page, so when I got to pricing (usually my second page) and see enterprise, now I am not sure if the landing page is just for commercial project and open-source is just a toy.

* As far as I can tell, you don't have any marketing pages for the enterprise offering beyond the pricing page? I want to know how it differs. I want to click on the list of additional integrations. I want some docs. Anything you can give me besides contacting sales is going to help. I am not saying it is a paper launch / painted door, but it seems like one, and a thin one at that. You may have a really great product, but your GTM seems like it is lacking, and GTM for products built on open-source is super important, but generally underdeveloped by dev led teams.

* Your landing page doesn't really tell me who the target audience is, unless I have context of another product. I totally get a play that uses the momentum of another company, but I do think you lean on it a bit too hard here. I need a tagline like "Refine is an opinionated project builder for React developers designed to 10x your ability to stop fighting frameworks and start getting work done" or something to tell me who this is for without relying entirely on knowing retool. Similar, the enterprise product has really compelling features, but without more context on pricing, use-cases, etc, I don't know if it is for me.

* Your CTA and getting started is reasonable, but, as others mention here, it is giving the user a ton of choices that they may not have a clear idea of pros and cons. I suggest changing from a horizontal card arrangement to a vertical card where you have more space to provide some guidance.

* Animations and graphics for marketing pages is very subjective, but imo, the animation of the different domains of front-end apps (backend, react, auth, etc) along with the rotating list of "enterprise-ready" features is really busy and don't tell me much. I kept thinking that the firing concern would link with the enterprise ready feature, but it seems they were unrelated which just made it noisy.

Architecture / deployment model / commercial differentiation

* Are you doing the hosting directly for enterprise version and including the direct database access? If that is hosted in your cloud... that seems really tricky security wise. If you are hosting in the customer account that seems really tricky in terms of a wide range of clouds etc. I would be curious to know how this has gone so far.

* With enterprise and the direct database offerings, it seems like you have two choices of generating APIs that connect to DBs or are doing a more "direct" API that can execute SQL. Both have challenges. is there anything unique there on offer?

* The differentiation of features between open source and commercial seems reasonable... but I am concerned that because you don't have *any* ability to use any server side integration, it becomes such a different beast that you don't have an opportunity to get people to learn it. I would think that moving a "taste" of the server-side capabilities into open-source might be really helpful

Small Nits / Fixes

* The generated project has a lot of places with hardcoded configuration that I know I would need to change for my app... but no docs/comments/hints to tell me to do that? For example, using Auth0, I have a hardcoded (real) auth0 key and secret. I found it by chance. I personally wouldn't do that in a production app, as I want to centralize all config in one place. That seems like something I wouldn't be happy with

* Your docs are expansive... but could use some refinement. They are hard to navigate, especially the tutorial section, which doesn't appear to have any representation in the left nav? They also aren't navigable they way I expect docs to be (for example, https://refine.dev/docs/tutorial/getting-started/chakra-ui is not a page). I also was able to break the back button in the tutorial section.

Hopefully this is helpful, that was a lot of negative stuff, but I do want to close with overall feedback that this is a *really* hard problem and it is clear that there is a lot of potential value here.

If you want to get deeper into any of these questions, the email is in my profile, always happy to chat with companies building cool things :)

I should say... Open source can be a great way to get a community of users, with a subset that will opt not to self-host, but, I think there are a lot of things to consider:

* Can you shepherd a community, that will have issues with self-hosting, prs, bugs, etc AND do all the stuff needed to run a company (once again, assuming one person show here)

* By going open source, you now are also "competing" with other OSS projects, which may drive toward features that don't drive value on commercial side. You may also find that the core value prop just doesn't translate well to open source, for example, if the target is JAMstack teams and you can't run this on vercel, cloudflare, fly, etc

* I think the most successful open projects like this tend to be something that is a segment only really solved by proprietary companies. For example, workflow automation like Zapier built a novel model and led to open source with hosted version projects like n8n to replicate that, with many people offering that as a service inside their company. I think "open source MailChimp" isn't a bad play, but I don't know if that is really what you are doing, so it is hard to say if open sourcing would led to sales or to useful software that no one wants to pay for

Congrats on the launch!

Lots of really good stuff going on here, a few bits of feedback, ideas, etc, and also responding to some of the comments in other threads.

* The quick explanation of how this works is pretty strong, but I think the differentiator/value over other services maybe isn't called out as well. To me, this seems pretty clearly ideal for smaller teams all in on JAMstack/serverless with a lean stack that don't/won't have an easy place to create an automated process for synchronization, and are more likely to be using a planetscale/supabase/neon where this model is more attractive. I would suggest adding some of that info, not only to help target customers better recognize the value prop, but also help discovery via SEO.

* While the landing page gives a good overview, as a dev, before I sign up, I want more details of how it works, more complete examples, etc. For SaaS apps targeting the general market, after landing/product pages,the most visited pages are generally use cases, success stories, industry specific info, etc. But from my experience and talking with many others, for dev audience, docs are the next thing people visit (if they aren't ready to sign up). You don't need every doc at once, but things like a quick start guide, concept overview, feature overview, etc can all be high value docs that you generally need anyways for your first customers

* With docs, you can better address and explain the security side of things. I would do less on the landing page in regards to security and push that to more in depth detail in the docs. Talk about best practices of creating a read-only user. Create guides for most popular db vendors, etc

* As has already been mentioned, you will get concerns about any access to the db and "why not an API". I think these commenters are right that this will be a deal breaker for many companies, but I don't agree that you want an API. Not only because I think that reduces some of the value prop but also because then you would need to store customer data. I obviously don't know what you are storing today, but I would think that this model could have a big advantage if you didn't need to store any PII data in cc.dev. The complexity of dealing with compliance and regulatory requirements is no small part of why something like MailChimp is expensive is that burden. I don't know if this is something you currently consider a value prop (or if you have engineered to support this)... But I certainly would :)

* To address the reality of private dbs/giving access to a db, I would look at potentially implementing an agent. This agent would then run the queries (still provided by the user) and just-in-time deliver the results when needed. Getting the model of this right that works across different companies view of security will mean this is probably best to do with a handful of potential customers.

* As far as deployment and monetization model... I would only open source it if you are giving up on commercialization or if you can open source a subset that can be useful enough to build a community. Once again, returning to JAMstack, maybe solve in open source (but integrate) some problem unique to those teams. As far as a paid self-hosting, given that this seems like a one man show, I would resist it. Trying to do SaaS and support of self-hosting is rough, even for well funded VC backed teams..

Quite a bit here, but if you want follows up on anything, my email is in profile or will reply to comments.

Congrats again and hope it goes somewhere :)

What an interesting and timely article.

My extended family recently opened a pizza place which just happened to be at the same time as me wanting to take a break from startup grind. I worked at a pizza place in high school, and have continued to make pizza at home, and loved the idea of being able to combine that knowledge with the experience of building software startups, so I have been heavily involved in the last 6 weeks on everything from ordering, to kitchen layout, to POS and other IT systems, to marketing.

In short, the experience has been super interesting, not only in terms getting to learn tons of new things and get familiar with how the tech side is done (spoiler alert, not that well) but also just generally how much of an opportunity there is to take the sort of thinking and skills required to build tech products and to apply them to a quick-serve restaurant (and I would guess lots of other small business).

I have been planning on writing something up, but this article is giving me good motivation to actually do it :)

I am going to write this comment with a large preface: I don't think it is ever helpful to be an absolutist. For every best-practice/"right way" to do things, there are circumstances when doing it another way makes sense. That can be a ton of reasons for that, be it technical, money/time, etc. The best engineering teams aren't those that just blindly follow what others say is a best practice but understand the options and make an informed choice. None of the following comment is at all commentary on questDB, as they mention in the article, many databases use similar tools.

With that said, after reading the first paragraph I immediately searched the article for "mmap" and had a good sense of where the rest of this was going. Put simply, it is just really hard to consider what the OS is going to do in all situations when using mmap. Based on my experience, I would guess that a ton of people reading this comment have hit issues that, I would argue, is due to using mmap. (Particularly looking at you prometheus).

All things told, this is a pretty innocuous incident of mmap causing problems, but I would encourage any aspiring DB engineers to read https://db.cs.cmu.edu/mmap-cidr2022 as it gives a great overview of the range of problems that can occur when using mmap

I think some would argue that mmap is "fine" for append only workloads (and is certainly more reasonable compared to a DB with arbitrary updates) but even here, lots of factors like metadata, scaling number of tables, etc will eventually bring you to hit some fundamental problems when using mmap.

The interesting opportunity in my mind, especially with improvements in async IO (both at FS level and in tools like rust), is to build higher level abstractions that bring the "simplicity" of mmap, but with more purpose-built semantics ideal for databases.

This is a great resource.

The PG docs are generally quite good, but logical replication is one of those things where experience and this sort of resource is invaluable.

That said, I also feel like logical replication is still pretty hard to use. One example, not covered here, is that the status of slots is not replicated to any secondaries. There are solutions for this now (see https://www.percona.com/blog/how-patroni-addresses-the-probl...) but generally the domain of dealing with logical changes is still one where I feel like MySQL binlog has a leg up (though still some sharp edges) in making it a bit easier to work with.

I have spent a lot of time looking in this space recently for helping to revamp documentation and I really really have fallen in love with Markdoc.

Markdoc just hits the sweet spot of being super easy to get started with but elegantly extensible that makes it scale. I think the OP here simplifies a bit though of what Markdoc is. While it is pretty simple to integrate into a next.js site for a SSG doc site, it is more of a library that can be integrated into almost any site or rendering framework.

In some ways, this is the biggest "challenge" of Markdoc right now. It isn't focused on a polished out-of-the-box experience like Docusaurus or MKDocs, but is instead more of a DIY tool.

That said though, what is there is really great. With the ability to create custom tags easily and then the ability to analyze and transform an AST in a simple, but easy to understand way, I think markdoc is actually a great option for more than just building a doc site, but as a more general purpose tool for authoring any text-heavy content.

With Markdoc, I have built: * a higher level utility for creating a "library" of content with consistent ids for stable and validated links * a validation library to ensure that doc structures follows best practices like having metadata tags in the frontmatter, properly nests headers and doesn't skip H3s, etc * an integration for authoring and reusing doc content in spectacle[0] presentations * have a clear direction of how to "scale" docs-as-code as we were struggling to do that with a simple, flat file of markdown files

I have started to toy with the idea of a more general purpose CMS built around markdoc... but in general, a really great tool and kudos to stripe team for building it :)

0 - https://formidable.com/open-source/spectacle/

StreamNative | Full Time | Engineering, Product | SF, LA, SLC, Remote | https://streamnative.io

StreamNative is the company founded by the original developers of Apache Pulsar. We build a fully managed version of Pulsar and are helping companies build high scale mission critical apps with our robust messaging and streaming API. We are seeing accelerating adoption of Pulsar and strong revenue growth.

I am the Chief Architect/ Head of Product and joined the company 2 years ago after getting involved in the community and being an early Pulsar adopter.

We are hiring across product and engineering, with the following roles: * Director of Product Management (lead our product management team across all open source and commercial, looking for team builder with technical background and data infrastructure experience) * Product Manager, both open-source and cloud (prefer engineer background and/or experience in data infrastructure) * Cloud Architect (work on our k8s based cloud and integrating Puls of stateful distributed systems) * Platform Engineer (working on open source Pulsar) * Senior Full-stack cloud engineer (working on the apps and service that front our cluster management API)

You can find more info at https://StreamNative.io/careers or you can ping me at addison at streamnative.io.

Interesting, the separate compute and storage tiers is another system going that direction which I think is becoming almost the standard at this point, especially for "cloud-native" things designed to run on k8s. From what I can tell (it isn't very explicit on this point) they are avoiding a distributed consensus at the storage layer and instead relying on a single writer/multiple reader model with the single writer being enforced by assignment of the tablets in the compute tier, with the tablet being responsible for writing to multiple storage nodes for durability? (But I might be wrong)

Assuming yes this approach, I think, is under utilized and is pretty similar to how Apache Pulsar works (my day job),but I am not sure how many distributed RDBMS have tried it out, will be cool to see how it evolves! It isn't clear how they ensure the assignment of a tablet to only a single compute node, but I think that is an easier problem relative to distributed consensus at the storage tier.

I think this might be an error in the text, HTTP status code 503 can be used for backpressure, which I think is what is being referred too

But yes, TCP uses windows for back-pressure, but that isn't really useful for application level backpressure as the OS controls the queues sizes, so pretty much most systems have their own backpressure on top.

Pretty clever. It is pretty neat how much a small team can do by intelligently "remixing" the powerful primitives that now exist at the linux and VM/container layers.

The cool bit is how many interesting things are here/arriving like bpf, io_uring, and wireguard in linux and with wasm and all of the interesting things it opens up with fast, sandboxed execution. I fully expect from some of these techs will really come innovation that will shake-up infrastructure even more than cloud and kubernetes already have.

(Also, hi Kurt and team, cool stuff!)

StreamNative | Remote, Full-Time | https://streamnative.io

StreamNative are the main developers behind Apache Pulsar. We are looking for multiple different roles to join our team to help us build and maintain our rapidly growing hosted Apache Pulsar offerings. We are VC backed and seeing strong revenue growth. We are hiring across the company, for example:

* Engineering - Looking for engineers to work on our cloud product, open-source Pulsar, or integrations

* Customer Success - Technical Account Managers, Customer Support Engineer, and Solutions Architects

* Product - Technical Product Manager and UI/UX

* Community - Community Managers, Evangelists/DevRel

See https://streamnative.io/careers/ for more details, but not all roles are listed there as we are just opening the reqs. We like diverse backgrounds, so even if you aren't squarely in one of these roles and are interested in the streaming space, please reach out :)

Tech:

* Java, Go

* Kubernetes

* Apache Pulsar, Flink

* Lots of Cloud (AWS, GCP, and Azure)

Reach out at addison [at] streamnative [dot] io

StreamNative | Remote, Full-Time | https://streamnative.io

StreamNative are the main developers behind Apache Pulsar. We are looking for multiple different roles to join our team to help us build and maintain our rapidly growing hosted Apache Pulsar offerings. We are VC backed and seeing strong revenue growth.

* Cloud SRE - https://streamnative.io/careers/cloud-site-reliability-engin...

* Cloud SWE - https://streamnative.io/careers/cloud-product-engineer

* Platform SWE - https://streamnative.io/careers/platform-engineer

* Solutions Architect - https://streamnative.io/careers/solutions-architect

* Community Manager - https://streamnative.io/careers/community-manager

If you don't fit cleanly into any of these roles but are interested, please reach out! We like diverse backgrounds :)

Tech:

* Java, Go

* Kubernetes

* Apache Pulsar, Flink

* Lots of Cloud (AWS, GCP, and Azure)

Reach out at addison [at] streamnative [dot] io

Hey, I work on Pulsar, will try and answer this :)

Topics (actually bundles of topics, called bundles) are what is assigned to Brokers. Topic assignment is dynamic, so when a new broker is added, the system will try and shed load from the busiest brokers to even it out on the system.

But unlike Kafka, when a topic is assigned to a broker, it doesn't have much state to move, mostly it just gets metadata added to it and opens a new "ledger" (which is just a chunk of the topics data over a time window, only one ledger is ever open at once). When it needs to serve data, it pulls that from bookkeeper nodes from previous ledgers, so the process of re-distributing load is pretty quick, it also doesn't eagerly pull in a cache.

Now, as far as the cache, that is primarily for "tailing reads", meaning, as writes occurs, and clients who are close to the tip of the recent data will just get it from the broker, without a need to pull it from bookkeeper. This is is one of the key parts about how Pulsar has multiple tiers of storage that help it have such good consistent latency.

Beyond processing writes, the biggest thing brokers do is handling "tailing reads" i.e., clients are consuming right near the tip of the topic. , this is the cache referred to. That means that when a new pbroker is three purposes:

1. Handling writes

This is definitely an area where Pulasr is trying to improve, getting started is not easy. That said, the progress I have seen in the last year since I have been involved with it is really promising.