nickmonad

  Location: Midwest, USA
  Remote: Yes, preferred.
  Willing to relocate: For the right position and location!
  Technologies: Golang, Zig, Rust, C, Python, SQL, cloud infrastructure, observability tools (Prometheus, OpenTelemetry, etc)
  Résumé/CV: https://nickmonad.blog/about - Can provide full details on request!
  Email: my username @pm.me

While it may not actually matter for liability, it must be stated if labs are going to attempt avoiding penalties by hinting “oops we created a super intelligence we don’t understand, nothing we can do!” Repeatedly stating the truth must continue, especially to remind those who aren’t technologists.

Once you have that pool of "objects" that can be recycled throughout the lifetime of the program, you have a guarantee that actual allocation can only be interpreted in a specific way, i.e. all objects have the same size, alignment, etc so you don't have nearly the same level of concern or detail of implementation as an actual allocator in the common understanding of the word. A simple free-list gets you pretty far.

As it stands today, we have an AWS-only product (for better or worse) so we can accomplish this already with CloudFormation. It's not the best "control plane" as far as updates and rollbacks go, but it works pretty well for our needs. Obviously, trying to do the same across multiple cloud providers, would mean bringing in a different tool. At that point, the vendor's architecture would need to be neutral as well.

> I am not sure if simplifying the application removes the need for this tooling as much of the complexity lies with establishing the enterprise trust boundary across accounts with things like identity, networking, permissions, artifact approval, upgrades, drift, auditability, governance controls, and eventually exit. This website tries to cover it in detail: https://byocanywhere.org/, WDYT?

I think that website is a solid resource. Those are certainly important issues to solve that all surround the actual vendor code itself, and I can see value in making an attempt to solve it. It's just my experience that every customer wants to do it differently. I'm just not sure how much slower and painful our contract negotiation would have been if we had to also sell them on a particular process other than "you provision these things with your own tools, and we'll hand you CloudFormation templates". Maybe other companies offering BYOC are better at selling the whole package, not just the "platform" itself.

This space is quickly becoming a bit saturated, as these "BYOC enablement" tools and platforms are starting to crop up, and I can see the value in them, but my experience doesn't always line up with it. I'm actually really interested in hearing about a large enterprise running vendor code that would even entertain the idea the vendor could automatically update the entire deployment without their direct involvement. Our platform is quite load bearing for our largest customers, and in some cases directly in their revenue stream. There is no way they would ever let us push updates without going through their own internal controls of validation, testing, and sign-off. One could definitely argue these controls aren't entirely founded in reality, but corporate reality isn't always the reality you and I might share.

Besides the issue of automatic updates, we struggle with wild configuration differences that need pretty direct involvement with the customer. Security groups in AWS are just one example of this: some customers allow all internal VPC traffic between services, and others are quite strict in that every service needs a perfectly scoped security group with inbound/outbound rules. These have to be provisioned with their own tools built by their own teams under their own version control. We just ask for a configuration file in a well known location that defines these IDs we can reference in the CloudFormation.

So knowing that we essentially have to operate in the self-hosted model, CloudFormation becomes our mechanism for defining consistent deployments. For us to switch over to something like k8s just to fit into the model a BYOC-enablement tool like this wouldn't be worth it.

I think the most important aspect here though, and the most subtle one, is the issue of deployment and operational complexity. For BYOC to work well, the vendor really should be designing for it from the beginning. Taking an existing multi-tenant SaaS architecture and shoving it into this model, even with the ideal BYOC vendor-maintains-control-plane scenario is no small task. Even when you're talking about large enterprise BYOC deployments, you may only have a few thousand daily active users, which is dramatically less than what most SaaS vendors design for when operating in their own multi-tenant environments. ("Web scale" and all that.) At that point, I think engineering teams need to take a serious look at their architecture, and realize just how far they can get with fewer services, fewer machines, and fewer "event driven" architectural decisions. The deployment model _should_ be a forcing function for simpler architecture and system design, but I realize that's not a given. I think what I'm trying to say is that, knowing BYOC done well requires designing for it from the beginning, a well understood daily load profile and operational context kind of eliminates (or at least dramatically reduces) the need for tools like this, and that self-hosting isn't the worst possible outcome.

> Bounds checks, checked arithmetic, strongly-typed alignment, strongly-typed error codes, tagged unions, explicit undefined are some rather important language features. If Zig didn't exist, TigerBeetle would probably have been written in C, and the safety gap between C and Zig is just gigantic. And safety is one aspect of the language, Zig has a lot of going on for it elsewhere!

[1] https://lobste.rs/s/6rkdik/rewriting_bun_rust#c_8gebpa

> "There's a dichotomy being presented here where you have to either choose a "style guide" or a programming language feature in order to avoid bugs. The sleight of hand misdirects the reader away from the main way bugs are eliminated: by dedicating engineering resources to it. You're not giving TigerBeetle nearly enough credit. Quite simply they put in the time to find and eliminate the bugs, they make an effort to maintain a healthy relationship with ZSF, and Bun did not do that."

The reference to TigerBeetle is important, and a bit under-explained for the point he's trying to make. They have consistently attributed things like design and deterministic simulation testing to their success and reliability, which has nothing to do with Zig as a language. Some things might be _easier_ in Zig, such as static memory allocation, but ultimately, a holistic approach brings success, not one individual tool used "right".

Sometimes its just right time, right place. But also, Zig has received attention via projects like Ghostty, TigerBeetle, and Bun (prior to rewrite of course)

This is very real.

I work with a deployment that operates in this fashion. Although unfortunately, we can't maintain _any_ connection back to our servers. Pull or push, doesn't matter.

The goal right now is to build out tooling to export logs and telemetry data from an environment, such that a customer could trigger that export on our request, or (ideally) as part of the support ticketing process. Then our engineers can analyze async. This can be a ton of data though, so we're trying to figure out what to compress and how. We also have the challenge of figuring out how to scrub logs of any potentially sensitive information. Even IDs, file names, etc that only matter to customers.

https://docs.tigerbeetle.com/operating/cdc/

Yes, but the important distinction is that its intention is to bring people together face-to-face, not isolate them to their desks for continued work. Just because the end goal is "productivity" broadly speaking, doesn't mean the mechanisms are socially/morally equivalent.

> I doubt anyone is forcing the employees to take the stimulants.

I agree, and I hope my comment didn't imply I thought that was the case.

Trying to blog more frequently with shorter posts!