Hacker News Reader

Comment by irskep | Hacker News Reader

This sentiment pops up regularly on HN, and I've seen at least one article per month for the past few months, but the trouble is, none of them seem to help you actually deploy it. They assume you're comfortable spinning up public web servers.

If you want to use a PaaS to deploy an app, because you don't want to spend your time learning to be a sysadmin, then all the tutorials are going to put you on the Postgres path, because that's what's supported. (Of course, you'll then end up paying $15+/mo for Postgres, which is hilarious for most hobby projects storing 50MB of data.) But in reality, you could just scale vertically on one machine and be completely fine. No need for "distributed" anything, in theory.

I took a shot at productionizing SQLite here as an experiment: https://cheapo.onrender.com/ But I'm not sure I did it right, because I don't have much experience working below the level of a PaaS. I'm an application developer and I do not want to become a release engineer. I resent even having to learn Docker. :-)

Anyway, if somebody wants to nitpick my Flask+SQLite deployment, I'd really appreciate it, because it seems really silly that people have to keep writing this stuff from scratch when 90% of hobby sites have the exact same needs. And the Fly.io/Render configs would apply just as well to Node, Ruby, etc. https://github.com/irskep/cheapo_website

Edit: Somebody got mad at the Docker bit. I tossed it in for effect, but I promise you I don't hate learning new skills. I wish people would recognize that weekend coding projects should be fun, and sometimes that means avoiding certain kinds of things that a person experiences as difficult or frustrating. Arguing on the internet sucks, who knew?

View on HN

> I'm an application developer and I do not want to become a release engineer. I resent even having to learn Docker. :-)

Sorry, but that's a terrible attitude to have. You don't need to be a full-blown sysadmin to know how to do basic deployments, and learning these things will make you a better developer.

I don't know why this is downvoted, it's probably the best advice on this page. Learning a few things about deployment definitely helps to make better software.

Well, he did cherry-pick one joking comment out of a well-intentioned post in order to say I have a bad attitude, so that's not exactly welcoming. And very patronizing, considering I have actually done that kind of work with Salt Stack a few years ago, and just don't want to bother with it at the moment because I have other priorities.

I learn new things every day, and one day, Docker was that thing. I am now better at release engineering, cool! But I also lost a day of working on shipping value to users and actually having fun, which is less cool. That's why PaaS has been helpful to certain kinds of people; it makes the curve of progressive disclosure necessary to deploy stuff a little flatter. My hope is that we can make more kinds of things really easy to do the right way, and not yell at people who just want to have some fun.

Sure, that comment came a bit rough. But I'm not even sure it was triggered by the Docker comment, rather than the developer/release engineer opposition. At least that's how I see it. And maybe I just don't see the added value of PaaS because I've been spinning up public web servers for too long now ;-)

Yeah, that's fair, and it was bad phrasing on my part. The engineer type distinctions are completely imaginary. It just really threw me off to have people respond to me sharing something I thought was helpful with claims that I'm somehow opposed to learning, when this project is nothing but learning.

I do think people who have experience deploying software have a blind spot here, just like I have a blind spot for people who have never learned how to center a div. jUsT uSe FlExBoX

Hey there, I've never deployed anything but will be going to in the (very) near future. I'm self-taught and really haven't fooled around with servers, ever. So instead of just going down the PaaS way I was hoping to kind of turn my computer/raspberry pie into a server as a fun learning project.

Any cool, beginner friendly resources you can share? Many thanks

Agreed. Too much reliance on AWS, Heroku etc. can leave you without vital Linux skills. Linux cli tools are not just for getting a job done, though they're great at what they do. No, they're part of the *nix culture which is what open source software is based on. Only recently have I become aware that it was better to have entered software development in or before the early 2000s, ie. before The Cloud took off. You were forced to learn Linux/BSD cli tools just to swim in the water. I can't imagine, back then, knowing Perl and not knowing Linux or FreeBSD.

I used slackware linux. I wrote assembly code when I was a teenager. I use 'perl -i -pe' constantly. I still don't want to learn a bunch of arcane flags from a bunch of tools with a million gotchas. Sorry. I'd rather focus on the code craft that gives me joy these days.

Sorry, I detest this attitude of: "Well I was hazed so why aren't you hazed too?"

Did I mention that Russ Cox and I used to write C code with pen and paper? https://news.ycombinator.com/item?id=32874759 If you haven't, are you even qualified to discuss what vital skills others are missing?

Or maybe because I learned perl because it was simpler than tr and awk, I should STFU? Give me a break.

> I used slackware linux. I wrote assembly code when I was a teenager.

Lol are you me?

I grew up with Slack/assembler/softice/etc and I used to host everything on Slack boxes on Linode (bless them for having a pre-built image). But I left it a while ago because I got sick of dealing with hand upgrades and compiling everything from scratch. I feel the same way about other things. I could probably out-CLI most of the people here saying "you should learn linux!" but sometimes I just need to get something done.

If I'm trying to get something out the door, I don't give a rat's ass about the details. It's fun and interesting when you're a teenager and it's 4am on a summer night and you've got nothing better to do, but these days I have strict time, limited energy, and I'm not going to spend more time fiddling with things than I need to.

I mean, you can just put it into automation and never have to repeat same linux cmdline dance to do the same thing again. There are plenty of configuration management tools to do so. Hell, I outright forgot some skills because I encoded whatever I will ever need from a given tool in Puppet manifest and didn't had a need to touch the internals again.

Docker, k8s and friends are essentially just level above the "just CM", still just use basic linux primitives but with some coordination.

The "new toys" are essentially abstractions on what sysadmin would do (anyone remember building chroots ? Yeah I don't liked it either, even if it felt "cool" the first time I did it) and every abstraction will leak which means once you are big and complex enough you will need to debug it. Just that random hobby project probably won't get there and even for work it probably might be someone's else job to debug that

> I should STFU? Give me a break

...you probably should, your whining and bragging doesn't add anything to discussion. "Oh I played with cool kids few decades ago, look at me Mr. Important"

I didn't spend six years on a PhD to be called Mr. Important.

I'm joking.

Okay, bringing this conversation back to a normal tone: I have specific needs and want to get specific things done. I agree everyone should learn basic unix. I also think people should get as low to the metal as possible so they understand performance.

But can you acknowledge that there is a near infinite number of things people should learn, and that the priority of that list might be different for other people? That for some, and their technical goals, they might not have the time to prioritize multi-server web app dev when they're focusing on figuring out how to get GPUs to do DDP correctly? I never learned k8s because I all my web apps have super low usage. e.g. data annotation frameworks. I don't know JS and I guess maybe I'd love to, but I'd also like to wait until the field stops moving so correctly so that I could just pick a great framework and not have to relearn everything constantly. HTMX is cool and gives me joy.

Can you acknowledge that building tooling that allow devs to focus on what gives them joy is actually a nice thing? We have enough problems in the work we like that we end up spending endless hours debugging. If someone refuses to learn what they need to solve their main technical pursuits, yeah that's a moral failing, but nonetheless there's a finite surface area we can cover and I prefer to focus on learning the things that come up in the line of duty.

p.s. what are you working on? Maybe we can play together. I am not being snarky. I'm collaborative and maybe you're cool too. :)

>I don't know JS and I guess maybe I'd love to, but I'd also like to wait until the field stops moving so correctly so that I could just pick a great framework and not have to relearn everything constantly.

/js rant start

I think when that happens it won't be because JS stopped moving, it will be because people will just go "fuck it" and replace it wholly with <favourite language> + WASM.

At least that's how I managed how to not learn JS properly for years. Web frontend work is just drudgery, and I only did it few times when we decided trying to explain a frontend developer ops stuff to make admin page out of is more complex that stitching some shitty code together.

And every single time it was miserable experience and I think only thing that I made and is still in supported framework is fucking jQuery, because every other one seems to shit on backward compatibility and decide to just make new framework. Then call it same name but just increase the number after the name.

/js rant end

> Can you acknowledge that building tooling that allow devs to focus on what gives them joy is actually a nice thing?

I did just say that yes we have those tools and it is fine that dev might not care about internals, I just want to highlight that the whole "I don't care what's underneath as long as it works" might eventually bite, so doing the boring part of understanding something about underlying system might save a lot of effort going forward.

Especially if that bit extra effort allows for overall simpler architecture, as easier debugging usually comes with it. "Just look in logs" or see what process is doing in system directly using installed tools is infinitely easier than fucking with docker/k8s CLI, sidecars and other methods to observe the app in container.

Recent example: A bunch of devs in company we provide infrastructure for got starry eyed for k8s and we just recently had to re-explain that no, the "just give us big POSIX filesystem mounted everywhere" on dozens of nodes isn't gonna work well, and the fact you can tell k8s "give me storage with ReadWriteMany" won't just work. That was after they tried to make shared block device and run XFS on it, which fell apart for obvious reasons

That after same company had year+ long migration from "a big GFS2 volume that was slow as fuck because GFS2 is slow as fuck when you have dozens of nodes for it" to plain S3.

One meeting later and they figured out they don't even need shared storage in the first place, but nobody researched that before and their solution "worked" on test 1 node k8s cluster (for obvious reasons). While a bit of research would have saved the whole ordeal

So less communicative company might just go "right on, we will set up CephFS for you and go ahead" then get more maintenance and the inevitable "this clustered filesystem doesn't work EXACTLY like my XFS partition on ubuntu" problems. It would probably be better for billable hours tho...

More low-scale example: My VPS just have few systemd services for few apps I put there. unattended-upgrades and it is near-zero maintenance for years. Granted, I knew how to do it because that's my day job but I have 100x less complex setup on my VPS. Maybe I should write blog about it...

> p.s. what are you working on? Maybe we can play together. I am not being snarky. I'm collaborative and maybe you're cool too. :)

Well, pandemic/nearby war kinda fucked up every project I started

I was working on midi2sid chip eurorack module. Which was delayed for year+ just waiting to get parts. I think they might be available again, maybe I should resurrect it. It did ran in rust and actually played notes, I just ran out of pins on the MCU I used so I had nice big board with bigger MCU and some niceties added, made it ready to be assembled... right as the chipaggedon started.

I was also working on car datalogger, even got prototype working, except recent economic fuckery put any track days on hold. The plan was to finally get a house, the inflation put any sensible mortgage out of question for at least nearby future so all the fun money are going to the savings now.

The other plan was to replace car's radio with something newer and integrate that with it (via dumb "just send rPi HDMI image thru the infotainment screen" probably), but yeah, can't really do that without garage in the first place...

I experimented with home automation a bit and had some ideas (involving embedded Rust, it is surprisingly palatable) but, well, again, the plans involved having a home, and not really that much possible to do in rented apartment

In meantime like 30 projects that scratch itch nobody else but me have and were not touched after scratching part started to work fine.

I did decide to implement the CPU from The Art of Computer Programming. But it looked annoyingly weird so I just made Go Z80 "emulator". Then rewrote it in Rust as exercise (both in ImGui as frontend). Then realized that getting the other chips than Z80 "right" is a ton of work and.... not that fun work to boot so I called it learning experience for Rust and ImGui and left it alone. It did run enough assembly to do something tho. No longer runs after Rust version upgrade tho. So much for backward compat...

But currently nothing really interesting. I have those cycles, where if work is interesting I spend a most of free time in brainless/semi-brainless fun bracket, and if work is boring I do the interesting hobby stuff, and, well, recent company buyout and some other changes made sure I'm having all the "fun" at work I can take so I haven't been touching hobbies much.

That often quickly leads to "chmod -R 777 makes the app work and I don't give a shit about doing it properly" if your passion is dev, not ops

>learning these things will make you a better developer.

Of webapps. Will make you a better developer of webapps. It may surprise you to learn that there are more kinds of software than that. Sometimes the final product is something like a binary that runs outside the browser, and deployment is no more complex that putting an tar.gz'd executable on a website.

This whole conversation is about using SQLite on web servers, so I think it's reasonable for OP to assume that's what we're talking about.

I'm not sure what that has to do with sqlite. I'm very much a non-web-developer, I never use Docker and "deployments" and stuff, but sqlite is absolutely trivial to install and run, and use for non-webapp stuff? Even operating system components use it.

That's my point. Everyone is assuming web app.

    You don't need to be a full-blown sysadmin to 
    know how to do basic deployments, and learning 
    these things will make you a better developer.

Yes. True. However, it is also true that we have a limited amount of time per week and a limited number of weeks on Earth. Time spent learning sysadmin-y stuff is less time spent mastering developer-y stuff. Think about what it means to be a "full stack" engineer in 2023:

  - Unix literacy including common tools (awk, sed, grep, etc)
  - Docker, etc.
  - Relational data stores
  - Key/value and document stores
  - Base frontend technologies (HTML, CSS)
  - Frontend frameworks and the associated language
  - Build pipelines to tie it all together

I've never worked with anybody who's deeply knowledgeable at all of them, or even most of them. I would submit that it is practically impossible to master all of them while also holding a full-time job and writing and shipping production code, and staying "current" on all of the above, unless you're willing to go way past 40-hour weeks.

I've grown extremely disenchanted with "modern" software development. Your typical full stack engineer is frankly bad at most of the damn stack and they're not great at any of it. Nobody masters anything anymore. You get people with IQs three standard deviations above normal thinking they need to spin up an armada of AWS crap just to render "Hello World" in 2023. And they still manage to make a mess of it. I can't tell how much of this is brain damage and how much of it is a sociopathic commitment to resume-driven development.

In practicality, I'm a fan of "T-shaped" engineers. Master a piece or two of the stack, and have light knowledge of the adjacent layers and what they do. e.g. If you are an application-level coder you should know what Docket/K8s/etc are and when/why you might use it and when you should eschew them. And honestly I think it should be eschewed fairly often.

The places you can go with a bit of good old vertical scaling are pretty impressive in 2023. A single box with e.g. 24 cores, 64GB of RAM, and fast solid-state storage is a god damn supercomputer when it comes to spitting out web pages. Think about how far Stack Overflow got with that model and at which point you might start to outgrow them.

---

edit: By "mastery" I mean "fairly comprehensive competency" and not "be a recognized global authority." You can fluently use 50-75% of what the tool offers and you know what the other 25%-50% is so you can learn it and utilize it when needed. You can ship code using this tool that is maintainable and performant. For example, perhaps you've never utilized partitioning in Postgres but you know what it is and you understand the appropriate use case for it so you can learn employ it when needed.

Unless you're one of the very few known as top in a field, mastery of a specific technology adds very little value and is always in danger of becoming more legacy than relevant. That's in comparison to someone with enough knowledge to assist in or handle any phase of getting a product into the the hands of customers and making money. Most things don't need master level work or knowledge to be successfully implemented.

    mastery of a specific technology adds very 
    little value

I disagree in the strongest possible terms but it's certainly possible that we're working with different definitions of "mastery" so I'll give you mine. My version is perhaps actually closer to "competency" rather than "deep under-the-hood knowledge." I'm talking about somebody who knows how to use a screwdriver without stabbing their own eyeballs out, and when to use a screwdriver, but isn't necessarily like... inventing new types of screwdrivers or innovating in the field.

Somebody with Postgres "mastery" should have a strong command of basic normalized table structures, they should know how to construct useful indexes, they should know the downsides of over-indexing, they should know how to find and optimize slow queries, and they should at least know of somewhat advanced topics such as partitioning, materialized views, foreign data wrappers, and so on. It is okay not to have a clue how to set up e.g. partitioning but you should know what it is so that you can learn about it and employ it when needed.

Somebody with "mastery" of an application framework such as Rails should be competent at basic OO design, they should be familiar with Ruby structures and idioms, they should understand what goes where in Rails' MVC paradigm, they should be able to write comprehensive and performant tests, and they should know how Rails' asset pipeline builds and emits front end code. I would not expect them to have mastery of Ruby's metaprogramming, but I would expect them to know what it is and have some idea of how Rails uses it to extend Ruby. (I realize Rails is controversial, but I'm not talking about anything Rails-specific. Feel free to substitute your language+framework of choice)

In short, the sort of working knowledge you might commonly gain with 1-3 years of real-world production experience with a given piece of the stack. Or perhaps less if you were really focused on it and just learning it full time.

Without guidance, folks with less knowledge than this tend to produce absolutely unscalable, unmaintainable spaghetti code for anything larger than a toy project. To return to the screwdriver analogy, they can kind of use a screwdriver but they're constantly stripping screwdriver heads at which point either the project is ruined or they need some help from more practiced craftsperson.

    always in danger of becoming more legacy than relevant

I would certainly agree that you can go too deep to be useful as a "full-stack" developer. For example, if you really understand Postgres internals and can write your own extensions and have a bunch of commits in Postgres itself, cool, but what happens when our next client wants us to work on their MySQL-based app? And while Postgres is unlikely to die anytime soon, what happens if you've achieved that level of expertise in a tool that does fall by the wayside? I've certainly "mastered" a few tools that are distant memories.

I really appreciate you taking the time to respond with these details. I do go deep on application-level tech (CSS, HTML, JS with and without frameworks, iOS dev, etc) and it's always disheartening for people to claim that choosing not to focus on specific technologies makes you a worse human being.

The other thing that's going completely over people's heads here is that I made this repo with the intent of helping other people who know less than me. I know I can figure out all the deployment bullshit, but I mentor folks who can't, and also don't have much money, and yet still need to deploy their portfolio projects to the Internet in order to find jobs. I was trying to find a balance of ease-of-maintenance vs monetary cost. If there's an even simpler solution with an even lower cost, that a random bootcamp grad could maintain, I'd adopt it in my template in a heartbeat.

If any of these people claiming it's easy want to work with me on a reboot of the "deploy a web site easy and cheap" template, I'd absolutely collaborate with them on it. I think it's important to democratize all the cheap computing power lying around these days.

>choosing not to focus on specific technologies makes you a worse human being.

maybe that'll make you a worse engineer, but engineers are not human beings.

> engineers are not human beings

That's funny, all the engineers I've known have been.

What crowd have you been hanging out with? :)

Don't know what version of Chat-GPT you're using, but last I checked engineers are still human.

And thank you so much for doing what you do. That use case of, "how the heck do I host my portfolio projects?" is really unreasonably confusing in 2023. You are working to remove a significant barrier to entry in our industry.

> and it's always disheartening for people to claim that choosing not to focus on specific technologies makes you a worse human being.

Hey, I'm the guy who wrote that original response to you. I apologize if that's how you interpreted my comment, but that's not what I meant by it. I wasn't even trying to say that you're a bad engineer (and certainly not a bad human being), I just wanted to say that I disagree with the attitude of that comment. I'm sure if I said it to you in person, you would have picked up on the non-hostile intent and narrow scope of my criticism, but the internet tends to twist things in very negative ways.

Nobody can know everything, and not knowing something doesn't make someone a bad engineer. Software development in particular is more about learning than about knowing, which is why when I see someone saying that they don't need to learn something because <X>, it bothers me. In your comment, you said that you don't want to be a release engineer, but IMO that's a poor excuse since "release engineer" isn't even a role that exists at many places.

And for the record, I didn't even look at the project you linked to. So if you also thought I was criticizing it, then rest assured that I'm much too lazy for that. When I was still learning, online tutorials and resources like yours were extremely valuable to me, so I know how helpful they can be to people even if they're not "perfect" by some snobby asshole's definition...so keep at it!

Fwiw, I’ve used all of those in my career, extensively here and there. I’ve written many thousands of lines of bash scripts. Even written my own php-for-bash-script style code tags that support arbitrary shells. I’ve written my own log based distributed kv store, gone down the YouTube trail of writing my own db, gone through the angular and react iterations, deep dove in docker, docker compose, k8s, crds and custom handlers, and handled infrastructure via infra as code and with a control plane.

I also have a wife and a 2.5 year old.

It’s certainly possible if you don’t stop yourself.

Maybe he wants to spend more of his time with his wife and floating point child.

My floating point children give me so much grief, they are all so irrational.

If they are proper floating point, they must be rational!

They’ll never be their true selves.

Sure, you can do all that, but have you also written video games, UI frameworks, compilers, Twitter bots, localization tools, and web-based whiteboards? Have you ever made a music video or gotten a novel published? If not, why not? I'd say it's because you had different interests and spent your time elsewhere, which is completely fine and not a problem at all. People are allowed to have their own interests and explore the amazing world of technology in their own ways.

It's weird to say that people aren't learning these things because they're "stopping themselves." They're just going in other directions. No one in my life has ever accused me of learning too few things and failing to learn by building things.

Edit: I think I misunderstood the parent comment, sorry for the defensiveness here

No whiteboard, music video, or a published novel yet.

But, come now, his point was he’d never worked with a full stack developer and that they don’t exist.

> that they don’t exist

To my reading, that's not what was claimed.

It seemed to be more like "they exist, but most of them turned out to be pretty shallow with each of the technologies".

As the sibling comment notes, yes -- my point was that the 2023 version of a "full stack developer" is distressingly shallow in most of the individual skills.

I'll use myself as an example. I am a "full stack developer." I have fairly deep backend knowledge. But my React skills basically amount to the ability to make small JSX tweaks, and my sysadmin-y skills are also at roughly the same level.

Conversely, the folks who are really sharp at front end technologies tend to write some psychedelically bad backend code in terms of scalability and maintainability.

For some projects this admittedly doesn't matter. Some projects are simple and don't need to be wonders of engineering.

No offense and not here to toot my own horn, but just because you are better on one side of the stack doesn't mean that true full stack developers are unicorns.

Like others upthread, I am one of them, and probably because I've been paid to do this job for 16+ years.

Honestly my challenge is making potential employers understand that yes, I know my way around Elixir or React, as I do around C or Rust, as I do around sysadmin (not only DevOps) or DBA work, as I do around low-level system code. I'm no world expert at any of them, but most companies need a person that can wear many hats when things go crazy and specialisation is for insects anyway.

Sadly, generalist engineers like me have lost against the trend of "full stack developer" to have been stolen from us by recruiters and less skilled devs to now mean "can use Express and React and maybe deploy on Heroku".

    I know my way around Elixir or React, as I 
    do around C or Rust, as I do around sysadmin 
    (not only DevOps) or DBA work, as I do around 
    low-level system code

To what level do you know these tools?

This is a challenging discussion because the idea of "knowing" or "being good at" a tool is so nebulous.

As I mentioned elsewhere I'm using a definition that is essentially, "the level of skill a solid engineer would acquire after working full-time with a technology for 1-3 years." Not world-class expertise necessarily, but enough time to surpass basic literacy and achieve a real fluency. To encounter edge cases and pitfalls and develop well-supported opinions about best practices.

If you have that level of fluency in all those tools, great! Sounds like you're pretty awesome. In my experience that level of mastery of the entire stack at once is exceedingly rare.

I've used Elixir full time for the past 6 years.

I've written Rust full time for the past 3 years.

I've written C since I was 14 in 2001. I wrote a small operating system (up to reading and running a binary from ext2) around 2004 in C, so I know how a computer works at low level. I might still be able to write x86 assembly.

I've been a MySQL DBA full time for 3 years.

I've administered Linux systems since I was 14 in 2001, and for all my professional career.

Then there's Python, Go, etc.

Probably the one I know the least is React, which means I was PM on a React codebase for a short while, and spent way too much debugging weird issues with Next.js. I stopped paying attention post-hooks, since the frontend world changes too fast, and I'm getting old.

Again, this is not to toot my own horn, it's just that if you live and breathe computers, and hate doing the same thing for long (I blame my ADHD), over a long enough time you tend to have quite the repertoire. I think I'm quite average compared to other people that have been around as long.

You'll soon notice there doesn't tend to be anything revolutionary in computing. After your third framework and language, you'll keep finding the same ideas and concept with minor variations.

    I've used Elixir full time for the past 6 years.

    I've written Rust full time for the past 3 years.

What does this mean? You've been working 80+ hours a week for the last three years?

I'm also fairly generalist although my title is cloud engineer, and have all the skills listed by the OP + IaC infrastructure deployment and cloud stuff, as does everyone on my team. I think that's the trick; everyone on my team is responsible for everything, so we're constantly taking tickets in whatever we're weakest at and consulting each other, which causes really fast skill growth. I can see how it would seem impossible in a more siloed environment though.

> I know my way around Elixir or React, as I do around C or Rust

Interestingly, I find that I've never been asked to write anything in a compiled language yet. I've been wondering how I could work those skills into my job, but it seems like it doesn't come up much in the things I end up working on

Good for you guys, but there is always someone doing more, always someone better, why start a pissing contest? You turned someone’s comment on why deploying SQLite is not trivial into an opportunity to tap yourself in the back so we can all say “wow look at berdon”, but in the end, it doesn’t really matter.

You misunderstood my post - I was trying to make your point - there is always someone better. Just because you haven’t doesn’t mean others can’t.

I've dabbled in the whole stack as you've said. I've written a web framework, done some Docker, done some fun Javascript demos, shipped web and desktop code in a bunch of languages. Built and maintained my own physical servers back when that was a thing people did. Windows, Linux. Currently relearning C and 68K game development (at a glacial pace) as a side project. Did assembly language in college.

Ran a business. Did everything from building the servers, the code, the marketing, the design, community management, merch, and event planning. Call it "ultra full stack," maybe?

So yeah, it's extremely possible.

But I'll also tell you bluntly I was only really good at a few of those things. Most of those things, I was bad at or perhaps more accurately I just acquired the bare minimal knowledge to get by in my specific and limited use cases.

And many of those skills decayed quickly. I knew a thing or two about SEO and front end development in let's say 2000-2012 but those fields change fast and little I knew back then is relevant now.

That's why I say that in 2023, I think the notion of "full stack developer" has grown untenable. You cannot be good (in the sense of shipping actual maintainable and performant production level stuff) at the entire stack simultaneously now that complexity at each level of the stack has multiplied relative to 2005 or 2015. For example, your own db -- is that a fun (and impressive!) hobby project or is that really production level stuff?

To be ultra clear, I am not knocking what you have achieved. It sounds awesome and I suspect you have dived deeper into more things than I. Kudos, and I mean that sincerely.

I heard a story that the inventor of python was once asked in an interview at Amazon to rate himself from 1-10 in python. The interviewer didn’t know who he was and it was just a standard question. He said 7.

I don’t think you have to be a 10 to be a good. But you might need to be 10 to be exceptional. And I will agree that it would be very challenging to be a 7, let alone an 10, across multiple domains at the same time. Certainly, across months/years one’s focus might wander and thus their exceptionality might as well. Being able to adapt and pick up new things at a 7 level though - is the real FSD.

But I see your point and agree with you about the shallowing of the FSD term.

That story is hilarious.

I think it highlights how difficult it is to even discuss these things. We're all using different definitions of "good" and "mastery" and "7/10" and "10/10".

    Being able to adapt and pick up new things at a 7 
    level though - is the real FSD

This is a really interesting line of thought.

By my definition, I actually don't think this is possible, at least not for larger languages/tools/frameworks.

My reasoning is this. To be a 7 (my definition) requires time. You need to not just understand the basic premise and syntax of a language, but you need battle scars. You need to have shipped some code in that language, gotten familiar with the ecosystem of libraries, you've troubleshot production issues, and become familiar with common pitfalls and how to avoid them. You've checked out some large codebases in that language and gleaned best practices and things to avoid. You have probably also spent some time in that community, watched/attended presentations from recognized leaders, and have a sense of which way the wind is blowing.

Everything I just described takes time. I don't think even the smartest person in the world can drop in and achieve that immediately unless we're talking about a relatively simple tool.

I mean, could somebody who already knows CSS pick up TailwindCSS and be a 7 quickly? Absolutely.

Could an engineer who is new to Ruby/Kotlin/Python be a 7 quickly? Not by my definition, not by a longshot. In my experience, seasoned developers drop into existing codebases in these sorts of languages and make a mess of things at first until they get used to the ecosystem.

> I also have a wife and a 2.5 year old.

You did all of these in the last 2.5 years? If not, that part seems a bit irrelevant.

But regardless of the response, some kids are way easier than others (easy kids are those who like to eat and like to sleep, imho), and some parents have more help or stricter separation of parental duties.

One consideration for small vendors who target mid-market or larger customer segments is Vendor Risk Management assessments. These typically dive into resilience (among other things) along with roles and responsibilities. If it's running on a PaaS a good chunk of responsibility can be delegated under the shared responsibility model.

At the same time, I kind of wonder at the direction of things when pretty much every package and application has decided to embrace becoming more complex over time.

It's only going to increase cognitive load, potentially without end (?) which doesn't sound wise long term. :(

Can’t even make 1 joke without attracting the ire of the supersmarts.

So many things (deployment and productionizing) are easily possible without Docker, so why add that layer of complexity unless you know you have a good reason?

Why make Docker and essential part of a system before you actually need it? There’s a reasonable chance that you never need it. I know this, because a lot of business and hobby systems have been in operation since before Docker existed.

Docker has value in many cases, but it certainly doesn’t have to be a necessity.

entirely depends what you're deploying.

Go app which is one blob and some static file ? Sure. Java app that just needs JVM in system ? Go ahead, .service systemd file with some limits is all you need.

Ruby on Rails ? Put that radioactive shit in container, else your server will need to have a bunch of -dev packages just to compile gems and make it running (or alternatively you'd have to compile the gems on same environment server is running, which is more work than just making a container)

> Ruby on Rails

I've been deploying Rails since v 2, and never with Docker. The only time I ever had trouble was setting up Rails dev on M1 Mac when M1 was new. In Linux production, it just worked; and pulling/building gems was less a drag than what people regularly experience in the NPM world.

To be fair, I don't use a ton of gems because I generally don't like additional dependencies unless the value is really there. So maybe my use cases have been too basic to experience the pain.

I mean, it isn't a big deal, just few packages need to be installed, just compared to "put a blob on server, run" it's more complex. It's all fine if you use any kind of configuration management as it is just write once in manifest and done.

But if you just install it directly via package manager, leave alone and need to say reinstall server or something, the knowledge is lost.

I remember having a lot of ping-pong between ruby devs where they, well, didn't note down what system libs and dev package they needed, just had them installed locally at some point and forgot about, then surprise when app isn't compiling on server.

Most of us are perfectly capable of learning new technologies, and there is no shortage of software and disciplines begging for our attention and adoption. For me it's not a matter of "can I learn this" but "do I want to spend my most limited resource--time-- pouring through docs and tutorials to become a few levels below competent at x, y, z, etc, etc, etc interesting systems." Some systems are intrinsically rewarding because they yield thinking tools that can be applied to other systems, while others, like Docker, are pretty much never fun and just a tool.

If you’re actually busy then everything you say yes to necessarily means saying no to something else. If I start a hobby project to learn about and play with technology X, and I end up yak-shaving technology Y, I’m wasting my time and not achieving my goal.

Here is the caveat - you have to learn and be interested with it.

If you just want to do the minimum to run the app, it will make you worse and usually utterly terrible sysadmin, as it usually is going to be "do the minimum required amount of copy-pasting from tutorials to make the stuff running".

Then inevitably don't update shit, nor even set unattended-upgrades package.

And if the server won't get hacked from bad firewall and no updates, the server will ran out of space because dev of course didn't thought about data management and some log file in wrong place without rotation overflowed.

That's from decade+ of fixing shit of developers that think that tutorial on internet made them sysadmin.

> Of course, you'll then end up paying $15+/mo for Postgres, which is hilarious for most hobby projects storing 50MB of data.

Supabase (https://supabase.com/pricing) has an amazing free tier for PostgreSQL which gives you up to a 500MB database.

Note: I'm not affiliated in any way with supabase.com.

Hey that's really cool, thanks! I'll consider adding a link to it in the repo.

I'm a little skeptical that any given PostgreSQL free tier will stick around indefinitely, after what happened with Heroku. And once you hit 500MB, you jump immediately to $25/mo, so if you're running a hobby project, your choice is either to delete data or start paying $300/year. On the other hand, I'd expect a well-optimized read-heavy SQLite app to scale to 10GB+ without breaking a sweat (speculating wildly) and costing more like $3/mo in storage.

I speak from experience here—until recently, I ran a site that would have been 10x cheaper if it had used pure SQLite instead of managed Postgres.

PasS my ass. What's so difficult about setting up a VPS and installing PostgreSQL, MySQL or whatever floats your boat? At Hetzner.com (no I don't work for them) you can get a dual-CPU VPS with 4Gb RAM, 40Gb SSD and 20TB traffic. That'll get you off to the races with Spring Boot and PostgreSQL if you limit the JVM to half the available RAM. For something like Rails, Laravel or Express even easier.

By doing so, I am a failure, and whatever I'm doing is not worthwhile because installing PostgreSQL on my $20 VPS means I cannot possibly achieve /Google/Facebook/Amazon/ scale.

For that price you can get a $10 dedicated server with similar specs. Installing and configuring Postgres isn't a big deal. There are several helpful guides available. For small projects the config isn't important at all. For everything else it is just editing a few lines in a text file.

I'm not sure where the impression that this is some arcane art comes from. For a typical Debian system it is just an apt-get. Yes, the default config has tiny limits for work_mem. Just edit the .conf

Part of this might be that AWS or whichever other proprietary environment has captured developer knowledge at this point. Instead of sane INSTALL files, projects tell users to use a prepacked Docker image.

Pocketbase may be a Supabase alternative for you then: https://www.programonaut.com/pocketbase-vs-supabase-an-in-de...

Fireship 3m video about Pocketbase: https://m.youtube.com/watch?v=Wqy3PBEglXQ

One significant caveat: "Free projects are paused after 1 week of inactivity." I'm not complaining, but I reckon a lot of hobby projects would see sporadic activity. For example, I used to upload a list of my CDs to a VPS, so that when I found something interesting at a music store, I could check to see whether I already own it (#FirstWorldProblems).

If you're going a week without activity then SQLite is basically as overkill as PostgreSQL is.

I think your message implies that performance is the axis that matters and if you don’t need performance at all you should use something other than SQLite (like a file system with json)?

https://news.ycombinator.com/item?id=33975635 is a previous discussion with some pointers that is linked from the primary sqlite website

https://www.sqlite.org/whentouse.html

See the "website" section for some benchmark estimates.

> They assume you're comfortable spinning up public web servers

I'd argue you should be.

I'm currently learning a low level language. It has nothing to do with my job, will not directly be a benefit to my work, and won't look good on my C.V.

It will (and has) _indirectly_ improved my work, because I have a better understanding of why certain things higher up are the way they are, which allows me to make better decisions.

You can ignore everything that's not "your area", whatever that might be, but I'm my opinion it means you'll be stuck at a certain invisible level because of it.

It's a c library. Other languages will have a library/package/whatever to use it. You point it at a file.

"Just point it a file" skips all the bits you need for a production system.

How do you back it up, replicate it, handle two different processes/containers/servers wanting to access the same data.

Using a PAAS solution for a database, you get all that functionality.

I think the point to be learned here is that SQLite fundamentally does not fit in a PaaS model. They are even transparent with this limited use case[0]. I work with embedded systems so I use it a lot, and all the web work I do nowadays is one-off project site and small utilities that are usually a single process so I end up use SQLite 90% of the time I’m reaching for a solution.

0: https://www.sqlite.org/whentouse.html

I think it's more that PaaS vendors aren't interested in first-class SQLite support when they can sell overpriced managed Postgres instead. Sure, it doesn't scale the same way, so it's hard to move upmarket and sell to Enterprise, but it's a shame that there's no one-click solution like there is for a managed database.

Let's say the following product existed:

1. You can only run one instance of your app.

2. There is a small window of downtime each deploy.

3. Your app has access to 10 GB of storage. The storage is persisted across deploys and can be used for sqlite.

4. Your sqlite data is automatically backed up and can be restored / downloaded as needed.

How much per month would you pay for that product?

Interesting question. I think you could probably get those features by deploying Piku or Dokku for like $4/mo on a VPS, and the equivalent managed Postgres on Render would be like $20/mo (16GB SSD), so something like $6-10 seems like the right range to me. Maybe not a booming business, but a nice margin percentage and worth paying for good UX. And the cheap version doesn't need to be the only version if people want more RAM, etc.

I'd find such a product really appealing, especially with a dedicated backup strategy that uses the SQLite backup API to do hourly snapshots or uses Litestream to S3 by default.

Let's say the price was 12. If this idea is compelling enough for you at that price point to venmo me an advance for your first month, I have some ideas to explore that could actually make this viable. FWIW, I've worked professionally on a PaaS and a managed database offering.

If you make it so I can have one SQLite database per user, I’d be a customer too

When you say per user, you mean per user of your app?

yeah exactly. I want each user's data to be silo'd in their own sqlite database each. It's basically just a multi-tenant setup using sqlite.

Cool, thanks for the clarification!

$20 per month assuming:

* Someone who knows Django but has zero devops skills can deploy my app with a few simple commands, add a fix, and demo it to me. Crucial: They must ask me zero questions.

* The backups happen to a non-you service, I one-click auth my google drive and/or dropbox.

* There are instructions on how to stand-up the web app on another service if you shut down. Those instructions might require two hours of my time, but should be complete.

I'd probably be willing to pay ~$10/mo. i.e. a 100% markup on a low-end $5 VPS box.

...but in addition to bravura's requirements, I'd add: I'd want SSL termination, ideally with the option of me bringing a custom domain.

That's why "most cases".

If you are trying to replicate or make available to a bunch of machines or similar - it's likely the wrong thing (although there are tools to do this, I've never used them). If you are just trying to back it up there is a pretty simple back up command.

I think we disagree on the "most cases" aspect that

For most of the things I touch, having a single-point-of-failure data store is typically not acceptable for production environments, except perhaps caching.

If you are a good software engineer, you probably work at a place that needs good software engineers, which by definition tends to be a place with a system that needs high availability or low latency or both or something else, while serving a lot of customers or load or something, all of which basically results in a pretty complicated distributed system...

So in your day to day job as a good software engineer, yeah sqlite isn't going to be a good choice. It doesn't work well for the sorts of systems you work on. But if you were going to build a new social network for your college or something trivial like that, and you just want to move fast and have a crazy simple deployment setup, you might well use sqlite.

It's a file is relevant. It allows you to think of it like other files:

- how do you back up a JPG? You copy it, cause it's a file

- how do you replicate it? You copy it, cause it's a file. Unless you're talking about fancy DB replication, in which case well, that's not really a thing we do with files much. You'll have to do more research. but that's cause you're trying to do non-file things to a file.

- how do you handle two different processes accessing a JPG? They both open and read the JPG. Same for SQLite. For other concerns, you're trying to do things that don't map well to files, so once again more research needed.

I don't update and append to jpgs though...

Only you can’t copy it if it’s in the middle of a transaction or you corrupt it and have to roll back the Journal. And coping the journal, WAL, and DB itself directly and trying to backup from that isn’t recommended. In fact SQLite itself has a purpose built backup API.

https://www.unixsheikh.com/articles/sqlite-the-only-database...

Yes, technically a SQLite database is just a file on disk. And for the most part, you can treat it like other files on disk. Except for when you can’t. The GP’s questions are valid

how do you handle two different processes writing to a JPG?

The sarcasm and intentionally-missing-the-point here is not really in the spirit of HN, but I'll try to address what you seem to be trying to say, which is that "it's obvious and I'm an idiot for not seeing that it's obvious":

- Will multithreading make it break? (Not with WAL mode, but you have to set it manually, as this article suggests.)

- When using a PaaS, you need to explicitly add a volume and mount it on your server machine, which you might not think to do if you're a brand new bootcamp grad

- You can't(?) run migrations from another process (or at least, people don't seem to talk about doing this), so you need to prevent your server from writing while you run the migration. Even if I'm wrong, and I would love to be, it's frustrating that people don't talk about the completely ordinary need to run migrations on a database.

- Backups just mean copying the file somewhere, which is nice, but you might need to configure that yourself instead of just using somebody's managed Postgres backup feature

So, there is at least some complexity that comes with managing a "real" SQLite web server.

(I'm probably at least 25% wrong on some details above, but that's kind of the point, it's not always easy to figure this stuff out.)

It's not sarcasm. The fact that it's a library and you point it at a file matters, and should be thought about. It implies that it's not built for distributed systems. It's not supposed to be a managed service. It's not a good option for what you appear to want. You deploy it as part of your application.. it's a library.

> It's not a good option for what you appear to want.

Ok but the article you're replying in the comments to says "SQLite is all you need for nearly everything", and what the comment you're replying to is describing is, to use their very apt word choice, entirely ordinary.

So how do we square this circle of somebody being told both "SQLite is all you need for everything" and "it is not a good option for your totally ordinary use case"?

I think the answer is that you are a different person than the author of the article and you don't agree with the article's claim. Which, great, that was irskep's entire point!

I tend to agree with irskep that this SQLite architecture is really interesting, but I also don't quite get it. There seem to be a lot of missing details for totally bog standard applications, that people frustratingly seem to just not even mention - like the migrations thing, or how to do zero downtime deploys in general, or a few other things - as if those concepts are irrelevant or uncommon. But they aren't, they're important and typical things to think about.

Third paragraph of the article says:

"In contrast to many other database management systems, SQLite is not a client-server database engine, but you actually very rarely need that. If your application software runs on the same physical machine as the database, which is what most small to medium sized web applications does, then you probably only need SQLite."

That's how we square it. It's right there in the article.

That is a claim, not an explanation. Satisfactory answers need to be provided for the typical requirements that a small to medium web app might have, or the claim is unjustified.

(Nearly) zero down time is a common requirement. Can a live backup be made while transactions are in progress? That appears to be the case. What about schema changes? Can columns or indexes be added without interfering with access to the tables in question? And so on.

> (Nearly) zero down time is a common requirement.

Are we still talking about small to medium web apps ? I'm sorry but if HN goes down things will be OK. In fact, there is a very large majority of services that can go down go down and things will still be OK.

I've realized that this whole thing comes down to the word "most" doing too much duty here. I don't think it's true that most applications just run a single db-and-application node. I've never worked on such an application. You and the author do seem to think this is true. It would be difficult for either of us to support our intuition empirically, so this is where the divide comes from.

Yep, that's probably true.

> I don't think it's true that most applications just run a single db-and-application node. I've never worked on such an application.

Rather than applications you've worked on (many of us spend years on end working on a narrow range of applications), consider software you use (most of us flip between multiple applications every day spanning the gamut of uses). Ignore, for a moment, whatever you know about their implementation, and focus on the user-facing functionality.

Do even half of them logically communication between application nodes?

(Consider that there is still a rich class of software that is fully capable of running locally on the "client"'s computer!)

Yes I think pretty much all the applications I interact with require fault tolerance and uptime that (to me) seems simpler to implement with separation between application and database nodes.

So you do not interact with

- Browsers

- Messengers

- The telephone directory of your cellphone

- SMS apps

- The sqlite.org Website

- ...

Distributed databases are rarely needed. I wouldn't call it an ordinary use case. The article isn't claiming Elon Musk can run Twitter off SQlite

It is honestly difficult for me to imagine what kind of applications people are working on that have these - to me - very lax reliability constraints. But we're just disagreeing over what "most" applications are like, based on our experiences, without any empirical data to say either way, so :shrug:.

WordPress is the obvious example of an application that can run like 99% of instances just fine with SQLite.

Interesting!

Off-the-top of my head.

* Hobby apps designed for yourself and your friends.

* Hobby apps that you hope will become products but probably won't achieve traction.

* Apps where you're trying a new stack or framework.

* Annotation apps for academia.

Basically, every single app written by people who aren't doing internet facing web dev as their core engineering function.

Also intranet apps as other commenter mentioned.

When I read an article like this that claims some architectural technique is broadly applicable, I don't think it is talking about hobby or just-for-learning applications. Certainly you can do whatever you want with those, but that's not very illuminating.

I'm not sure exactly what you have in mind for annotation apps for academia - things like zotero that run client side? If so, sure, there is a big world of client side software where a database is useful, and I think SQLite (or DuckDB) seems like a no brainer there.

I don't really agree about intranet apps, which are often even more critical to the people using them than an arbitrary consumer app. But I'll grant that for a company that spans a small number of time zones, you can at least have downtime windows outside work hours.

In any case, as I've said all over this thread, the only disagreement here is over what kinds of software is "most". And my intuition for "most" is based on my experience working on and using applications where a lot of effort is made to keep the thing running all the time while still evolving it. Maybe you're all right that "most" software isn't like that.

you almost always need it if you don't want service interruption because server died.

Most things don't need zero downtime deployment. Now it's nice to have and might be an interesting technical challenge to solve, but it's usually not strictly needed.

edit: when I worked as a sysadmin we would have to schedule outages for updating apps that were designed for zero downtime anyways because those were the processes of the organisation.

Yeah I've realized this is just a disagreement over the word "most" which won't really be resolvable empirically. It doesn't fit my experience that most applications are single node with lax uptime and failover requirements.

That's fair.

There’s likely to be many more intranet apps than customer-facing ones.

What? I've never worked for a company that would tolerate downtime during deployments. Downtime is ok for personal projects, but not for most business applications.

Most business have daily downtime where the entire business is closed as in not business hours. Being sensitive to downtime is more common for companies that has some kind of online service as their primary product, but most companies are not online companies, or even global companies with offices across all the time zones.

Even for online business some downtime might be acceptable or even preferred. I used to work for a company that made most of it's money through a website and it would take the site offline for 8 hours a few weeks before important events to run a full scale load test on the production infrastructure.

Another place I worked we had to schedule downtime even when updating applications that were designed for zero downtime.

I have never worked a place that didn't allow for downtime or even heard of such a place other than the big tech companies.

scheduled maintenance downtime at night != unplanned failure downtime at random time

Correct, but in the context of deploys I would hope they are not random failures, but rather planned events that you do when you have updates to... deploy.

Yes, but you also don't want to have to deploy in the middle of the night in order to avoid downtime during peak hours.

Not only is that a pain for whoever is monitoring the deployment, but now if the deployment breaks something you're going to have to go wake up all of the relevant stakeholders, if you even know who they are.

Not to say late night deployments are never justified, but definitely not something devs want to be doing regularly.

There’s alternatives like making rollbacks really easy, and then automating it.

I haven't looked into how everyone else does migrations in SQLite.

The way I do it is to check the DB version `PRAGMA user_version;` on application start and run migrations if required, before the app starts taking connections. Yes, this means my app will be unavailable for a few seconds when upgrading. If you want a zero downtime solution then you will have to do it in a different way.

Just this morning I was wondering about where to host the next iteration of my side project (GiftyWeddings.com). It currently uses Go + SQLite, and I deploy it to a small $9/month instance on AWS, but the setup involves a bunch more Ansible and messing around than I want, and I'm not even sure the Ansible scripts I wrote a few years ago would work anymore (or work a second time). A simpler PaaS-like system that works with SQLite at a few bucks a month would be great. I'll check out your repo and fly.io in more depth -- thanks!

PocketBase is worth a look. It can run standalone as an API server that wraps access to the SQLite database, or it can be used as a Go framework.

I don’t get this. Go can be compiled to a single binary containing SQLite. Scp the executable and run it. This will work on any Linux vm without any setup…

That's what I do now. But there's a bunch more setup:

- Install and configure Caddy to terminate the SSL. Caddy is great, but still stuff to think about and 20 lines of config to figure out.

- Configure systemd to run Caddy and my Go server. Not rocket science, but required me figuring out systemd for the first time and the appropriate 25-line config file for each server.

- Scripts to upgrade Caddy when a new version comes out (it wasn't in the apt repos when I did this).

- Ansible setup scripts to clone the repo, create users and groups for Caddy and my Go server, copy config files, add cron jobs for backups (150 lines of Ansible YAML).

It looks like you don't need most of this, or get it without additional configuration with Fly.io and Render.

It's kind of like the difference between Dropbox and "you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software." (https://news.ycombinator.com/item?id=28153080)

Fair enough, but those things are unrelated to SQLite. Any app which uses a Postgres or MySQL PaaS would need to overcome the same hurdles.

It'd be nice if those things were easier, but the SQLite part of it cannot be any simpler or easier than it already is.

Backing up SQLite can be done with a 2 line shell script

    ssh vm 'sqlite3 my_database.db ".backup my_database.db.bak"'
    scp vm:my_database.db.bak .

That's true, but previously the PaaSs I've looked at didn't seem to have the same concept of persistent volumes, or maybe they were costly, I forget. In any case, I tried on Heroku before, and one other provider, and they didn't really support this use case, or pushed you to use their hosted PostgreSQL offerings, which were expensive (for my budget). Fly.io looks a lot simpler and cheaper!

Your backup script is basically what I do, but I use a little Python script that also uploads it to S3, keeps the last 10 days worth of backups, and so on.

Company offering compute and expensive hosted DBs, thus making SQLite difficult to use? Color me surprised :-)

For use cases similar to yours, I'd probably bite the bullet with a raw VM from somewhere and manually configure systemd and Caddy on it and manually upgrade Caddy on it from time to time. We would probably get a re-usable Caddyfile and systemd unit from this initial setup, making it even easier to change VMs later if needed.

It's so simple, it doesn't need a PaaS.

I really appreciate all your replies and sharing your experiences. It's helping me think through deploying my own side projects which, you guessed it, will use Caddy and systemd on a VM somewhere.

>Of course, you'll then end up paying $15+/mo for Postgres, which is hilarious for most hobby projects storing 50MB of data.

You can run small project on $5 VPS with Postgres just fine tho...

Managed services in cloud are usually massively overpriced

The problem is when you start to approach oltp. This is not a space for sqlite.

I have thought about jacking inotify into sqlite_busy_handler() to get the exclusive writers doing better than random waits, but this full api isn't exposed to PHP (where I need it), and doing it at C would suggest alternate approaches, maybe even with xargs at the shell.

Oracle has a DBWR process that manages itself, but a write-heavy app on sqlite must explicitly declare one, and there are many further traps down this path that will trip the unwary.

I don’t get this. SQLite can perform 500k writes/s (or 5k write txns/s). What app are you building which requires more than this?

To optimize the writes, you must look at inotify.

The sqlite commit is a specific pattern, it is either write or close/write.

Using inotify events can see these faster than random waits.

Respect. I’m curious what use case for >500k writes/second or 5k write txns/second you’re implementing? I know nothing about inotify and random waits. What else can you tell me?

In SQLite's C API, there is a busy handler that you can set that accepts a pointer to a function. This is where inotify would be very practical:

https://sqlite.org/c3ref/busy_handler.html

The normal operation of the busy handler is to accept a maximum timeout (also implement at the SQL level with a PRAGMA). Setting this at the C API also sets a dedicated handler.

https://sqlite.org/c3ref/busy_timeout.html

The inotify interface (which is specific to Linux) allows the kernel to alert a process when writes or close/writes occur (among other filesystem activities). The arrival of such events is a more efficient way to check if the lock has been released, rather than a wait/retry of limited duration.

Let's look at this from the shell:

  # session 1:
  $ sqlite3 test.db
  SQLite version 3.34.1 2021-01-20 14:10:07
  Enter ".help" for usage hints.
  sqlite> create table foo(bar);
  sqlite> .quit

  # session 2:
  $ inotifywait -m test.db  
  Setting up watches.
  Watches established.

  # session 1:
  $ sqlite3 test.db
  SQLite version 3.34.1 2021-01-20 14:10:07
  Enter ".help" for usage hints.
  sqlite> insert into foo values('hello, world!');
  sqlite> begin transaction;
  sqlite> insert into foo values('so long, world!');
  sqlite> delete from foo;
  sqlite> commit;
 sqlite> .quit

  # session 2 output:
  Setting up watches.
  Watches established.
  test.db OPEN 
  test.db ACCESS 
  test.db CLOSE_NOWRITE,CLOSE 
  test.db OPEN 
  test.db ACCESS 
  test.db ACCESS 
  test.db ACCESS 
  test.db ACCESS 
  test.db MODIFY 
  test.db ACCESS 
  test.db MODIFY 
  test.db CLOSE_WRITE,CLOSE

"Path units" under systemd also use this interface.

Note with Neon now you don't need to spend $15+/mo for Postgres because they separate compute from storage. So compute can scale down to 0 and storage is cheap.

Neon only recently entered public preview and I am unable to find any pricing information.

Launching on mar 15th

Congrats on the upcoming launch! I'm definitely very strongly considering using Neon in my upcoming project. Mind if I ask a couple questions?

Would you be able to comment on if Neon is a good fit for having one postgres database per user and how well that would scale? E.g. what if millions of users?

Also with the managed service, is there help with applying migrations or helping manage migrations for such a multitude of postgres databases?

Lastly, is there any way of querying across neon databases, e.g. aggregating data from many user's databases? I saw you partnered with Hasura a bit. Hasura would be a great fit here for aggregating and "federating" each user's postgres database, but with their most recent price change I'm scared of integrating too much with them. Wondering how you think one can best query e.g. a million user's individual postgres databases?

Thank you and feel free to ignore I know it's kinda asking alot

One database per user is something many of our customers are already using today.

I think if there are millions of users it’s a bit of an overkill. Because likely you will have some very light users that you can still collocate on one database but give heavy users a dedicated one.

With Neon you can do either. Our minimum configuration is 1/2 core which may still be too much for one light user.

Thank you! That makes a lot of sense. Yes its probably overkill to do one db per user. Better to split off when they become heavier users. Thanks again for the awesome tech eh and good luck on the formal launch!!

I've used dokku to great effect.

SSHing into a new linux VPS can be intimidating but there a few good digital ocean tutorials that'll take you from new VPS to dokku and once you're there it's very much a PaaS

This is cool to see! I’ve been working on a website for online ordering, and a key requirement is free hosting [0]. After some searching I figured fly.io + SQLite would be enough for my use case but I can’t be 100% sure until I actually deploy, which I haven’t yet. It’s nice to know someone else has done it this way successfully!

[0] - It’s for a small business not in the US with no serious requirements on availability etc so free, or close to it, seems achievable.

Also "as long as your web application can run on the same machine as the database, which it can in 99% of the time"

Where did 99% of the time come from ?

Not the poster you replied to but from my anecdotal experience -- from most of my work ever.

I can only remember 3 companies out of all 40+ I consulted and contracted for that actually needed a separate machine for app and a DB, let alone such that actually truly need several of each.

It's a very non-romantic truth but most projects out there can easily fit in a VPS with 4 CPU cores, 8GB ram and 100-200GB SSD space. App + DB + self-hosted telemetry included.

You didn't add all points to Strength so you could only be a "application developer", did you?

is $15/month really that much of an expense? i share my PostgreSQL instance across projects, really does not seem like that big of a deal to me considering it's such a big hobby of mine.

Why do people use hosted databases instead of just installing them? I don’t understand it.

If you install your own copy, you are on call for it, and the colo is unlikely to offer much help. You also have to set up monitoring or you won’t even know when it fails. Then there’s replication. Backups. All this stuff is work that PaaS vendors are ready to automate away, if my time is expensive for the org.

For fun, sure, dink around and learn as long as there are no customers to affect.

Can’t you just run a script that sets all that up for you? Just like any other programming task?

I suppose Terraform could do it (I’m pleasantly surprised to see it has a PagerDuty plugin), but it’s a lot more to write than a PaaS API, and you probably can’t borrow a specialist SRE/DBA for a small fraction of his salary.

from another hand, vendor can screw things up and you don't have control over it.

You are not wrong but for each personal project this upfront cost is 2 hours maximum, with accumulated 5-6 more over the course of the next 3-6 months.

Not a huge sacrifice. Though I do get the argument of "I want to pay $5 and it to just work" and I've done so part of the times. Just pointing out that the upfront investment in doing it on your own is not so big.

Because operating a database sucks.

Specifically: setting up database replication sucks, setting up failover sucks, setting up backups sucks (even with a PaaS you need to do this, but you can use it as your first layer), migrating database clusters to new versions or hosts sucks, keeping your host's software up to date sucks, and setting up alerting and monitoring sucks.

It's easy to get an open port to connect to, but hard to keep it there for five years.

I've done self-hosted databases. It's easy to take it for granted if you've done it before, because once you've done it, you know that it is easy. But, even though it easy, it requires work. You have to spend time learning how to do it well, and you have to go through all the steps setting things up and configuring it. There's all the backups and monitoring that you want. Death by a thousand cuts, and all that.

I still self-host databases for personal projects, but I can see why so many people don't want to bother. When I've done it professionally, I was basically working as a sysadmin and managing stuff like database servers was a core job function, not something tacked on to a development job.

Reply on news.ycombinator.com