Show HN: PocketBase – Open Source realtime backend in one file | Hacker News Reader

Show HN: PocketBase – Open Source realtime backend in one file

github.com

563 pointsrandomwebdev111 comments

Very nice, I thought it was a basic wrapper around SQLite but then realized it actually had more features then that, such as it's own admin dashboard. Then I realized you built multiple auth designs into the system, then I realized there was a javascript SDK to interact with it, then I found your beautiful documentation site, then I realized there is an actual product inspiring it's development (https://presentator.io/)

Each of these is another encouragement to use pocketbase. I went from "eh, I'll just keep my own wrapper" to "pocketbase has a lot of promise, I should try it"

PocketBase is a small open source project that I've been working for the last couple of months.

You could think of PocketBase as a lightweight Firebase/Supabase alternative.

In short, it is an open source Go backend (framework and app) consisting of:

- embedded database (SQLite) with realtime subscriptions\

- backed-in files and users management

- convenient Admin dashboard UI

- simple REST-ish API

And all of this compiles to a single portable binary.

It's still a little rough around the edges, but you could give it a try and share your feedback.

This looks cool.

Nice!

It looks like you put a lot of work into this, what was the motivation behind this project? Did you not like the existing solutions available?

Thanks! It was initially created to be the backend for the next version of my other open source project - Presentator v3 (https://github.com/presentator/presentator).

You could read more at https://github.com/presentator/presentator/issues/183, but to summarize - I wanted something that could be self hosted with almost no additional setup and no dependency on 3rd party services.

This is quite a cool project! Congrats on the public release.

Very nice, whats planned next?

I'm confused about one detail:

Does it render pages automatically based on Schema? I see your really simple example under examples, and its so minimal I want to think you just get all the tables and render views from that. Am I crazy? If so this is really neat.

Did you ever consider using something like GORM? Because I could see this working with other databases and being extremely useful. I use Django to build really quick CRUD interfaces, but sometimes I need to be able to customize it more, and the admin UI is not as extensible as I would normally want, they even suggest you just build custom pages if its not doing what you need, which is a bummer because Django Admin covers the majority of my use cases.

"Collections" stores a single records table meta data (eg. name, fields, validators, etc.), so when you create a new collection a new related table will be created (see https://github.com/pocketbase/pocketbase/blob/3d07f0211dc747...).

The admin UI just shows the collections through the web api (https://pocketbase.io/docs/api-collections/).

I actually started with GORM, but its too complex and I ended up replacing it with a simpler query builder package (ozzo-dbx). While the query builder has abstraction for other databases, I'm not planning supporting them at the moment.

No worries, I've seen another project called UAdmin that kind of reminds me of this, my problem with UAdmin was that there were bugs and I didn't quite get why it worked how it worked, but having used Django since I realized what I was missing out on.

Thank you for following up, I did see you had a package for extending the built-in Go package for SQL.

feedback: re the site:

i have no idea what you mean by a "backend". If it's got a "admin dashboard" it's got a UI which means it's got a front-end which means it's not a backend. I'm assuming "go backend" isn't specific to go but just indicating that it's written in go.

it's got a example code block but doesn't explain the point of it. Why would an existing framework with a UI and everything need me to write a `main` method? Why doesn't it have one already? If it doesn't have one why doesn't it. If it does have one why am i not using it? My thinking here is that there are many existing patterns for registering plugins / extended functionality that don't require altering the main method of the thing you're hooking into. Why aren't you using that? Is this maybe intended to be like... a starter template for a Sinatra like thing with a pre-defined set of functionalty? Kind-of like how Django sets up model administration UIs for you?

re your summary here and the site:

i have no idea what "backed-in files" means and I've been doing web-dev for decades.

"convenient Admin dashboard UI" ... to admin / do what?

"simple REST-ish API" a) why "ish" and not actually REST b) ... that let's me do what?

--- in summary.... this looks useful but i don't get what you're actually offering or what it's intended usage scenario is. This is compounded by the fact that you're using a very atypical definition of one word "backend" and a very uncommon term "backed-in files"

It's a lightweight replacement for Firebase. I That's why it is described as a backend, because it's primary feature is to act as a data store. If don't understand what a project does, and you're unfamiliar with the comparisons given (Firebase, Supabase), the right thing to do would bw to look up the comparisons.

I know you have user management, a database, and do something with files. As noted before the "backed-in files" isn't a commonly used phrase so i'm left with 2 pieces of meaningful info, neither of which is particularly notable because every multi-user app has both of those.

Firebase gives me a list of use cases https://firebase.google.com/use-cases and documentation whose left nav gives me a good insight into the high-level functionality it provides "authentication, storage, hosting, security rules, extensions, machine learning, etc"

I think it's safe to assume you don't provide all of those, but i don't know what subset you DO provide. As a result, if i give you the benefit of the doubt, telling me it's a "replacement for firebase" is not meaningful because i don't know what parts it can replace unless it's just the few listed in which case it's absolutely _not_ a replacement.

I'm not OP; I wasn't aware of the project until I read the HN headline.

"Lightweight replacement" also implies missing features, IMO. This project has - at the minimum - authentication, storage and hosting in a single binary that can be self-hosted. This absolutely can replace Firebase for some use-cases (e.g. on my LAN/homelab)

Aside: your tone is weirdly - and needlessly combative. If you see no value in this project to yourself, that is fine, but it appears to be making you angry for some unfathomable reason.

I’m familiar with firebase and an electron developer and I still don’t understand what this project does. I wish I did, because it seems it’d be interesting to me. Even the app that uses this (Presentator) is described in a way that leaves me grasping for “bit what does it DO?”

There is a bunch of software in this space like supabase, flat backend, and appwrite that attempts to be a generic self hosted backend for apps in a manner similar to firebase by providing data storage and authentication with a rest API in an integrated fashion, usually with a built in admin interface for schema design, user management, etc.

You can also think of them as software like postgrest but with the addition of authentication and other app oriented features, and with a JavaScript client library out if the box.

This program is doing the same thing in a more lightweight way as a single executable using sqlite.

The idea is that by using this type of generic backend software for your database and authentication, you can basically just write the remainder of a typical app or webapp entirely as client side/frontend software, which is something that you can also do with firebase, but this allows you to easily self host it without having to create your own custom backend software.

Ok, so it's a self-hosted CRUD boilerplate with auth written in Go? Or a self-hosted Notion but with less view/UI customization out of the box?

> Ok, so it's a self-hosted CRUD boilerplate with auth written in Go? Or a self-hosted Notion but with less view/UI customization out of the box?

I guess you could describe it as "CRUD boilerplate" although that could also make it sound like the main purpose is to provide database table/form views like Microsoft Access which isn't really accurate.

It provides a database rest API like PostgREST with authentication so that you can write whatever frontend you want for it.

Since you say that you're familiar with firebase, imagine you're writing an android app using react native or something that uses firebase as the backend to store users' posts (or whatever they are).

The idea is that if you want to self host it instead of using firebase, you just run this type of software on your server and it will provide the basic functions that any client side app would need on the backend: storage for users' data as well as handling accounts.

I don't think it's really that complicated; it's basically just an interlace to a database, but if you look at something like PostgREST that automatically creates a rest api for a database, you would probably think "wow, I could almost create an app just using this as the backend without having to create my own backend, but I would have to handle authentication/registration myself," so these types of platforms just go a little further and handle that as well.

Gotcha. Thank you for the explanation.

It can be used as a CRUD boilerplate for Go projects (where you extend it), alternatively you may use it as a standalone REST(-ish) API backend that integrates auth and storing & retrieving state from some other app in any language. It's similar to DjangoREST which can bw used in both modes (as a base for a project written in the same language, or purely as data layer interfaced via REST)

You haven’t tried really hard:

> PocketBase could be used as a standalone app or as a Go framework/toolkit that enables you to build your own custom app specific business logic and still have a single portable executable at the end.

if it's used a standalone app then what functionality is it providing beyond "user admin". Like, what problem does it solve in that use case?

> a Go framework/toolkit that enables you to build your own custom app specific business logic

that definition applies to literally every framework and toolkit. It tells me nothing.

It provides an authenticated rest API (with a premade JavaScript client library) to an sqlite database, so you can store users' data similarly to how you might use firebase.

> and still have a single portable executable at the end.

You're quoting selectively in a way that elides meaningful parts of the sentence, and then stating that the description lacks meaning.

feedback: the "in one file" marketing confused me for a bit until i realized you meant "as a single binary". i would say that instead.

I got tripped up on the same thing.

> backed-in files and users management

Did you intend to say baked-in? Like it's built-in? If so, I feel it's better to just use "built-in" because it's more familiar. The same typo's in the site too and confused me a bit.

Yes, built-in sounds better. I'll update it.

Impressive work!

I'm doing something along these lines in Go with Postgres, but I'm definitely not as productive as you.

What do you think about Echo? Was it your first choice as a web framework? How in your opinion is better than Gin for example?

Out of curiosity, how much time did you spend building the admin dashboard? Did you have previous experience with Svelte?

Looks very nice.

One thing I noticed is you are using server sent events for subscriptions. SSE have a lot of nice properties but do have one nasty limitation (from https://developer.mozilla.org/en-US/docs/Web/API/Server-sent...):

"Warning: When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100)."

This can lead to some weird bug reports from users.

Yes, I'm aware of this limitation, but PocketBase uses HTTP2 when combined with the --https flag (this is by go's default net/http implementation) so it shouldn't be too much of a concern.

First time I’ve heard about the 6 connection limit.

I’m not fully understanding the response - are you saying that the limit is not imposed if you use https? Or am I reading wrong?

Go's default `net/http` package will serve HTTP2 when a https configuration is provided (also most web browsers don't support h2c, aka. HTTP2 without tls).

The default limit of ~100 connections should be more than enough for most applications (additionally, the JS SDK client maintains a single SSE connection for a page no matter to how many things the user has subscribed, so that also helps).

HTTP/1.1 has a limit, set by each individual browser vendor, for the maximum number of connections between a client and a unique server domain. So, if you exceed 6 simultaneous connections to that server (across multiple tabs and windows), it will move the request to a stalled state (like a queue) until one request is completed.

Best solution today is to move to http2 on your server -- which has an SSL (TLS 1.2) requirement.

Looks like pocketbase implements this. If you use another server, like nginx, you have to enable this for each site.

Wow, super impressed. I have always wanted to build something like this where everything is embedded in 1 single file and I knew Go could be a great way to do it but always wondered how to handle the database part. Along comes SQLlite.

One feedback which will probably deviate from everything in 1 file concept but adding an option to upload files to an S3 URL will be awesome. Perhaps a pull request ? I am thinking of use cases like document management using this and if I can have the actual files in S3 (or S3 compatible), that would be a huge plus.

EDIT: I spoke too soon. You already have an option for S3 as I tested the demo. Very cool.

How much work went into this btw ? The github, documentation everything is well done. I envy work like this :)

By default PocketBase uses the local filesystem, but you could also configure a S3 compatible storage (AWS, DigitalOcean, Vultr, etc.) from "Settings > Files Storage > Use S3 storage".

I've started working on the project ~ 3-4 months ago. The last 4 weeks I've taken a longer break from my day-to-day job so that I could finalize the first release.

Uploaded files must deviate from „everything in one file“ regardless of using S3, right? How (and why) would you store user uploads within the binary?

yea it was a silly question. It does allow S3 anyway so we are good.

My intent was not to point out a „silly question“. I was legitimately interested if there is a reasoning and what a use case for that would be. Thank you for responding nicely!

Uploaded files and your database are stored outside of the binary.

This here is some quality work top to bottom, including the documentation and the website.

+1 .. Very amazing!

This looks like a great solution for rapid prototyping scenarios where you want a basic CRUD backend with as little friction as possible. The API preview feature in the admin dashboard is a really nice touch.

This is cool. I think it would also be neat to have option to have Litestream [1] integration so that users can use this without any persistent volume or worry about back ups

[1] https://litestream.io/

PocketBase uses embedded SQLite in WAL mode, so it's already compatible with Litestream if you want to integrate it.

Or do you want Litestream somehow embedded in the Pocketbase binary so it's part of the "one file" artefact? They're both written in Go, so I guess that could theoretically be possible.

This looks really great. Have you thought about adding Operational Transform or CRDT primitives to your offering? Realtime subscriptions are nice - but they're even better with realtime collaboration. Because you're using SQLite directly, it's easy(er) for you to build in efficient CRDT mutations to your database rows because SQLite's BLOB column supports incremental IO: https://www.sqlite.org/c3ref/blob_open.html

This could allow you to save a CRDT update operation without needing to read the entire CRDT blob into memory, parse it, and then save it all back to disk.

Currently any additional data transformation is left to the developers to extend via event hooks or custom client side handling.

I've worked with CRDT in the past (yjs), but it may not be very useful in PocketBase considering that the application was designed to run on a single server and db writes are practically queued (you can have only one sqlite writer at a time). I'll investigate it further and may consider it for future release.

I think if you integrate Yjs or a OT library first-party, you'll have a really exceptional framework - open source firebase alternative top-to-bottom, ready for "local-first" collaborate own-your-own-data apps. Sounds like my dream stack for small-scale projects or friend/family apps.

Nice but my super anal quip is please don't call this thing "realtime". It doesn't make sense in this context. What is realtime about it? Are there timing guarantees or are timing constraints provided in API requests? Because that's what realtime systems are. Realtime does not mean "real fast". For example, a train track switching system might be getting run by a 1MHz CPU and still be real time. It's not about speed, it's about deadline guarantees.

It’s not a real-time operating system. In this case, real-time means it is built such that a client gets access to updated database values as soon as they change, as opposed to a non real-time database which doesn’t necessarily have a built-in mechanism for reliably on time updates.

You mean like eventual consistency vs immediate consistency? Because a database that reflects its changes at the end of a transaction is called an ACID database not a real-time database.

this looks really great! i had recently been thinking “boy it’d be really great if there were a ‘supabase but sqlite’ available” and here you are! now i just need a clever project as an excuse to use this.

1 question: are the docs in a repo anywhere? i don’t see them in the main repo (i might just be missing them somehow) or anywhere in the org. if they are public somewhere would you be open to a docs polishing PR?

The documentation is not auto generated and is part of the static landing site (built with SvelteKit).

I didn't bother open source it because it is kinda messy and a little complicated to be edited by users not familiar with the codebase, but I'll try to find some time in the future and may publish it.

In the meantime, if you find typos or think that some of the wording could be improved, feel free to open an issue or discussion in the main repo and I'll fix them.

This looks awesome! I will definitely give it a try. Right now I use postgres in a docker compose but its definitely a bit heavier for single machine prototyping.

One thing that I like about my setup though is that there's a clear migration path to horizontally scale. Do you have a recommended way to move past pocketbase should an app get to that point?

I guess there will be differences from use case to use case, but in general migrating from PocketBase to a more common horizontal supported stack is not different from rewriting your application.

Internally, each Collection creates a standard SQLite table that holds the collection records, so migrating the data structure shouldn't be too troublesome. The only thing that may prove difficult to migrate could be reimplementing the access rules and filters.

But in my opinion, when your application reaches that level of growth requiring multiple servers and services, your business use cases mostly likely will have changed several times already from your initial idea.

Perhaps sharding data by geographical location could be one path - e.g. deploy one server in the US and another in Europe, each with its own database.

The whole concept of edge computing - deploying app servers closer to your users for minimal network latency - seems to go hand in hand with using an embedded database like SQLite. (I haven't tried it in production yet, but looking forward to.)

For anyone curious, another popular open-source backend as a service is AppWrite: https://appwrite.io/

I've tried them out for some dummy projects and they seem pretty great, and they are continually iterating and pushing new features.

Potentially silly question (I'll go out on a limb anyway).

I would like to create a multi-platform app for non-technical users where their data is transparently self-contained/self-hosted. So, they open a single binary and it presents them with a UI, and the data created from within the app would be stored in their local PocketBase database. However, I wouldn't want the user to have to start a server directly (they wouldn't know what a server is).

So, I think the binary would have to start both the PocketBase server and a front-end server, and then launch a browser (similar to Electron) pointed to their front-end server's port.

Does this sound like a good approach? What could you recommend to orchestrate this seamlessly to make for a simple user experience when launching the app?

Thank you.

Honestly I'd look at using Electron because you can take a hard dependency on the browser capabilities and state. If you rely on the user bringing their own browser you're going to have headaches of people on ancient browsers or just supremely jacked up browsers (messed up browser plugins, proxies, turned off javascript, etc.). Node can easily launch processes and there are even fancy process management libraries like pm2 you can add as a dependency should you need it, so the logic in your electron layer can be very simple and just launch some other processes and ferry requests to the view layer.

Alternatively a webview-based framework or something like Lorca might be another option if you don't want to use Electron: https://maori.geek.nz/golang-desktop-app-webview-vs-lorca-vs...

Thank you!

Hm, there is nothing preventing you to start multiple services from a single binary.

I'm not sure that PocketBase would be suitable for your use case, but it is distributed also as a Go package (see https://pocketbase.io/docs/use-as-framework/) and you could combine it with any other GUI Go package (fyne, go-astilectron, etc.)

Thank you!

Checkout wails.io and tauri.app

:cheers:

Nice project! It looks pretty similar to my own LiteStore: https://h3rald.com/litestore/

The feature set is pretty similar, but I focused mainly on the free form/nosql-ish aspect of it, and although I added support for server-side JavaScript middleware I didn't include authentication OOTB (although it could be implemented or you can integrate with an OAuth2 system - it can validate JWT tokens).

And yes, LiteStore's web app hasn't been updated in years and overall your project looks much more polished than mine.

Good job! Bookmarking ;)

I notice that it has full text search support, too, which pocketbase doesn't seem to.

So good to see a renaissance of "portable apps", now on the web.

All of this extra complexity associated with multi-tiered multi-microservice arch is not needed for many categories of apps, and especially for small teams.

Congrats! What a lovely project. What a fantastic job!

Question: I am currently evaluating Ory/Kratos as authentication backend for a side project. Ory Kratos is a mature and excellent project. However, it has very little documentation when it comes to self-hosting. Further, I also feel like maintaining code that uses it down the line could is going to be problematic given its shear size and complexity.

How do you think Pocketbase compares with Ory Kratos?

How are the real time subscriptions with SQLite achieved? Is this a feature of SQLite itself or something that PocketBase itself implements on top of it?

It's implemented in the application, it's not a feature of SQLite, but its very simple and robust - when a record is created/updated/deleted I'm broadcasting the change to all subscribers that are allowed to receive it (the access control is based on the colleciton's list and view rules).

The realtime api implementation could be found at https://github.com/pocketbase/pocketbase/blob/master/apis/re...

That makes a lot of sense. Thanks for the link. I've not worked with Go before so I couldn't find exactly where the implementation was.

This looks like a very interesting project and great for some of the small client projects which I currently use Firebase. I'll be sure to take a proper look in the future!

Reading through the code/docs it looks like you can subscribe to individual records (database rows) or collections (tables), is this correct? In your Records.getList method, you have filter functionality that would be nice to subscribe to (ie. subscribe to changes in "demo" collection where totalComments > 10). Subscribing to a specific result set seems like it could scale better than notifying subscribers of all table changes, which then get filtered in the subscribe() callback func in the SDK. Very cool work!

Yes, you can subscribe to individual records or collections.

User defined filters for the subscriptions are not supported, but it's a good idea and I may consider it in the future.

Currently the subscriptions are filtered through the collection's list and view rules - they also act as "admin level filters", aka. filters that are always applied and regular users cannot modify (getList user defined filters are only appended to the search query together with the admin defined filters).

Really neat! I’ve been wanting to make something similar for _years_ to be a standardized data modeling testbed as I explore/experiment/prototype some new feature or problem space or even doing ad-hoc fixes. I think this is a much better starting place than spreadsheets/similar.

Good luck with the project! Seems like you’ve got some good interest :-)

I really like this. All my current projects use Go as a backend and I definitely need my own routes and business logic, so PocketBase looks way easier to integrate into my existing code than the JavaScript based solutions.

And it would also give me more time to focus on business logic instead of Auth - i will definitely try this in the future.

I would love it. But: postgres, +backups, +replica managment. Basically a managed database for postgres in a container.

You may want to check Supabase or Nhost in that case.

PocketBase was specifically designed to be self-contained and running only on a single server.

Check out https://thin.dev/ :) It's similar, supports self-hosting and uses postgres. Quick demo video here: https://www.youtube.com/watch?v=-jj19fpkd2c&t=3s

(I'm founder of Thin)

This looks perfect for storing the internal data of my side project! Love the website design too.

Sweet. How tightly is this coupled to SQLite? I’d like to embed Dolt [0] instead, specifically for the Dolthub-backed collaboration model.

[0] https://github.com/dolthub/dolt

Other than some json functions, PocketBase doesn't use too much SQLite specific features, but I don't plan adding support for other databases at the moment.

Please also note that PocketBase is designed to be deployed only on a single server, so it cannot scale horizontally out of the box. Supabase/Nhost are better suited for that.

This looks amazing, congrats on shipping! Do you plan to add Firebase-style query optimizations too?

https://news.ycombinator.com/item?id=31832266

Looks amazing! Love the idea of a Firebase/Supabase like thing I can easily run locally and deploy as well.

Also props for the dashboard design, looks and works very nice!

Well done! Really impressive, will give it a spin.

This is great, if it was available a year ago it would have saved me lot of time in creating my home-grown solution ;)

This is a really nice idea and I love the execution as well.

Congrats and thanks for sharing.

This is fantastic work. Supabase local development has always been a pain.

Member of the Supabase team!

Would love to hear your pain points with local development. Always looking for feedback so that we may improve your experience.

Well all the UI stuff. It is super easy to add a column and change something fast with the UI when doing rapid development. But that piece is missing when developing locally. I have gone on to create 2 projects (one called local and one prod) and manually applying the changes. Would be nice to apply changes from one project to another. PocketBase here for example provides the UI locally too. That is the real selling point IMO.

Thanks for the feedback! Passed it on to our frontend team.

This looks great! Does this have support for offline usage on the client?

Looks very interesting!

looks like document database, but how i can create indexes?

Currently you cannot create indexes through the admin UI, but it's in my todo.

As an alternative, you can always create indexes through DB migrations or add the index directly via the `sqlite3` cli.

thanks for reply! is it schema free? or i need to define scheme via migrations?

Its not schema free (although you can create a json field and put whatever you want in it).

You can define schema via migrations or through the admin UI (see https://pocketbase.io/docs/manage-collections/).

This actually looks suuuupppppeeeerrrr cool!!!

Any client SDKs for Android, iOS, React, etc.

There is already a JavaScript SDK (https://github.com/pocketbase/js-sdk) that works both in the browser and node.

Dart SDK (targeting mainly Flutter) is also on the way.

Unfortunately I'm not very uptodate with Android (Kotlin) or iOS (Swift) development and therefore community contributions are more than welcomed.

This is much more than I expected, thank you !

Suitable as the basis for a CMS ?

The existing admin interface could be used as a very rudimentary content management, but it was really intended for quick internal data explorations and edits. Additionally, currently there are no roles for admins so permission controls within the admin area are not possible.

You could always implement your own custom CMS SPA frontend that interacts with the PocketBase API (that is how I'm planning to use it with the next version of my other open source project - Presentator).

Maybe I open an issue for discussion. Mainly I wonder about making architectures mesh.

This is incredible.

why is this written in Go , are there specific features that shine over Python3? I've been looking to learn Go, it seems lot of network intensive software use Go.

If a selling point is run from "one file", that's quite difficult to do in python. There are things like pyinstaller but you end up shipping the entire interpreter in the bundle.

The pex tool lets you build single file python executables which are well suited for server deployments.

oh wow thanks i didn't know about this. in that case it would be super simple for me to wrap up a FastAPI server with SQLite or even some in-memory database that runs elsewhere, deliver as a single file!

You could also just use Docker to wrap anything

Yeah this kind of thing was common when I worked at Twitter (where AFAIK pex/pants was developed) on an infrastructure team. It's a cool tool that few outside of Twitter seem to be aware of.

How big is a python3 runtime when pex has packaged it up? And do you happen to know what the output binary is linked to?

As I recall, pex doesn't package up the runtime. It essentially packages up your venv, code, and resources and runs them under an installed runtime. It makes use of a relatively unknown bit of python functionality, which is that CPython will try to treat a zip file as a valid python script.

Reply on news.ycombinator.com