My most complicated build required somewhat of an SOA approach, but rather than go heavy-duty microservices and use grpc where you need a bunch of load balancing and service discovery involved, I went with something simple called nats, that ended up costing me 5 dollars a month, unfortunately.
Is the Go service directly accessible from Internet, or is it hosted behind a reverse proxy?
When you deploy a new binary, is there a small downtime between stopping the old binary and starting the new one?
How do you supervise the Go process? You use something like systemd?
When SQLite backup is ongoing, does it block writes to the database?
When you backup to S3, if an attacker gets control of your EC2 instance, is he able to erase your S3 backups, or is it configured in some kind of append-only mode?
Where do you store your logs and do you "read" them?
- The Go service is hosted behind a reverse proxy (nginx or haproxy) to enable zero downtime deployments, by 1) starting the new process, 2) directing new requests to the new process, and 3) gracefully stopping the old process.
- Since we've started to use Docker, we let the Docker daemon supervise and restart our services. Before Docker, we used systemd. Before systemd was available on our system, we used supervisord.
- We thought about using SQLite for some apps. But SQLite can only have a single writer at a time, which goes against the zero downtime deployment described above (two processes can be processing requests at the same time). Thus we use PostgreSQL (and MySQL for legacy reasons) which provides online backups. Must be noted that online backups are possible with SQLite, provided the application implement it using SQLite Online Backup API [1]. Another solution, which doesn't require application cooperation, is to snapshot your disk, if your system supports this.
- We backup to rsync.net, which provides an append-only mode, through their snapshot feature [2]. An attacked cannot override or erase the snapshots of your previous backups. I think it's possible to do something similar with S3, albeit in a bit more cumbersome way, using S3 versioning and MFA deletion.
- About logs, we're still not satisfied by what we use currently.
I'd be curious to read about what others do :-)
https://blog.sgmansfield.com/2016/06/how-to-block-forever-in...
The entire headache of choosing a web framework, generating a seed of magnificient complexity, wrestling with the javascript build ecosystem, figuring out how to balance state between JS ui components, JS state management, backend state management and databases. Not necessary. Do an SQL query, put that into a neat datastructure and run html templating, followed by <go build>. Maybe add bootstrap for some neat CSS.