Urgh, yes. This is a really difficult place to be coming from, and I totally feel your pain!
My background was working as an SRE at GoCardless, starting when the company was around ~30 and leaving at around 700. During that time we did the whole "oh crap, wtf is observability" that coincided with a big push in the industry to define the term, and I worked on the team (Observability Working Group) that tried rolling out these practices.
The truth is this is much easier if you have someone with you who knows what good is, though I was in your position at GC and it's possible to learn it by first principles.
If you're doing this, the best advice I can give you is to think really critically about _why_ you want observability.
Usually it's "when something goes wrong, I want to be able to understand what lead to it, and what was going on at that time". If that's the case, you can't make a wrong step if it improves your ability to understand that- even if what you do is simple.
At GC we began with logs, as everyone was familiar with them. We encouraged people to start thinking about logs as structured data, so drop the "Posted message to Slack" log line and go for something like:
```
{
event: "slack_message.posted",
slack_channel_id: "CH123",
slack_user_id: "US123",
etc...
}
```
When you get your logs looking like that, you can setup something like Grafana to expose visualisations that are built from your logs. We were using ElasticSearch for log storage, which is quite simple to build graphs on top of.
Visualisations are really compelling, and help you persuade people it's worthwhile to consider this stuff.
Beyond structured logging, you'd want to look into time-series metrics (Prometheus) which can help you monitor things in a bit more real-time, then traces if you want that type of insight.
I've often compared observability to testing, in terms of how you should think about it/use it. You'll find a load of dev teams who think testing is a waste of time, but most high performance teams won't ship without tests.
They'll say testing doesn't just help catch errors, it helps them build faster, due to the confidence it gives them.
You'll know when your org has adopted observability when they feel that way about instrumenting their code, and it's second nature to write log/trace/metrics into their software.
Not sure I have any reference links in mind just yet, but I'll give it a think.