A thousand times this. The retrospective blows all the other elements of $YOUR_FAVOURITE_PROCESS out of the water.
3 things about it: if you do sprints, and they have busy beginnings and endings, then there's no harm in planning the retrospective halfway the sprint instead of at the end. Improvements can be made at any time. Also, if your sprints are weeks, consider making the retrospective bi-weekly, that's usually enough.
Second, don't skip the retrospective. It's never urgent, but it's very important. Every team is different and every situation is different, and no agile book can cover all cases. The retrospective makes you own your process, makes the entire team feel like their opinion matters. It improves both effectiveness and morale. Give it time - especially in the beginning it might last 2 hours easily. It's worth that.
Finally, do the retrospective right. You want actionable results out of it. If it's just a complaining session, you're missing out on its value.
What I usually do is this: I make a poster with 3 columns, :-), :-/ and !. Every team member has to make at least 2 green and 2 red sticky notes, on which great stuff and improvement points are written, respectively. More is OK, less is not. This forces them to really think about what could be better even if on first thought they think things are going pretty OK.
Make every team member put the sticky note on the board in the left two columns and explain what it's about and why it matters. Have people group sticky notes that are about roughly the same thing. Finally, and this is the part people tend to forget, distill a bunch of action points for the third column that will help tackle the problems in the 2nd column. Make these actionable, assignable. You might want to make them tasks in your scrum process, just like programming tasks. Without action points, a retrospective is mostly a waste of time.
If a discussion about one action point is long or very technical (discussions about how to use source control come to mind), ask the people who care about it to discuss it amongst each other present the solution a day later. This works great for distributed teams too ("Problem: the git is becoming a mess. Solution: $NAME makes a Slack channel and presents the outcome tomorrow"). Everybody who doesn't involve himself in that discussion is expected to agree automatically.
You can't improve the process super much in a short amount of time. If you make too many action points, half of them will not get done - people need to make software, too! So if there's, say, >4 real decent problems in column 2, ask the team to score problems and pick the ones with the highest score. One good way is to give everyone 3 votes that they can arbitrarily spread across the problem stickies (multiple votes per sticky allowed). Pick the 4 problems with the most points.
Consider using real stickies and not computer stuff, even if your team is partly remote. Tangible things that you move across a board somehow makes things seem more real, like they matter more.
Good luck!