You do have to be careful, it can be annoying on ARM chips certainly. If you're using GCC, __attribute__ ((packed)) fixes alignment issues.
This method is so much less error prone than pulling stuff out a byte at a time. Plus you can use unions for network addresses, etc.
It's a simplified example to show what you can do.