The compiler can optimize this. See https://gcc.godbolt.org/z/hxW7hhrd7

  #include <cstdint>
  uint32_t read_le_uint32(const uint8_t* p)
  {
      return p[0] | (p[1] << 8) | (p[2] << 16) | (p[3] << 24);
  }

ends up as

  read_le_uint32(unsigned char const*):
          mov     eax, dword ptr [rdi]
          ret

This works with Clang and gcc on x86_64 (but not with MSVC).