Have you tried passing everything by value? That is, instead of:

    bool hit_sphere(const struct sphere* sp, const struct ray* ray, struct hit* hit)

you write:

    static bool
    hit_sphere(struct sphere sp, struct ray ray, struct hit hit)

IME, clang is insanely good at optimizing pass by value calls.