If there's no downside to leaving the ladder in place, then I would think yes - there's a reasonable expectation that people will die due to your actions. You'd likely have to argue about "involuntary manslaughter" vs something more intentional though, depending on circumstances.

If there is a reasonable downside, probably no? You have a right to try to keep yourself alive, in nearly all contexts.