If they need your iPhone address book they can get a search warrant.
While I agree that's our right as protected by the 4th, some states have determined that only probable cause is required to confiscate and search cell phones (e.g. California). In those states refusing to consent will not help (except of course if you plan on taking the case to SCOTUS).