If you are a US citizen, they are not entitled to your unlocked phone. They need a judicial warrant if they want you to unlock your phone with a PIN/password. It is settled case law that that falls under your protections under the 14th Amendment.

They can take it for a "reasonable amount of time" (inconvenience you for a few hours and make you miss your connecting flight) while they copy an encrypted image. They then must return it to you.

But did they? Was there an actual investigation going on?

I'm not sure this is 100% true. Courts have found that in order for the exception to the 4th amendment at borders to apply the search has to bear some relationship to the purpose of securing borders. For example

"to conduct such an intrusive and nonroutine search under the border search exception (that is, without a warrant), the Government must have individualized suspicion of an offense that bears some nexus to the border search exception’s purposes of protecting national security, collecting duties, blocking the entry of unwanted persons, or disrupting efforts to export or import contraband. See also United States v. Ramsey, 431 U.S. 606, 620 (1977) (“The border-search exception is grounded in the recognized right of the sovereign to control, subject to substantive limitations imposed by the Constitution, who and what may enter the country.”). If a nonroutine search becomes too “attenuated” from these historic rationales, it “no longer [will] fall under” the exception"

-- US v. RAYMOND IDEMUDIA AIGBEKAEN (https://law.justia.com/cases/federal/appellate-courts/ca4/17...)

Of course whether border agents know or care about this is a different story and by the time you are quoting precedent, you're probably beyond any of this being helpful.