You could generate a CSRF token or something similar in a hidden filed based on a JWT token (yes I know) on the server side. The JWT token can either contain some timestamp after which it's valid or the time it was created.

People will be able to write programs that circumvent the feature but that's also true for the javascript solution. The point of it was that it gets rid of most spam because most bots fill in the form faster than 4 seconds and are not made to circumvent this feature.

Bots can also circumvent this JS thing, so it's the same either way.

    <?php echo '<input type=hidden name=starttime value='.time().'>';

On submit:

    <?php if (time() - $_POST['starttime'] < 4) die('2fast4me');

Revealing the error condition (that it was submitted too fast) is nice for users and bots alike, of course. Up to you.

I've had websites where I was too fast in submitting a form before. Not any kind of anti-spam, just their server was so fricking slow that I had input the date (iirc it was a reservation system) and clicked next before the JS blobs had finished triggering each other and fully loaded. It would break the page somehow with no visual indication. I found out by looking in the dev console and noticing stuff was still loading in the background. How normal people are able to use the Internet with how often I need the dev console to do entirely ordinary things is a mystery to me.