I am testing it on an AWS instance and the speedup effect is not as consistent as I hope. The speedup varies between runs.

Intel Xeon Platinum 8259CL CPU @ 2.50GHz 128 GB RAM Tesla T4

  ./main -t 12 -m models/gpt4-alpaca-lora-30B-4bit-GGML/gpt4-alpaca-lora-30b.ggml.q5_0.bin

  llama_print_timings:        load time =  3725.08 ms
  llama_print_timings:      sample time =   612.06 ms /   536 runs   (    1.14 ms per token)
  llama_print_timings: prompt eval time = 13876.81 ms /   259 tokens (   53.58 ms per token)
  llama_print_timings:        eval time = 221647.40 ms /   534 runs   (  415.07 ms per token)
  llama_print_timings:       total time = 239423.46 ms

  ./main -t 12 -m models/gpt4-alpaca-lora-30B-4bit-GGML/gpt4-alpaca-lora-30b.ggml.q5_0.bin -ngl 30

  llama_print_timings:        load time =  7638.95 ms
  llama_print_timings:      sample time =   280.81 ms /   294 runs   (    0.96 ms per token)
  llama_print_timings: prompt eval time =  2197.82 ms /     2 tokens ( 1098.91 ms per token)
  llama_print_timings:        eval time = 112790.25 ms /   293 runs   (  384.95 ms per token)
  llama_print_timings:       total time = 120788.82 ms