My prof spent 8 whole months teaching us systems modeling and PID mathematics only to tell us there's no reasonable way to set your PID values... You gotta guess and check!
The mentioned use of optimal control is one, although if you go there you need to be careful about setting your costs -- LQR and other optimal schemes assume a perfect model of the plant; the higher you set Q the higher the probability that your system will ring or go unstable.
There's various robust control methods, all of which I haven't used in a Good Long While, because for the most part swept-sine measurements work nice.
The one I've used most involves taking swept-sine measurements to get the plant response in Bode plot form, then using either Bode or Nyquist plots (or both) to tune my PID (or whatever controller I'm using).
For a large class of industrial problems, swept-sine measurements won't do, either because measurements need to be undertaken on production lines that are in operation, and the operators get cranky about things like noticeable sinusoidal variations in the product (think aluminum foil or paper), or because even when operated within safe limits, large machines undergoing swept-sine measurements can be downright scary. In such cases one usually ends up using random excitation or steps (often called "bump testing" if you're in an oil refinery or a paper plant) and some sort of system identification step like ARMA.
If you do end up doing testing followed by system ID, you'll most likely get an approximate plant transfer function -- so it's wise to either use a grain of salt when doing your optimal design, or to use some robust design method or other.
It’s also certainly true that a LQR won’t help you if you have no a priori knowledge of your system, but for many of the mechanisms we need to control this isn’t a problem.
Also from my experience you still need to tune the Q and R the same amount that you would P I and D. and I'm not convinced that it is strictly better at all. In industry at least PID absolutely dominates.
It's also important to realize that just because the LQR is derived by solving an optimization problem, that doesn't mean it gives you the best possible gains for whatever you want. You got the optimal controller to minimize a quadratic cost you made up for a linearized approximation of your system.
Iterative design (guess and check) is absolutely still the state of the art for control design, and using an LQR does not escape that.
Oh, you system model isn’t first order? Now you need an estimator/Kalman filter. Or more sensors. That’s just more complexity for questionable benifit, which is why lqr is beloved by academics [0]. For anything that would be adaquetly controlled with pid, stick with that. After about 2 hours fiddling with the knobs, it’ll be close enough.
This whole idea of optimality is based on bad intuition. Which states do you care about? Why? Is that more valuable than control effort? Why? Who is doing the economic analysis to determine what ultimately costs us more money? In the end, this thing are tuned just like pid: you stop when the step response looks nice. Besides all that, for a lot of systems, and in particular flexible structures with a lot of states, “penalizing state excursions” isn’t really useful intuition to begin with for almost all state space represtations. You are better off with a pid and notch filter.
[0] I decided to complete my PhD in controls so could make statements like that with at least marginal credibility.
If I'm missing something in the formal background, though, I'd love to hear it. My own implementation of digital PID controllers have been based on using Ziegler-Nichols for tuning bc I was under the impression above. Any literature or methods contrary to that would definitely be welcome!
(Edited because my first line didn't make any sense)
[1] https://pages.mtu.edu/~tbco/cm416/tuning_methods.pdf
[2] https://en.wikipedia.org/wiki/PID_controller#Loop_tuning
B. Adjust P for 1/4 wave damping , Adjust D to improve response , Adjust I to correct Offset
Repeat B. Repeat B. Repeat B. Repeat B. Repeat B. Repeat B. . . . .
Many processes like temperature controls want to settle, you just have to help them out a bit. Applying a full PID algorithm to every control problem is a bit like using TCP/IP to solve every networking problem. It will work, but you'll often do better with a domain-focused solution.