The same things that create utility in the hard sciences are a liability because they limit creativity. You put a lot a risk by looking foolish.
What I would say are real limits to creativity are the arbitrary and absurd evaluation metrics we use. Publish or perish paradigm encourages quick work rather than good work, which correlates with a decrease of novelty and substance. Makes the process more incremental. It also encourages railroading as exploration requires more time.
The review process also exacerbates these problems, as the system is not actually aligned with the goals. They are arbitrary noisy filters rather than quality filters, and we have substantial evidence that, putting it nicely, reviewers are good at identifying bad works but not good at identifying good works (i.e. default to reject). This causes one to continually rework a project rather than moving on or even making it better, as the feedback generally isn't about how to make the work better and each set of reviewers wants to add more and more.
The pursuit of novelty is also not only fundamentally anti-scientific but the basis of an existential crisis. To get funding or to pass review, we must be creative in convincing reviewers that we are clever while ideas generally far more obvious after they are revealed to you (regardless of lack of prior works). The need to innovate also prevents replication, which is the basis of science! It is far more difficult, often nearly impossible, to get funding or even approval to replicate works. Science is the process of ruling things out, not confirming things. Each replication has its own set of noise and biases, and this stochastic nature helps reveal hidden confounders. It is also quite essential to the learning process and failing to perform this task results in information being passed like a game of telephone.
But by nature, science is anti-authority and highly creative. You _must_ challenge pre-existing notions (trust but verify) and you must think outside the box. Though I do not blame anyone for missing this given the structures and incentives (shackles) the bureaucratic system has imposed upon us.
I would argue that creativity is where all science begins. What more is a hypothesis really, than a science fiction story that has yet to meet reality?
With all the resources going to academia, how much of them are focused on understanding the behavior of organizations? It's a reasonably tractable problem that's way underdeveloped, and would allow us to design new institutions that behave the way we want, rather than being beholden to institutions that having to tolerate the ubiquity of organizations with outcomes antithetical to the goals of the members.
The first problem to focus on could be assessing the effect of a decade-long induction process to be considered qualified to make the most miniscule contribution to academia.
I feel like I've noticed academics getting more and more cloistered, many of them rabidly defensive, completely shunning anyone with the slightest criticism of the process, credentialed and not alike. I'm guessing it has to do with (a) the thinning out of public resources, so more competition, more consequences to being wrong, more emphasis on sounding right rather than being right. (b) the fact that in most fields, the point of diminishing returns has been reached on the foundational axioms that define the field. And yet, defining a new field with new foundational axioms is extremely difficult, with scarce resources to explore it.
You could readily define new kinds of mathematics with different axia. The hard part is building off of those axia to make useful theorems. Some axia might lend themselves more readily to proving one theorem versus another, or might make it easier to model one thing versus another. We can see this with ZFC versus category theory. Similarly, analyzing fluids in terms of laminar versus turbulent flow yields distinct benefits. They rely on models with different core assumption. At the foundation of every field (or subfield) is a model & set of core assumptions that allows work to be built on top of them. Market economics versus game theory is a good example. The two models hold fundamentally different assumptions. Game theory is studied in economics departments, though, so its full potential isn't tapped. Any results from game theory can't too directly contradict what's come from market economics if it wants to get taken seriously in an economics department. If game theory had developed as a branch of sociology instead, we might see a very different ideas coming from it. No matter what, it can only be developed in directions is can get funding for.
If we really want to get at the truth, for every assumption we make, we should also define and investigate a theory that has the opposite assumption. For instance particles are indistinguishable. Great, that has yielded immeasurable results. Now let's fully flesh out at least one model that assumes they are distinguishable and hammer results out of it until they meet experimental predictions, too. We will likely find that there are some situations it can describe much more richly than our current models. And, that way people aren't going around forgetting it was an assumption to begin with.
"Science requires a substantial amount of creativity. In fact, they are necessary. The nature of breakthroughs requisite thinking outside the box and breaking from convention."
And if you do not have adequate depth in philosophy, you may find yourself parsing this seemingly simple phrase in a highly misinformative manner - as just one example: are you measuring things on a relative scale or an absolute scale? Also: are you actually measuring things, or are you....doing something much more interesting, that may not be obvious (and thus not taken into consideration)?
> What I would say are real limits to creativity are the arbitrary and absurd evaluation metrics we use.
Indeed!
> The need to innovate also prevents replication, which is the basis of science!
And also a substantial, rarely discussed drag on progress, across several dimensions!
> It is far more difficult, often nearly impossible, to get funding or even approval to replicate works.
That which is not studied, is unlikely to be understood, and this lack of understanding may register as NULL (as in: it may not be on one's radar, thus it "is" "nonexistent", as fans of science will assure you passionately, "because The Science says"). That science actually says the opposite is of no help - be it religion or science, once the Normie mind is captured, it's a goner.
See:
https://en.wikipedia.org/wiki/The_Adventure_of_Silver_Blaze
-------------------
Gregory (Scotland Yard detective): Is there any other point to which you would wish to draw my attention?
Holmes: To the curious incident of the dog in the night-time.
Gregory: The dog did nothing in the night-time.
Holmes: That was the curious incident.
-------------------
> The pursuit of novelty is also not only fundamentally anti-scientific but the basis of an existential crisis.
An ideological simulation, but also very true for science as it is imho.
> Science is the process of ruling things out, not confirming things.
There's also a substantial streetlight effect going on in science. Of course, such things are unavoidable to some degree, but not the degree to which they currently are in science, and there is also the issue of whether this is realized and disclosed, regularly and without aversion (as opposed to only when one gets caught telling some tall tales about science's abilities and on the ground performance, much of which is unknown and unknowable....maybe this is why science types tend to mock &/or hate philosophy and the skilful practice of it so much).
> But by nature, science is anti-authority and highly creative.
Including to it's own beliefs (including invisible metaphysical axioms) and authority?
Science, comprehensively, is what it is - and, what that is, precisely, is unknown. <--- I propose that this is not knowable to most scientific thinkers, including many(!) highly competent scientists.
Typically, the mind won't even allow the proposition to be contemplated, often only heuristic processing (guess according to one's sub-perceptual biases + post-hoc-rationalize, or downvote/delete-via-moderation/throttle) is available....all of which has very real consequences (though, we cannot measure them, thus they do not exist, so we are told and trained to think). Gee, what could go wrong with this approach!! (Possibly related: wow, there sure seems to be a lot of suboptimality in the world....could these two things maybe be related!!???? lol)
> You _must_ challenge pre-existing notions (trust but verify) and you must think outside the box.
Actually, you do not have to - only pretending* to do this is also very effective, and common.
What does limit creativity is having science a full-time job, as this does come out with some expectation of results. This is especially true for people pursuing academic carrier - a grad student might spend two years on failing projects and then shrug and leave for industry; but someone shooting for a tenured position better show successes.
Unfortunately I don't know how to fix this. If one is does not have to work for some reason, or at least has a high-paying job which leaves plenty of free time, they can do some research.. but that does not scale. And plenty of hard science research requires equipment which is expensive, dangerous, and large.
You're getting to my point. The thing about any research is that it is a lot of failure before success. Which makes progress hard to impossible to measure essentially until success is made.
Butt eh expectation of results is my problem. The tyranny of metrics. The fuzzier a goal is, the fuzzier the metric's guidance is. All metrics are guides btw. An academic's quality is extremely difficult to measure and citation, h-index, i10, and others do a shit job. The only way is to be subjective, because honestly only a researcher's peers know if they are doing good work or not, and that's even unreliable. I mean the job is literally to explore unknown areas. I'm not sure why anyone expects constant and big results. If you're constantly finding things you either hit a gold mine or you are staying pretty local and not doing much exploring.