Are you seriously suggesting that society could operate on system where people just work at any time of the day whenever they felt like it?
If an office worker walked into a construction site and declared they were doing everything wrong from the lack of paperwork documenting completion of every shovel of dirt they would be called a clueless, arrogant ass. Why would it be any different if the subject was a construction worker in an office building declaring it unproductive for a lack of tonnage of paper relocated?
We're discussing flexible working times for all of society, and what I'm reading is people who somehow assume that most of society is in a job with privileges like being able to work from home or on "async" schedules etc. Few jobs are like this.
Come to think of it, many of my high-school teachers worked part-time too. They didn't get to dictate their own hours, but they didn't work 7-15 every day either, some days they came in at 11 and other days they left at 12.
There are cultures where you just close up shop when you've had enough income for the day. Not to mention rural communities all around the world that are more flexible about timekeeping in general.
I'd say all the services which require face to face interaction will become way more costly if such buffers will be needed to cover off hours. For example: there is a notion of skeleton crew and the service might not be operational without that crew so the choice is actually to spread the cost of off hours low load to the high load hours or close the business for low load hours.
If there is a will, a structure for all work can surely be created where even more intermediation happens and productive artifacts are exchanged rather than physical presence. This applies much more broadly than so-called "knowledge work", it just happened first for knowledge work because it's most obvious and low-cost there.
Very few jobs in society are compatible with flexible time where people can work when it suits them. Any job which requires some kind of real time coordination with others will conflict, which is most jobs. That is not even counting parents with school age children.
Most tasks with real person in time coordination also have fixed locations, and optimal environmental work times. Roofers work early in the morning so they don't need to literally risk their necks to darkness while also avoiding scorching heat.
Lighting was also expensive and very inadequate compared to sunlight. This lead to convergence. Notice all of the old proverbs and parables speak in terms of light and not time on clocks?
In addition to the manual craftmanship, before standardized parts creation was done to fit by one person - something where you would prefer to do all of the relevant tasks with one person when possible. And manufactured goods were expensive precisely because most spent time on raw material related tasks.
Actual coordinated work has music associated with the proper rhythm. Sea shanties, and rowing drums for instance.