"The big issue is really good P&R with working optimizer for anything bigger than toy designs and chips. It seems Cliff has done some real progress yes."
Yep. The only one I know about is VPR:
http://www.eecg.toronto.edu/~vaughn/vpr/vpr.html
Another angle on the topic listing specific tools and issues:
http://people.kth.se/~frobino/foss/presentation.pdf
You might want to test to see if Qflow/Yosys and/or VBR can handle your designs. Need to know where they're at in terms of usability and performance to identify what improvements remain to be done. Also, crypto algorithms are usually small enough to fit on the Lattice FPGA's. So, try Cliff's IceStorm synthesis flow for them. Would be great if that worked given their take-up in embedded and how many algorithms could be put on them.
"What would you say would be the best path for Cryptech to try and go open with the flow? Alternatively, to mitigate black boxes in the FPGA of commercial chips?"
"I've toyed with the idea of mapping a synthetic FPGA into a commercial FPGA and then implement the real use case core in the synthetic FPGA."
That was the recommendation I wrote last night in another comment haha. To be honest, given poor demand, I'd actually recommend you do nothing for open hardware. :( If you're ideological or brave, there's basically three paths you can go down: cheap ASIC w/ MPW's over time; S-ASIC; FPGA clone. Let's look at them in reverse.
Cloning Xilinx/Altera FPGA's was one of my first ideas, too. The concept was that their HLS or place-n-route tools can get it going in an efficient way. Then, OSS tools can suck the bitstream out and re-apply it to my comparable device. A combination of QFlow w/ VPR could be used for open alternative directly to my device.
The biggest risk here, other than NRE costs and tooling risk, is lawsuits from Xilinx or Altera esp over patents. They could make an ASIC look cheap. Still, maybe cheaper per unit if a royalty was negotiated given outrageous unit prices of commercial FPGA's. And you have a box you can trust and use for other products, even license to third parties for onboard FPGA logic in tehir SOC's. Could go quite a long time without a lawsuit while keeping the FPGA part a NDA secret.
Also, worth considering the security advantage of controlling how I.P. got onto the FPGA to prevent external attacks or using anti-fuse variant of Xilinx/Altera to maintain integrity. Software attacks are the more practical concern. I'm not convinced Big Two do enough on that. So, having trusted loading or secure interfaces to FPGA could be justification enough to design one given your market.
Alternatively, fund more academic development of something like Archipelago w/ VPR etc to target it. Keeps anyone from claiming you're using their stuff outside of occasional patent suits which happen anyway. Can also keep the complexity and cost at levels you like rather than what Big Two think is marketable.
The next option, Structured ASIC's, are quite under-utilized. They're basically FPGA's where the routing is done with a fixed layer in ASIC that vendor generates during conversion. That it's only one layer makes it way cheaper than ASIC of same size. That the wires only connect to blocks you use means it goes faster and uses less power. That it's an ASIC, not FPGA, means lower unit prices. Vendors that do this include eASIC's Nextreme and Triad Semi's VCA (their analog blocks can be TRNG's btw). I know eASIC does maskless prototyping for low cost plus has lots of 3rd party I.P. on theirs already. All I'm saying for S-ASIC is get some estimates for specific NRE and volume as jury is still out on whether they're a good idea. Also, make sure your FPGA HW is designed for easy conversion to ASIC up-front as it's not as easy as they promise. ;)
The last option is Standard Cell ASIC. The cheap way of doing this is to figure out the absolute least amount of features (esp I/O interfaces) you need to get the job done on the oldest process node that can handle them. Then, just gradually move I.P. onto it piece by piece with multi-project wafer runs to keep costs in check. The 180nm and 350nm nodes are popular right now because masks are cheap due to fully-depreciated equipment and most bugs having been worked out. I believe the fabs will even hold your mask for you so you can keep making more chips from it over time. Curious what that costs. Anyway, Tekmos has MPW offerings at from 350nm-65nm with a 2.5D option that combines 180nm/350nm logic with 90nm flash memory. Never used them so unsure of cost/reliability.
In any case, your options all cost significant money if you want to hold a chip in your hand that you built. The costs, whether Standard Cell or S-ASIC, are lower than they've ever been. Homebrew FPGA on a node like 45nm (90nm minimum) with academics and their cheap tools doing the development seems ideal as it will keep paying off. Can always use a 3rd party tool for synthesis if OSS doesn't cut it. Otherwise, can put your crypto on ASIC piecemeal or see if eASIC, etc can do it cheap. And safest of all: don't do shit about the problem and let your customers worry about it until they pony up the cash to let you solve it for them.
Wish I had specific recommendations that cost you a new car instead of a house or two. But this is HW development after all. Hope some of this helps in your own explorations on FPGA's or developing your offerings. :)