The first step is a system / block diagram. You need to encapsulate all the features and capabilities of the product. I mean, you won't know everything, but you should have a good idea. This will drive your prototype and your eventual bespoke hardware device. If there are pieces you don't understand whether or not you'll need them, you should go and test with your adoring audience (you have one of those, right?). You can test some of this with an early prototype, or a partial prototype.
For the prototype (really what I would call a proof of concept or POC), you should be working from off the shelf pieces, as much as possible. You're testing out an idea. Seeing your followup question, you ask about custom parts for a prototype, but you should be trying to do as much as you can without engaging manufacturing. This might involve a small custom PCB at some point, but if you can get away with soldering wires between breakout boards and a microcontroller platform, you should go that route. You really really need to validate an idea first.
Oh right, we have validated the idea...right? If not prior to this, then surely the POC exercise above. Well now you might start to think about making your first true prototype. You will be refining your idea and considering more design elements like industrial design and useability now that it's not a 3D printed case and a bundle of wires. You might engage a mechanical engineer, an industrial designer, an electrical engineer, a firmware engineer...or you might be doing that all yourself. It's up to you. But this is where things get hairy and expensive. With early prototypes, you will be making 10s of things and testing more.
Now your device is really tested (right??) and you're engaging manufacturing, and possibly going through yet another design iteration for cost down, best fit, and desiging your product so that it can be tested and you will have a high yield at the factory. You'll also need to consider regulatory testing, if your device has radios on it, and emissions testing pretty much regardless. If your product has custom plastics, you'll be enaging toolmakers and building out the entire assembly. If you have other elements aside from PCBs and plastics, you'll need to talk to wire harness manufactures, source auxiliary components, and discuss standard operating procedures (including testing) with your contract manufacturer.
Hardware is a long and arduous process. As a hardware design consultant, I have seen my share of messes personally. And if you watch the industry (or hell, just watch kickstarter), you will see people running into the same issues over and over again. Mostly that they price things poorly and they don't understand the depth of steps required to get a thing to the market. Having a well tested idea and a good audience will take you a good chunk of the way, but after that, make sure you're refining your idea many times; iteration is the name of the game.
If you're interested in just the electronics side of things, I show people how to design circuit boards and go through the block diagram -> circuit board manufacturing and testing component on my course. I show how to build a bluetooth and cellular board. If you're a true beginner, it's probably not the right fit for you. But if you've done some hardware and want to do a more complex project (or just see how it's done), this might be of interest: https://contextualelectronics.com/courses/advanced-ble-cell-...
1. How does one produce a prototype? e.g. if you're in need for some custom part - do you contact to some manufacturer/factory, provide them some CAD files, etc... ?
2. Same question if you decide to produce a batch - where/how do you approach manufacturers/factories?
Also, how to make sure you are not going to be ripped off - e.g. those manufacturers can just produce your product and sell by themselves?
As for protecting yourself. Any prototype house worth anything should be willing to sign a non-disclosure agreement. They aren't in the business of ripping off their customers. That is a good way to kill their core business and spend a lot in legal fees.
Now, is your custome part a mechanical part or electrical like an IC? Your product is likely to made up of a bunch of off the shelf parts with a tiny bit of your "secret sauce". The secret sauce could be mechanical, electrical, software or some combination. Your goal at first is to prove out that part. Often this is done using develepment boards all cludged together to something that looks nothing like the final product, but demonstrates the concept. Then it is the next step to create more custom parts (PCBs, 3D printed case, etc.) so it can be prototyped in something closer to its final form.
Yes.
There are a wide variety of shops, and I am just talking here about proto-typing.
If you just want some metal tool plate precisely cut/drilled/tapped to specification (CAD drawings), there's a bunch of shops that specialize in prototype runs. I've used emachineshop.com, it was super easy and the results were exactly to spec. I like to design assemblies with such custom plates using 8020 extruded channels (8020.net) that can be ordered in cut/tapped sizes. Usually overkill and not appropriate for a finished product, it's heavy and expensive, but very customizable and robust. Perfect for few-of-a-kind builds or system-integration work.
The same thing goes for sheet metal fabrication and front panels. There's frontpanelexpress.com, they make front panels precision cut/finish and with whatever graphics you want.
For basic PCB enclosures, I like Hammond Manufacturing extruded aluminum enclosures. They have customization options but I haven't used those (https://www.hammfg.com/electronics/small-case/mods).
If you're serious, I recommend you build some prototypes, see how things go, then partner with experienced folks to go further.
Counter-intuitively, it only gets MORE complex, expensive and time-consuming AFTER the prototypes.
Alan Cohen's Prototype to Product is the most comprehensive guide I've found.