1.) Keep two heaps, a min-heap for sellers and a max-heap for buyers. The top of the seller heap is the "ask price". The top of the buyer is the "bid price". The difference between them is the "bid/ask spread".
2.a.) When a limit order comes in, put it in the appropriate heap. Then compare the tops of the heaps. If the prices overlap, send a fill message to the parties. Pay attention to quantities; you may have to send a partial fill to one party and a fill to another, and then repeat the process with the next order in the order book.
2.b.) When a market order comes in, immediately pop the top of the opposing heap and send out the appropriate fill or partial fill message.
That's basically it. Depending on language, you're looking at anywhere from a page to a few hundred lines of code. An actual stock exchange is a bit more complicated, since it has to handle things like order cancellations, cancel-pending messages, transaction logs, and fault-tolerance, and has to operate on thousands of trades a second with minimal latency. But conceptually, it's simple.
(And if Joshua corrects any of this, listen to him and not me. He was an actual quant; I just worked at a financial software startup.)
Many exchanges are well documented, from internal workings to protocols. Check out XETRA, LSE, etc. NASDAQ too, but it's more complicated.
The problem here is going to be achieving regulatory compliance and then liquidity. Both of these are harder.