The first thing to do is to ask oneself if it's worth completing the project at all. All stakeholders _will_ have to be ready to make some sacrifices; is it worth it? Also, like others have said, communication is key. If the team can't have open conversations up-front, it may be best to walk. But, if the team can communicate openly and most are on-board for an adventure, then there are things that are possible.
This is also where a team 're-org' may have to happen, unfortunately. At this point, there may be a huge amount of internal tension that's probably slowing things down. This is a very difficult task, but identify those who have become burned-out and/or antagonistic. Sometimes it's better to cut the team down from 5 to maybe 2-3 instead of adding more people. Is there a 2-3 person team that works most effectively together? The others who are 'cut' will have bad feelings about this. If possible (it usually is), get them on part-time contract and affirm that they are important to succeed (they probably are). Let them cool-off and slow down and mediate conversations with them to use their existing knowledge most effectively.
Don't think about the sunk costs. This will just depress you. Instead, think about all that you have (e.g. knowledge, code, documents, connections, relationships) as resources to help you get to a destination.
Redefine the destination. Take an honest inventory of your current state and distance to the destination. Where can you realistically get to, with a best effort? Which promises will have to be broken? Which ones must be kept. This is going to irk some people, so it's going to have to be very calculated. Sometimes this means just finding a way to get the ship to a close port, if only just to refuel.
If there isn't one, stand up an automated test harness with as many tests as possible, in the shortest period of time. If you can find a 3rd-party tool that's easy to set up, use it. But, you may have to write some throwaway code just to get something working. Don't worry if it takes a day or two to get going, it will be worth it. Shoot for broad tests, rather than unit tests, at first.
Divide and conquer; Isolate and build out from the solid parts. Which parts of the system can be hidden behind a black box interface and add value? Which parts of the system are so rotten that they have to be re-written? Slice the system up into pieces and remove the bad ones. Unfortunately, this requires a bit of skill, art, and talent. I most often deal with systems in OO languages. So, it's quite easy to apply a refactoring interface/factory pattern to things. The really rotten parts can even be replaced with mocks for the short-term, if an interface can be defined. Another thing: the Pareto principle. Re-write the 20% of the system that is creating the 80% complexity.
Aim for early, easy wins. Shrink the development cycle. Show some forward progress. That will help revitalize the team and add momentum.
The gut feeling for a lot of people in this type of situation is to 'brute force' it. People get addicted to 'putting out the fires' But, that's completely the wrong path to take. Come up with a new plan. A plan that can be done with everyone working 40 hour weeks. Let some of 'the fires' burn, but contain them. Don't lose hope, and instead use brains.
However, in the end, it is also completely ok to walk away, especially if there's no incentive to continue.