In addition to the other advice here, regarding how to handle this politically and from a communication perspective, I think you need to quantify to what extent you're not going to make the release.
- You need to write down all the things that need to be done, the granularity of task should be between half a day and two days. (Anything larger than 2 days will be a task like "refactor this system" that hasn't been though through enough, and thus likely to be completely wrong.) Do this in a meeting with all your team members, they will be telling you what they foresee they need to do.
- If you can't estimate things, because e.g. the system design hasn't even been started yet for those things, you need to design them, at least on a very high level. So that you know, at least to some extent, how long you think they're going to take.
- Write down estimates for the aforementioned tasks, in person-days, by getting your team members to estimate these tasks.
- Ideally use some kind of task planning system such as MS Project, LiquidPlanner, manuscript.com or some JIRA plug-ins that can handle this sort of stuff. But even a Google Spreadsheet shared between your colleagues will be better than nothing.
- Now you'll know an end date. How far out is it? 4 months rather than 3? 12 months rather than 3? That makes a difference. If you're going to be late, you management may wish to give you an extension. If the project's going to take another year, management may wish to cancel or change the requirements significantly. You have to provide them with the information to enable them to make that choice.
- If it's decided to continue, then update that spreadsheet all the time. Get your team members to update it, and if they don't do that, then talk to them every few days and get them to tell you what they still need to do. Focus only on the future: remove tasks when they're done, add new tasks as they arise, change estimates on tasks as they're partially completed. So even if you decide to continue, maybe in one month it becomes apparent you haven't made as much progress as you'd like, or there have been unforeseen difficulties resulting in more tasks. Know if it's going off track again as soon as possible.
Estimate will be wrong. Things always take longer. This is a minimum bound. But if your spreadsheet says you need another 12 months on a project that only has 3 months to go, you can be sure that it won't be done in e.g. 2 months, that's useful information to have.