You're totally allowed to publish client software that has the URLs of the official servers in the code, including that forked from the official client. Believing otherwise is a common (and false) misconception that I wish people would stop repeating. (However, if you do fork the program,
pick a new name! "Signal" is trademarked, don't use it in your name.)
The ToS for the signal service API applies to end users who connect to that API. The ToS of the API does not apply to software - the AGPL license applies to that, and the AGPL permits anyone to make forks (including ones that are configured to talk to the official servers).
EDIT: Disregard the following paragraph. See below.
There is a little bit of an issue, though, because contributions to Signal's upstream are not accepted unless the contributors sign the CLA, which allows Signal to relicense future versions more restrictively as they wish. They could relicense and then introduce intentional incompatibilities in the protocol if they really wanted to harm forks, but that would make them non-free, non-open-source software.
EDIT: Turns out I'm wrong in the above - the Signal CLA, to their immense credit, only permits relicensing under OSI-approved free software licenses. Fork away!
Moral of the story: don't listen to bluster, stick to what the license says (and the AGPL says you can fork), and never sign a CLA. You don't have to sign a CLA to contribute to the Linux kernel, and you shouldn't ever sign one to contribute to any other open source project.
I also wonder if it's even possible to enforce an "official clients only" restriction in the ToS for the service API. This would be a little bit like Google claiming that you aren't authorized to use Firefox to access google.com (i.e. insanity). Technologically if you tracked upstream closely enough, they probably wouldn't even be able to tell if a given 3p client user is running your fork or the official one.
(This whole comment is a paraphrase of a blog post[1] I published just a few hours ago that tries to dispel this whole "but you can't fork Signal and use the official servers" false meme.)
[1]: https://news.ycombinator.com/item?id=26940342