Writing a new protocol with an open source reference implementation will probably be a better idea, because:

1) Reverse-engineering isn't clean and can be circumvented.

2) BTsync has not reached a point where it can't be beaten by a competing protocol.