I Get so annoyed when I forget to toggle back on "silence all unknown callers," especially when I'm driving/listening to music in the car and oh hey it's call yet not for me rather some scumbag is trying to steal ppl's money.
Though I only turn it off when I'm expecting a call. If your app can figure out when I'm expecting a call and when that call is done then toggle silence all unknown callers on then I'm sold.