The "right to remain silent" comes from the Miranda warning, which is a simplified formulation of the constitutional right. Your "Miranda rights" are a superset of the rights directly guaranteed by the constitution. But they're judge-made (i.e. "activist judge") procedural protections which only exist in a narrow context. Importantly, the Miranda warning also says you have a right to an attorney. The notion is that until you have an attorney to explain your inviolable constitutional rights and to counsel you which questions you must, should, or should not answer, the court grants you a [mostly] unconditional right to silence.
The whole notion of Miranda rights is still highly contentious, especially among conservative jurists, and that's why an increasingly conservative Supreme Court has narrowed and carved out exception to Miranda rights. Because of the conservative exceptions, you should never just remain silent. You should politely ask for an attorney whenever a question is asked, but even then you might still get dinged for refusing to answer some types of questions. For example, if you're not yet in custody, or for simple questions like your name. It's complicated, which is precisely what the original Miranda rights were intended to safeguard against.
The irony is the right against self-incrimination, and many other rights copied from English law and enshrined in the constitution, were originally judge-made rules. By creating Miranda rights the "liberal" Supreme Court was following in the footsteps of traditional Anglo-American legal practice, and exercising their inherent powers. Courts have inherent, constitutionally protected authority to control what is and is not allowed to be presented in court. (Though it overlaps with legislative powers to control court procedures.) Almost all the rules for doing so were crafted by the English courts over nearly a millennium. It's not at all out of sorts for the highest court in the land to craft a new rule instructing lower courts that testimony is presumptively compelled if given before a Miranda warning and to reject it at trial.
To be fair, the conservative counter argument is that in England the highest court in the land was the House of Lords, which was also a House of Parliament, which was and remains the seat of legislative power. Procedural protections didn't always arise in the House of Lords, but the closer you get to 1776 the fewer instances there were of lower courts making such procedural rules. So similar to the Second Amendment, you can pick and choose a window of time that best support a claim to historical precedent.