If anything, Grandma being the card-carrying pot user and not you would likely be evidence in your favor - to be presented and argued by a competent defense attorney in court, of course, not by you to some power-tripping cops interrogating you (the only correct response to a cop asking you anything for any reason at any time under any circumstance is for you to invoke your rights to silence and an attorney - and such a "shut the fuck up" strategy has indeed gotten people out of pot-related convictions, at least if these lawyers are to be believed: https://www.youtube.com/watch?v=sgWHrkDX35o).
Was I an opiate user? If I just keep the script but never fill it am I more an opiate user than the guy on the street with a little baggy of white powder? People seek out and obtain scripts for stuff they never use, all the time.
Given that they were prescribed to you, "yes" would be a reasonable assumption, barring you able to provide evidence introducing reasonable doubt (e.g. "I still have every pill I was ever prescribed").
In any case, "was" is the key word. You were given a prescription for a finite number of doses, whereas usually a medical marijuana card is more indefinite.