It's pretty obvious from the context and you don't need Paul to get there. I am going to quote from the paper (which I wrote and linked in another comment) on this subject.
The story of the fig tree takes place within a Markan intercalation. Jesus’s entrance into and cleansing of the temple is sandwiched between the cursing of the fig tree and the observation that the fig tree has withered. Edwards’ central thesis is that “the middle story nearly always
provides the key to the theological purpose of the sandwich.” The fig tree is best interpreted through the lens of the temple.
In the middle of the cleansing of the temple, Jesus declares “Is it not written, ‘My house shall be called a house of prayer for all the nations’? But you have made it a den of robbers” (Mark 11:17, NRSV). The quotation is from Isa 56:7, which takes place amidst an eschatological vision of the covenant being extended to Gentiles. The failure of the second temple to adequately address Isaiah’s vision suggests that it is not the ultimate fulfillment of Isaiah’s promise. The
phrase “den of robbers” comes from Jer 7:11. Kirk comments that word den is “aptly chosen to represent the practice of a people who see the temple as a place of safety to which they can flee after committing various forms of injustice in other spheres.” Thus, the leaders who are in the temple are not merely condemned for their practices in the temple, but for the way in which they assume participation in the temple system guarantees their forgiveness and safety regardless of
their unjust actions elsewhere. They are not the sort of people Isaiah commended, who “maintain justice, and do what is right” (Isa 56:1) but rather seek their own profit and work to kill the Messiah (Mark 11:18).
This is why Jesus’ teaching in response to Peter’s comment in Mark 11:19 focuses on the destruction and replacement of the temple. His explanation in 11:20-25 focuses on prayer and how “this mountain” (i.e. the temple mount toward which they were heading) may be thrown into the sea. The use of the demonstrative pronoun τούτῳ in reference to the mountain indicates that Jesus was referring to this specific mountain, not offering a discourse on the general power of prayer. The mountain serves as a metonymy representing the entire temple system, which is to be overthrown; the fig tree serves as an additional symbol of how this temple system will be
cursed. To further clarify, Jesus proposes an alternative pathway to fulfill the function that the temple performs. Rather than perform sacrifices in the temple to receive forgiveness from God, the disciples need only pray and forgive others. This alternative route to forgiveness indicates the
temple system is no longer necessary.
Shortly following this sandwich is a story with parallel language that also illuminates the meaning of the passage: the parable of the tenants (Mark 12:1-11). The two words καιρὸς and
καρπῶν are found in both passages. Few commentaries have focused on the use of the word καιρὸς, likely because it is such a common word. However, a closer look indicates that it is used in the gospels primarily in two ways: either as a common phrase (e.g. Ἐν ἐκείνῳ τῷ καιρῷ in Matt 12:1) or with an eschatological reference point (e.g. ἄχρι οὗ πληρωθῶσιν καιροὶ ἐθνῶν in Luke 21:24). Unlike the other synoptics, Mark never uses καιρὸς as a common phrase, which suggests that there may be greater linkages between passages where he uses the term. The placement of these words so closely to one another would likely raise more commentary if the fig tree was not already part of a Markan sandwich. Specifically, the themes raised in Mark 11:12-25 continue to be raised in the subsequent chapters in complex ways, such that the fig tree may be part of a “double intercalation” or triple decker sandwich.
Needless to say, I was very hungry after writing this paper...