Look, scrolling needs to have a purpose. Either (a) it's because you want to read more text than is visible, or (2) it's because you want to view more photos that are not currently visible. It's never both at the same time, in your case, simply because multiple photos cannot exist on the same screen as an overflowing text box.
In the case where no text box is being shown, the intention (1) does not exist, because there is nothing to scroll. So when viewing photo A1 (being the first photo in series A), by sliding horizontally I am expressing the intention: "Let me view photo A2". When viewing photo A2, which does show any text box yet, by sliding vertically I am expressing the intention: "Let me view photo B1".
There is no text box yet. Therefore, by scrolling down the app cannot possible perceive my action as intending to scroll the text. And yet your app decides that's what it means, and shows the text box out of the blue. It's simple causality. If vertical scrolling is to fullfill the intention "Let me view more text than is currently on screen", then the text box needs to be visible at all times. Simple as that.
Your scrolling thing is still equally broken for the other reason I stated in the previous comment.