Even with a fast home server Element is quite slow. Slow enough that even I, trying to get my circle to use it, feel some reluctance to open the app. Hopefully the upcoming Hydrogen client can solve this.
Have you tried fluffychat on android? Or any of the other clients https://matrix.org/clients-matrix