I'm not on MacOS but find Inkscape pretty snappy in general, even with big projects.
Perhaps it's not the Layers but the effects? Blur in particular is pretty slow. Have you tried reducing the displayed filter quality? This doesn't effect the image itself, only the display in Inkscape. It's in the render tab in Inkscape preferences.
At View -> Display Mode you can also toggle between full display, display without filters and only outlines. If Inkscape is significantly faster without displayed filters, setting down the display quality will definitely help.