Yoga and relationship

Hello
I have arguments with my boyfriend about the subject of yoga. He says I’m going too deep into this and says yoga is a religion. I said that the person chooses whether to take only the physical aspect of the yoga or beyond. So he says what do you choose? I said that for me yoga is beyond the physical practice and he is not pleased with it.
How should I respond?
Thank you