This could be easily conceived as a break between you two as you spend some time apart from each other. He might even go for an incall with an escort. Set some ground rules to be followed and stick to it.