That's an interesting point you raise. That outside the concept of infinite space there is nothing that is infinite. Even the universe is finite so talking about an infinite space isn't a valid concept. But at the same time the mathematical thought experiments have such profound effects on our lives.