Two people being kept apart by circumstances not within their control. The singer is thinking that he's wasting time just doing nothing...when he could be with the one he loves.
I think it's a beautiful song, very expressive.
Now...Saturday Night's Alright For Fighting is the one I don't get.