Great question. I think it’s best you answer yourself with the questions: What’s the final result of all that daydreaming about sex? Where does it lead to? What do you accomplish when you’re spending time thinking about sexual scenarios?
Did Piers Anthony have a thing about daymares? Since you're talking about day dreaming in a "positive" sense I thought of negative day dreams. I think a better question is why are you fantasizing about whatever you are daydreaming about? Even the non-sexual stuff. Deciding it's bad doesn't tell you anything about your own mind, and if you knew something about your motivations you're in a much better position to deal with it. Bad is just a categorization with no insight into why it comes up.