Is it possible to have Take Your Daughter to Work Day backfire?

If you're not screwing around with other women then you have nothing to worry about. Girls talk, and if it doesn't have any substance it'll eventually blow over (if your wife decides to become concerned).