This expression is only used by a man speaking to his wife, and would be considered improper if used for any other woman.