Do I know you?

I was talking to a guy I hadn't seen for ages earlier.

After a couple of minutes, I said, "You don't have a clue who I am, do you?"

He winced. "Sorry... What gave me away?"

"It was when you asked

Spoiler:

'How's that lovely wife of yours?'"