Can you say to a person 'you would have been handy' or is that rude?

No it is not rude. You would have given him a laugh,though, had you added some circumstances.......when the phone was ringing, the baby crying and his milk boiling over on the stove.