If this happens to you often and you don’t feel good about it, I would probably tell the guests not to mention it in the reference (for safety reasons or for not making other people feel weird if you choose not to give them keys).
3 Likes