A curious case of customer support gone awry: when AI support software leads to a mix-up
The CEO of a note-taking app company, Obsidian, stumbled upon an intriguing scenario involving a video game firm with the same name. The story revolves around a customer support issue that highlights the potential pitfalls of relying solely on AI for handling customer queries.
Steph Ango, the CEO, shared an email in his social media post, revealing that Obsidian Entertainment's support team had mistakenly directed a user to the wrong company's support address. The user, a player of Outer Worlds 2, was attempting to provide feedback on the game, but encountered a two-factor authentication (2FA) system that seemed to block their path.
The email, which Ango provided, instructed the user to contact the 'specialized account security team' via a specific email address. However, this email address belonged to Ango's company, not the gaming firm. The email also requested the user to provide details about the issue and any proof of account ownership.
Ango's suspicion that the email and its incorrect address were AI-generated sparked a discussion on the potential dangers of trusting AI with customer support. He questioned whether the issue was related to reinforcement learning, where the reward function aims to reduce the number of support touches. This scenario raises concerns about the reliability and accuracy of AI-generated responses.
This incident is not an isolated case. The article mentions other instances where AI support software has led to similar mix-ups, such as the X platform's passkey reset feature and the Have I Been Pwned service's ban on resellers. These examples underscore the importance of human oversight in customer support to ensure accurate and secure interactions.
The story also highlights the potential risks associated with AI-generated emails, especially when users might inadvertently share personal or sensitive data to prove account ownership. It serves as a reminder that while AI can be a powerful tool, human intervention is crucial to maintain the integrity and security of customer support processes.
