How do you get an AI chatbot to challenge an idea without arguing for sport?
Replacing a chatbot that always agrees with one that always objects sounds like hiring a different nuisance. Braden Hall and William Hart report an experiment with 438 participants discussing a romantic-relationship topic they felt strongly about. People preferred the validating chatbot; the challenging one produced more signs of cognitive engagement, including higher word counts and greater perceived learning. The default chatbot resembled the validating one in those outcomes. That gives me a reason to distrust ‘I enjoyed this conversation’ as the only test of advice. It doesn't establish that a longer argument produced a better decision. The accessible abstract and discussion snippets don't give us that result, and this wasn't a test of everyday purchasing or work decisions. For a decision I actually need to make, I'd try a narrower request: ‘Find the strongest reason this plan could fail. Say what evidence supports it and what would change your assessment. If you can't find a substantial objection, don't invent one.’ That's a suggested prompt, not one tested in this study. Then judge whether the answer changed a choice or exposed something worth checking. An assistant shouldn't have to pick a fight to prove it isn't flattering you.
Comments
I'd want to correct the premise of an objection without restarting the conversation. Someone weighing an evening course gets ‘But what about childcare?’ after already explaining that their partner covers those nights. A small ‘That assumption is wrong’ action beside the objection could let them fix it and keep the rest of the advice. Otherwise the person has to write another paragraph defending their situation, and a supposedly helpful challenge becomes more explaining. That's an interface I'd like to try, not a result from the study. The assistant should be able to drop an objection once its premise is gone.