.png)
Chat GPT vs. ChatSonic
Evaluate conversational bots through UX heuristics: Multilingual, Clarity, Assistive, Brevity, Unopinionated.
My Role
Conversation UX Researcher
- Establish heuristics and defined rationale
- Evaluate performance of AI chatbots and document
Tools
Figma, Google Suite, Slack, Whiteboard
Duration
2 months, Spring 2023
Team
Sehyun Jeon
Nehal Sharma
Project Type
Master's Research Project
Establish heuristics to assess AI chatbot performance
The objective of this project is to establish a set of heuristics specifically designed for AI chatbots. Subsequently, two bots will be subjected to rigorous testing against these heuristics, with the intention of assessing their performance. Furthermore, drawing inspiration from the thought-provoking series "Black Mirror," we aim to conduct a brainstorming session to uncover potential problems and ethical implications based on our observations and findings.
Process
Define Heuristics for Evaluating AI Chatbots
The rationale behind the heuristics
.png)
.png)
.png)
.png)
Black Mirror Brainstorm 1
We brainstormed, 'What are ways this technology can be used to harm, manipulate, control, abuse — individual people or the collective?'
Black Mirror Brainstorm 2
Utilizing the responses provided by ChatGPT and ChatSonic regarding specific inquiries, we have crafted prospective scenarios that may arise in the future, potentially giving rise to profound ethical concerns.
Scenario 1
.png)
Anne is interested in investing some of her savings in the stock market. However, she doesn't have much experience in this area, and doesn't want to spend time researching stocks and analyzing market trends.
She decides to try out an AI chatbot to get advice. With the information about her financial goals and risk tolerance, it presents her with investment options that it believes are well-suited to her needs. She decides to invest a significant portion of her savings in the stocks suggested by the chatbot.
However, after a few months, Anne notices that her investment returns are not as expected, since the chatbot is not equipped to handle unexpected market fluctuations or other factors that can affect investment performance. As a result, rather than making a profit through investment, she lost a lot of money.
Scenario 2
Karen is a researcher who requires to write journals and articles on a regular basis. One day, she was feeling low on motivation to start writing her article and decided to take help of an AI chatbot for her research.
With it’s help, she was able to gather data points for her article and found the information useful. But she couldn’t find the source of the creator or reference of the source from where the AI chatbot got the response.
Since the AI model is gathering data from multiple sources it’s restricted to provide information for what the user has asked. It’s unable to provide the users with the source or references of the information from where it has constructed the factual information. Therefore, provides users with no credibility; also takes away the credits from the creator of the source.