At Monday’s meeting, Max and Haiyi suggested to keep testing out our chat bot script design on other people for the week and make changes accordingly. Then, we tested our active/non-personalized bot design on a group of people in the GroupLens research lab. Three of us simultaneously worked together to operate the chatbot. The lab participants were testing the limits of what the bot could do, so it was quite difficult to act out the bot, but we still managed well. After they came to a consensus and completed the decision making process, we all met to have a discussion about the chatbot. They provided a lot of insightful feedback that helped us establish what we needed to fix. For example, they felt confused as to whether the voting during the conversation was for the final decision or for discussion purposes. Most of the users felt that the voting was for the final decision, so they felt very pressured to decide quickly. Wednesday, we met briefly with Haiyi and Max to update them on how the trial with the lab members went. They also found the results of our trial to be helpful and determined that using personalization as a variable might not create a significant impact. Because of this we started brainstorming possibilities for a new experimental design. For the second half of the week, we worked hard to finish the IRB draft for submission, and determining a new experimental design. We are moving towards testing variables of active/passive, enforced structure/not enforced, and advice/no advice. Our goal for the next week is to finalize the design of the study so that we can run it within the last weeks of my REU stay. In addition, we hope to have the IRB submitted and approved. This weekend, some of us went to the Minneapolis Institute of Art. This museum was quite different from the Walker. It had a wide variety of exhibits, contemporary, classic, cultural, and historical art. I was extremely impressed and would highly recommend anyone to check it out!
0 Comments
On Monday, there was a big deadline for the Computer-Supported Cooperative Work(CSCW) conference, so everyone in the research lab was busy putting the final touches on their papers. Because of all the chaos, Haiyi was not able to join our Monday meeting. However, we conducted our meeting per usual and each briefed the group on what we have accomplished since last week. My group presented the new script design for the active chatbot and Max responded suggested that we test it out to check the flow of our structure. The next day, we tested out the bot design with each other by taking turns acting as the bot and acting as the users. It turned out to be really fun and revealed some details that we needed to change. We determined that we need to be exact in when to intervene the conversation to implement structure into the conversation. For example, we created timed intervals of when we would prompt the users to move on to voting from the discussion phase.
On Wednesday we were finally able to meet with Haiyi. For the bot designs she suggested that we make a script difference for the Personalized/Non-Personalized variables rather than restricting it to the recommendation algorithm. By adding this variable into the dialogue, we will be able to see how user interaction is affected by it. In addition, we have decided to run the study through Amazon Mechanical Turk rather than MovieLens because the process will be much easier and there will be more longevity for future studies. Therefore, my task for the rest of the week is to learn and familiarize myself with the MTurk API and conducting a synchronous study overall. To learn about this I have been using the Amazon Web Services documentation and resources. For the weekend, I explored some spots in Downtown Minneapolis, like the Nicollet Mall (which is under construction for the next Super Bowl), the Central Library, and a very large Target. In addition, a group of us went to the Como Zoo and Conservatory. The entire Como park was peaceful and beautiful. And the Zoo and Conservatory were excellent for the free admission! Due to holidays, we weren’t able to have our usual Monday meeting to get changes approved and to discuss the direction of the project. Luckily Max was able to meet with us on Wednesday to look over our script design and give some advice. Our passive bot received approval, but our active bot design needs some more changes. Previously, we strived to differentiate the 2 designs a little too much and had created some confounding variables. Max suggested that we design the bots to be parallel but to make the Active bot stand out with it’s facilitation of structure and offer of voting polls. So we are working to edit the design for the active bot this week.
In addition, we started writing the user satisfaction survey. At the end of the experiment, the participants will complete a follow up survey to give feedback on their satisfaction with the bot, the movies recommended and the decision making process. The results from the survey will serve as our data to determine which bot design is the best for aiding group decision making. On Thursday, the PhD student asked another undergraduate and I to write a draft for the Institutional Review Board (IRB) proposal. Writing an IRB proposal consists of detailing the objectives and the procedures of a research study. After a proposal is written, it is sent to the IRB to be approved. In order to be approved, the research study must follow the IRB’s guidelines which ensure the beneficence, respect, and justice of the participants. Writing the IRB proposal is an important task, so I was a bit nervous and intimidated. But I’m glad I can contribute towards something so significant. On Friday, I continued to write the draft for the IRB proposal. But in the morning there was a communicating science workshop with the entire Big Data REU group. During this morning session, we went over some strategies for writing abstracts and creating posters in order to prepare us for the symposium at the end of the summer. The workshop was extremely helpful and made me feel more comfortable writing my abstract and prepping my symposium poster. Outside of lab this week, I enjoyed celebrating the 4th of July! For the holiday I went to Stone Arch Bridge to take part in the festivities and to watch the fireworks. Later in the week a group of us took a trip to the Walker Art Center and Sculpture Garden. The Art Center features contemporary art and the Sculpture Garden is home of the famous Spoon and Cherry. On Monday we pitched our experimental design idea to Haiyi and Max. Our PhD student is away for an HCI conference, so it’s up to us undergrads to present. In our presentation, we explained our plans for our 2x2 study by detailing the variables, the participants, the experimental environment, and the outline of the chatbot script. Haiyi and Max liked some elements of what we had so far, but had many helpful suggestions and steered us in the right direction. So the next day (Tuesday) the 3 of us started writing the more detailed script with the new changes in mind. We felt comfortable with the alterations in the bot’s designs, specifically increasing the difference between the passive and the active bot. For the passive bot, we are continuing to make it act upon mention only. But we are also making it solely informative. It will never prompt user action, but merely react and respond to what the members of the group request of it in the chat room. For the active bot, we are having it be more of a discussion facilitator. For example, it will now suggest conversation structure by making statements like “Let’s start the decision making process by brainstorming”. In addition, the active bot will have a voting feature (by polls) to further support the decision making process.
As for the bot with information on MovieLens profiles versus none, we have changed these variables to personalized recommendations versus non-personalized. This made the script writing process easier, because we could just focus on 2 different scripts. One for active and one for passive, rather than 4 separate scripts for each cell. Additional topics that we discussed were which platform to use for the chatroom and where to get the population of participants for the user study(MTurk, Movielens, or Offline). This is more difficult for us to discuss because we do not have a wide knowledge of how these services operate (etc.). Therefore we have made this week’s task to be researching these options in addition to brainstorming more for the script. To accomplish this, I’ve been trying to read more research literature in addition to reviewing past publications I have found to try to find ‘inspiration’ for the script. I have also been informing myself on services that will help us develop the chatbot/chatroom. The tools I have been reading up on are Wit.ai, IBM Watson Conversation and the Slack bot. On Friday, our PhD student came back from his conference so we worked hard to finalize the details on the script. We spent a few hours writing /re-writing everything. We also discussed the chatbot/chatroom set-up. We are leaning towards using Slack for the chatroom, and utilizing the Slackbot feature to create our movie lens bot for the study. As for participants, we are now looking to gain our population from existing MovieLens users. We’ve made many changes this week. Now it’s up to our meeting on Monday with Haiyi and Max to see if these changes are good, or if we need to make more adjustments. For this weekend, we are going to Minnehaha falls! The group has been wanting to go since the start of the program, but the weather forecasts haven’t been in our favor. But this Saturday it will be clear skies! |
AuthorHi! My name is Mary Solomon and I'm from Northwest Ohio. I'm a rising Junior studying Data Science with a music minor at Bowling Green State University. I plan to graduate in May 2019.. Archives
August 2017
Categories |