BYU Student Author: @Jacob_Dutton
Reviewers: @Jimmy_Han, @Dallin_Gardner
Estimated Time to Solve: 30 Minutes
We provide the solution to this challenge using:
Need a program? Click here.
Overview
Welcome to the AI Model Training and Application Challenge. This challenge provides a hands-on introduction to the fundamentals of Artificial Intelligence (AI) and Machine Learning (ML). Through a simulated training exercise using ChatGPT, you’ll explore how AI models are trained and applied, even though ChatGPT itself, as a pre-trained model, does not require this process for tasks like sentiment analysis.
Introduction to AI Model Training
AI involves algorithms that perform tasks requiring human-like intelligence, with Machine Learning being a subset where these algorithms learn from data. Training an AI model typically involves preparing a dataset, which is divided into training and validation sets. The model learns from the training data and is tested with the validation data, adjusting its parameters to improve accuracy and efficiency. This challenge will demonstrate these principles in a simplified context, enhancing your understanding of how AI solutions are developed and deployed.
Objective
To develop a foundational understanding of how AI models are trained and to apply this knowledge by simulating the training process with ChatGPT.
Instructions
- Simulated Training Exercise:
- You are provided with a dataset (Training Data.csv) that includes customer reviews labeled with sentiments. Feed this data to ChatGPT so that it can learn to recognize patterns in the data.
- Model Testing and Quantification:
- Now upload Test Data.csv to ChatGPT, and instruct it to predict the sentiment for each line using the patterns from the training data. If you would like more accurate results, also instruct ChatGPT to use its best judgement for classification as well as the patterns from the training data.
- Deliverable: After running the sentiment analysis with ChatGPT, count and report the number of positive and negative sentiments identified in the test data. Provide a brief analysis of the results.
- Evaluation and Iteration:
- Evaluate the accuracy of the responses from ChatGPT. Refine your prompts if needed to improve the model’s accuracy and re-test to see if the sentiment count changes.
- Reflection:
- Submit a brief reflection including the number of positive vs negative customer reviews from your analysis.
Data Files
Suggestions and Hints
This challenge requires the use of ChatGPT 4.0. As of creating this challenge, ChatGPT3.5 does not allow for the uploading of files.
Solution
One of the nuances with AI is that it will produce a different solution each time. Because of this, there is no exact solution to this challenge. The test data has a total of 10 positive and 10 negative reviews.
Solution Video: Challenge 192|CHATGPT – AI Model Training and Application
According to ChatGPT, there were 12 positive results and 8 negative results from the provided sentiments. However, upon further personal analysis, I discovered that two of the positive results the ChatGPT should have been negative. One was sentiment number 8 which stated, “Not worth the price. I feel completely let down by the poor quality.” And the other was sentiment number 14 which stated, “Totally dissatisfied with the purchase. It didn’t meet any of my expectations.” The rest of the results I agree with.
1 Like
Time to complete: 20 mins
Rating: beginning.
After running the sentiment analysis, ChatGpt found that there are 13 reviews with positive sentiments and 7 reviews with negative sentiments in the test data.I found out that there should be 10 positive and 10 negative reviews. It also has different answer each time, it has 9 positive and 5 neutral results the next time.
Time to complete: 20 minutes
Rating: beginner
Comments: After running the data through ChatGPT and making some edits, it came back with results of 9 positive sentiments and 11 negative sentiments. However, when I actually went in and counted myself, it was 10 positive and 10 negative. I ran it through again and asked ChatGPT to count a second time and the result that it gave me was 10 positive and 10 negative.
Time to complete: 15 minutes
Rating: beginner
Comments: After adjusting the data to be readable without uploading the excel file, as the older version of Chat-GPT doesn’t process file uploads, I was able to give the processor the data. After giving it the 20 prompts it correctly guessed all 20. When asked how confident it was about its predictions, it was very confident and explained the common identifiers of positive and negative responses succinctly and clearly.
Time to complete: 20 mins
Rating: beginning.
Comments: After running the data, ChatGPT found that there are 12 reviews with positive results and 8 reviews with negative results in the test data. I found out that there should be 7 positive and 13 negative reviews. It has a differnet answer everytime in the last one it had 10 positive and 6 neutral results in the last try.
Time to complete: 15 min
Rating: Beginner
Comments: After running the data, ChatGPT found that there are 8 positive reviews and 3 negative reviews. I went and recounted it and it was right. I kept changing the prompt and it still gave me the same answer.
Time to complete: 25 minutes
Rating: Beginner
Thoughts: I was surprised to see that ChatGPT did not get a perfect score on it’s sentiment analysis. It returned 12 positive messages and 8 negative messages. Message number 8 and 14 were both negative messages, but ChatGPT analyzed them as being positive. This was very interesting to me, and even after adjusting my prompt to emphasize looking at the entire context of the message, it still incorrectly marked these messages as positive. All-in-all this was a great techhub challenge and very insightful into verifying what GenAI outputs.
Time: 30 minutes
Difficulty: Beginner
Comments:
Similar to the others, It got at least 2 wrong. It took me several iterations to correct everything and I basically had to walk it through every mistake it made and ask it why it made them. 4 or so iterations of this got all the original errors and errors generated in the interim smoothed out.
Time spent: 30min
Difficulty: Easy
Comments:
At first, my chatgpt had only 3 rights and all wrong. It even put neutral sentiments which I never asked for. It then switched to put nnegative as the default prediction. Then it started to put more positive prompt that were correct but still had a couple wrongs. I didn’t want to tell Chatgpt which was wrong. After 7 promptings, chat was finally able to give the correct sentiment prediction. I noticed that if you ask one review at a time, it gives the right answer. But if you ask to predict the sentiment for all of them at once, it has many wrongs before the system is able to accurately predict the sentiment.
Time to complete: 30 minutes
Rating: Beginner
Comments: After running the training data through ChatGPT and then having it analyze the test data, it initially came back with 18 positive and 2 negative. I then asked Chat to walk through each line of the data and explain why it classified it the way that it did. After that, it realized that some of the errors in its analysis. It took 7 iterations of prompting before Chat classified 11 positive and 9 negative reviews.
Time spent: 30min
Difficulty: Easy
Comments:It took approximately seven iterations to train the model to interact with the information effectively and predict the correct patterns. I utilized ChatGPT to add new reviews and sentiments to the original training data, allowing the model to better recognize and predict the appropriate sentiment. My approach involved prompting ChatGPT to identify specific lines that were incorrectly classified and then generate new reviews to strengthen the training dataset’s ability to guide the model toward the correct patterns. After repeating this process seven times, the model successfully learned to make accurate predictions
Time to complete: 30
Prompt:
Refined Sentiment Analysis Results:
- Positive Sentiments: 6
- Negative Sentiments: 7
Brief Analysis:
After refining the sentiment patterns, the model now classifies 6 reviews as positive and 7 as negative. The refined logic has improved the classification by incorporating additional phrases identified during manual validation, such as “well-designed” and “first-rate” for positive reviews, and “total disappointment” for negative reviews.
This iterative refinement demonstrates that model performance can be enhanced through careful analysis and adjustment of its logic. Let me know if you would like further adjustments or deeper insights
Time to complete: 45 Minutes
Difficulty: Easy
This was a very interesting test. ChatGPT identified that there were 12 positive reviews and 8 negative reviews. I found there to be 10 of each. However, there were 2 where we disagreed that makes the outcome looke better. It was wrong on 4 of them. This was very interesting to me, I think the model is looking for key words and doesn’t yet understand that some of those words can be used negatively as well.
Time to complete: 30 minutes
Rating: Beginner
ChatGPT found that there are 13 positive and 7 negative. I found that there were 12 positive and 8 negative. Pretty interesting that it was as accurate as it was. I tried messing around with prompts to make it better but couldnt get better than 1 error.