Conversations with GenAIs
My conversations with GenAIs made me rethink our 'accepted' ways of evaluating answers to questions!
As I caught up on news today, I read this article that compared the response of the latest Claude 3, from Anthropic, ChatGPT-4, from OpenAI and Gemini, from Google. One of the prompts on the time concept intrigued me and since I had some time to get my day started, I decided to try it on Gemini and ChatGPT4 as those are the two I commonly use. Here is the outcome of the prompt
If it takes 1 hour to dry 15 towels under the Sun, how long will it take to dry 20 towels?
ChatGPT-4
To determine how long it will take to dry 20 towels, given that it takes 1 hour to dry 15 towels under the Sun, we can assume that the drying time is directly proportional to the number of towels. This means that drying more towels will take proportionally longer, assuming all other conditions are constant (such as the intensity of the Sun, the humidity, and the way the towels are spread out to dry).
Time for 20 towels = Time for 15 towels × Number of towels (20)/Number of towels (15)
Let's calculate the time it will take to dry 20 towels.
It will take approximately 1.33 hours, or about 1 hour and 20 minutes, to dry 20 towels under the Sun, assuming all other conditions remain the same.
Python code generated by ChatGPT-4:
# Given values time_for_15_towels = 1 # time in hours to dry 15 towels number_of_towels_original = 15 number_of_towels_new = 20 # Calculate time for 20 towels using direct proportion time_for_20_towels = time_for_15_towels * (number_of_towels_new / number_of_towels_original) time_for_20_towels
Gemini
It's difficult to determine the exact drying time for 20 towels based solely on the information provided. The drying time of towels depends on various factors like:
Material: Thicker towels made from materials like cotton will take longer to dry compared to thinner, microfiber towels.
Sun intensity: The strength of sunlight throughout the day and weather conditions like cloud cover can significantly impact drying time.
Air circulation: Good air circulation allows for faster evaporation of moisture, speeding up the drying process.
Towel size and thickness: Larger and thicker towels naturally take longer to dry than smaller and thinner ones.
Since the drying time is directly proportional to the number of towels, it would take more than 1 hour to dry 20 towels under the same conditions. However, without considering the factors mentioned above, an estimate of drying time cannot be provided accurately.
As the article concluded, neither of the AIs gave us the right answer.However, as an educator it got me thinking about how one should be evaluating the answer. Having trained and enabled hundreds of professionals to make transitions from non-tech backgrounds to the data industry, I am always curious to find ways to explain concepts of AI. I am an engineer by profession and thinking and logical explanation get me to my aahhaa moments. However, the same kind of reasoning does not always get my learners to their own personal aahhaa moment. I encourage them to ask questions constantly to me and to their fellow colleagues and this challenges all of us to talk and help develop skills to convince our clients or stakeholders in the real-world who might not be as excited about the AI-based tool that you are developing or selling!
This conversation got me nostalgic of the passage from Swami and Friends, by RK Narayan. One of my all time favourite books and authors. Thanks to the internet, I was able to indulge myself by reading this passage, which I reproduce here for your joy.
Half an hour later Swaminathan sat in his father's room in a chair, with a slate in his hand and pencil ready. Father held the arithmetic book open and dictated:
"Rama has ten mangoes with which he wants to earn fifteen annas. Krishna wants only four mangoes. How much will Krishna have to pay?"'
Swaminathan gazed and gazed at this sum, and every time he read it, it seemed to acquire a new meaning. He had the feeling of having stepped into a fearful maze.... His mouth began to water at the thought of mangoes. He wondered what made Rama fix fifteen annas for ten mangoes. What kind of a man was Rama? Probably he was like Sankar. Somehow one couldn't help feeling that he must have been like Sankar, with his ten mangoes and his iron determination to get fifteen annas. If Rama was like Sankar, Krishna must have been like the Pea. Here Swaminathan felt an unaccountable sympathy for Krishna.
'Have you done the sum?' father asked, looking over the newspaper he was reading.
'Father, will you tell me if the mangoes were ripe?'
Father regarded him for a while and smothering a smile remarked: 'Do the sum first. I will tell you whether the fruits were ripe or not, afterwards.'
Swaminathan felt utterly helpless. If only father would tell him whether Rama was trying to sell ripe fruits or unripe ones! Of what avail would it be to tell him afterwards? He felt strongly that the answer to this question contained the key to the whole problem. It would be scandalous to expect fifteen annas for ten unripe mangoes. But even if he did; it wouldn't be unlike Rama, whom Swaminathan was steadily beginning to hate and invest with the darkest qualities.
Swaminathan waited with interest for the miracle to happen. Father studied the sum for a second and asked: 'What is the price of ten mangoes?'
Swaminathan looked over the sum to find out which part of the sum contained an answer to this question. 'I don't know.'
'You seem to be an extraordinary idiot. Now read the sum. Come on. How much does Rama expect for ten mangoes?'
‘Fifteen annas of course,' Swaminathan thought, but how could that be its price, just price? It was very well for Rama to expect it in his avarice. But was it the right price? And then there was the obscure point of whether the mangoes were ripe or not. If they were ripe, fifteen annas might not be an improbable price. If only he could get more light on this point!
'How much does Rama want for his mangoes?'
'Fifteen annas,' replied Swaminathan without conviction.
Very good. How many mangoes does Krishna want?'
'Four.'
'What is the price of four?'
Father seemed to delight in torturing him. How could he know? How could he know what that fool Krishna would pay?
'Look here, boy. I have half a mind to thrash you. What have you in your head? Ten mangoes cost fifteen annas. What is the price of one? Come on. If you don't say it--'
His hand took Swaminathan's ear and gently twisted it. Swaminathan could not open his mouth because he could not decide whether the solution lay in the realm of addition, subtraction, multiplication, or division. The longer he hesitated, the more violent the twist was becoming. In the end what the father was waiting with a scowl for an answer, he received only a squeal from his son.
'I am not going to leave you till you tell me how much a single mango costs at fifteen annas for ten.'
What was the matter with father? Swaminathan kept blinking. Where was the urgency to know its price? Anyway, if father wanted so badly to know, instead of harassing him, let him go to the market and find it out. The whole brood of Ramas and Krishnas, with their endless transactions with odd quantities of mangoes and fractions of money, were getting disgusting.
Father admitted defeat by declaring: 'One mango costs fifteen over ten annas. Simplify it.'
Here he was being led to the most hideous regions of arithmetic, Fractions.
'Give me the slate, father. I will find it out.'
He worked and found at the end of fifteen minutes: 'The price of one mango is three over two annas.' He expected to be contradicted any moment.
But father said: 'Very good, simplify it further.'
It was plain sailing after that. Swaminathan announced at the end of half an hour's agony:
'Krishna must pay six annas,' and burst into tears.
Over our 10+2 education system, we have all learnt the art of boxing our answer to the “subject” test on which the question appears. Math questions are meant to be answered by equations and numbers and not in long text. Why? Because that's what the evaluation key expects!
With multiple disciplines amalgamating to form interdisciplinary studies and today’s AI race to develop an AGI (Artificial General Intelligence) which can use as our assistant to increase our productivity at work, these interactions with GenAI machines made me ponder on
How should we evaluate the response from the AI?
Should we compare it with what a human would say?
Should we compare it to the average response from multiple humans?
Should we evaluate it to the correct logical answer, i.e., output that comes from a calculator?
Should we evaluate it to the entertainment value it provides?
Should we evaluate it based on the thinking process?
Should we evaluate it against the structure of the actual question asked?
(Personally, as an educator, I have spent many hours explaining or simplifying my questions to cater to multilingual speaking audiences. There is certainly a knack to simplifying and setting the context right for a question)
BTW, there are many companies working on this problem of evaluating the response of large language models (LLMs) and LLM systems including Arize. They offer a bunch of courses that talk about different ways to metricize LLMs systems.
I decided to test my AIs with the question that our dear Swaminathan had to suffer through…
“Rama has ten mangoes with which he wants to earn fifteen annas. Krishna wants only four mangoes. How much will Krishna have to pay?”
and the computer coughed up the following answer…
ChatGPT-4
To find out how much Krishna will have to pay for four mangoes, we first need to understand the price Rama wants to set for the ten mangoes in order to earn fifteen annas in total.
Given:
Rama has 10 mangoes.
Rama wants to earn a total of 15 annas from selling all 10 mangoes.
First, we calculate the price per mango based on Rama's total expected earnings. After that, we can determine how much Krishna will need to pay for 4 mangoes.
1 anna is 1/16th of a rupee, so 15 annas is 15/16th of a rupee.
Let's calculate the price per mango and then the cost for 4 mangoes.
Krishna will have to pay 6 annas for the four mangoes, based on the calculation where Rama wants to earn a total of 15 annas from selling all 10 mangoes, which sets the price of each mango at approximately 0.094 rupees (or 1.5 annas).
Response Analysis
ChatGPT is clever here, it understands what annas meant and tells you even the conversion logic (which I had to google!)
But if I was a math teacher looking for only the answer, I would be annoyed with ChatGPT’s verbose response as I have to hunt for the answer in the test! I would give feedback to ChatGPT to shhh itself a bit and just give me the answer in a box so that I can evaluate fast instead of reading the whole thing!
Gemini
We cannot determine the exact amount Krishna needs to pay as the price per mango is not provided.
If we assume all mangoes are priced the same, Krishna would need to pay 3/5th of the cost of 10 mangoes.
Response Analysis
Gemini fails to understand annas. Maybe the paid version does better! However it does give some logic which when applied give 3/5*15 = 9 annas. Incorrect!
My Verdict
ChatGPT-4 certainly wins this one and gives me the right answer! It also gives me an interesting tidbit about anna to rupee conversion making this a double BAAM moment!
However, both AIs fail; they do not make the process of Math as entertaining as RK Narayan does!
Do you agree?
PS: Answer from Claude is similar to ChatGPT4 and reproduced here. However, it does not give us the annas to rupee conversion tidbit to make it a teachable moment!
To solve this problem, we need to find the cost of one mango and then calculate the cost of four mangoes based on that.
Given information:
Rama has 10 mangoes.
Rama wants to earn 15 annas from selling the mangoes.
Krishna wants 4 mangoes.
Step 1: Find the cost of one mango.
Total cost of 10 mangoes = 15 annas
Cost of one mango = Total cost / Total number of mangoes
Cost of one mango = 15 annas / 10 mangoes
Cost of one mango = 1.5 annas
Step 2: Calculate the cost of 4 mangoes for Krishna.
Cost of 4 mangoes = Cost of one mango × Number of mangoes
Cost of 4 mangoes = 1.5 annas × 4T
Cost of 4 mangoes = 6 annas
Therefore, Krishna will have to pay 6 annas for 4 mangoes
