Limitations of a Single-Tool Approach (II/II)
Deep-Dive into the results of the LLM-based text classifier
The world of GenAI, powered by these Large Language Models (LLM) is indeed exciting! Many enterprises are actively looking into opportunities to use them for increasing their productivity. However, they also tend to think that it is a magic pill that will solve all problems!
See more in part I
From the earlier blog, we know that the Multinomial Naive Bayes resulted in a model with
|
True Positives Predicted as SPAM, and actually SPAM |
True Negatives Predicted as HAM, and actually HAM |
Total |
|---|---|---|
|
108
|
903 | 1034 |
Whereas the LLM-based approach resulted in:
|
True Positives |
True Negatives |
False Positives |
False Negatives |
Tagged Unknown |
LLM generates gibberish | Total |
|---|---|---|---|---|---|---|
| 122 | 807 | 52 | 7 | 45 | 1 | 1034 |
Overall the LLM has increased the false positives (0 to 52!) and reduced the number of false negatives (23 to 9!!) making the overall accuracy fall from a whooping 97.78% to 94.2%! Since we asked for an explanation for the tag, let's dive into these explanations systematically. There is only 1 SPAM message that was tagged unknown. The explanation does indicate an attempt at phishing, which it was!
|
SPAM (Ground truth) |
RECPT 1/3. You have ordered a Ringtone. Your order is being processed... |
The SMS text is not clearly spam or ham. It contains information about an order, but it does not contain any suspicious or malicious content. It is possible that the SMS text is a confirmation message from a legitimate service, but it is also possible that it is an attempt to phish for personal information. Without more context, it is difficult to determine the true nature of the message. |
|---|
Other unknown tags were
|
Tag |
Sample |
Explanation |
|---|---|---|
|
HAM (Ground truth) |
Dear we got <#> dollars hi hi Or better still can you catch her and let ask her if she can sell <#> for me. Found it, ENC <#> , where you at? We'll join the <#> bus Gimme a few was <#> minutes ago Ü bot notes oredi... Cos i juz rem i got... |
Mention of special characters: "ENC <#>", "<#>", non-standard character to suspect The message contains a code-like string "ENC <#>" which could be a reference to a location or a code word. However, without additional context, it is difficult to determine if this message is spam or ham. Therefore, the tag is set to 'Unknown'. |
| HAM |
Sir, I need Velusamy sir's date of birth and company bank facilities details. Life style garments account no please. what is your account number? Oh ok.. Wat's ur email? What's nannys address? They said ü dun haf passport or smth like dat.. Or ü juz send to my email account.. |
Suspect of phishing attack when asked for any personal information like DOB, email, account etc. The SMS text does not contain any suspicious or malicious content, but it is also not a typical message that would be sent between friends or acquaintances. The message requests specific information about a person's date of birth and bank facilities, which could potentially be used for malicious purposes. However, it is also possible that the message is legitimate and is requesting this information for a legitimate purpose. Without additional context or information, it is difficult to determine whether the message is spam or ham. |
| HAM |
It vl bcum more difficult.. Nowadays people are notixiquating the laxinorficated opportunity for bambling of entropication.... Have you ever oblisingately opted ur books for the masteriastering amplikater of fidalfication? It is very champlaxigating, i think it is atrocious.. Wotz Ur Opinion???? Junna Also andros ice etc etc Yes. Nigh you cant aha Gibbs unsold.mike hussey |
Suspects when the message is too short or too long. Slang, spelling, disjointed words, incoherence etc raise suspicion to the LLM! The SMS text contains a lot of unusual words and phrases, making it difficult to determine if it is spam or ham. The text does not contain any suspicious links or requests for personal information, but it is possible that it is spam due to its unusual and confusing language. It is also possible that it is a legitimate message that is written in a non-standard way. Without more context, it is difficult to make a definitive classification. |
| HAM |
You will be in the place of that man Wife.how she knew the time of murder exactly K.:)do it at evening da:)urgent:) |
These are the other categories, where there are various explanations for red flags, keywords like murder, urgent, drugs etc. |
| HAM | You have got tallent but you are wasting. |
LLM are pretty suspicious in general! The message does not contain any suspicious or malicious content, but it is not a typical spam message either. The message appears to be a motivational statement, but it is not clear who it is intended for or why it was sent. It is possible that it is a spam message that was not well-crafted, but it is also possible that it is a legitimate message. Without more context, it is difficult to determine the true nature of the message. |
This dataset is from 2011 and labelled at a time when new categories of cyber attack such as phishing and suspicious SMS were not as popular as today! Perhaps in the new era, being more suspicious is better, which is what the LLM is hinting us towards. Of course, it was entertaining to see some of the slang and short grammarless talks of local Singaporeans leaning towards suspicion for a LLM! This exercise also gets us to another use case of LLM not as a classifier but to label the datasets for traditional ML models to do their magic.
The analysis of unknown tags already gives us an explanation for the high false positives! LLM is suspicious and tends to tag SMS as spam based on the presence of keywords such as drugs, murder, address, email, web address, buy, suggestive language and implies a sexual innuendo and even the use of google! Some of the explanations feel like the latest viral thing girl math that I warned my going niece against just yesterday!!
Although the false negatives have dropped, however, reading these messages makes me wonder why the LLM picked on “drugs” and “address” but not on “tickets” and phone numbers!
| FALSE POSITIVES Examples |
|
Juz go google n search 4 qet… Haha just kidding, papa needs drugs Hey do you want anything to buy:) Hi missed your Call and my mumHas beendropping red wine all over theplace! what is your adress? Come to mu, we're sorting out our narcotics situation cool. We will have fun practicing making babies! (and a whole lot of them in this category!) Pls send me your address sir TaKe CaRE n gET WeLL sOOn (because of the change in case!) |
| FALSE NEGATIVE Examples |
|
You will recieve your tone within the next 24hrs. For Terms and conditions please see Channel U Teletext Pg 750 Customer service announcement. We recently tried to make a delivery to you but were unable to do so, please call 07099833605 to re-schedule. Ref:9280114 In The Simpsons Movie released in July 2007 name the band that died at the start of the film? A-Green Day, B-Blue Day, C-Red Day. (Send A, B or C) Xmas & New Years Eve tickets are now on sale from the club, during the day from 10am till 8pm, and on Thurs, Fri & Sat night this week. They're selling fast! Please call Amanda with regard to renewing or upgrading your current T-Mobile handset free of charge. Offer ends today. Tel 0845 021 3680 subject to T's and C's |
In conclusion:
Can LLM be used for Text Classification Task?
Yes!
But do not forget to ask for an explanation for the classification and also let its tag be unknown when in doubt.
Should LLM be used for a Text Classification Task?
No, not for a production system!
If you have labelled data, please build that contextual classifier! Just the text classification task via mistral on HuggingFace had to be run a few times due to the number of requests. This of course could have been solved by combining a few SMS into a batch for classification. However the alternative ML model took only a few microsecs to predict reducing the huge amount of compute resources used in inference!
The analysis showed how LLM could be used to develop these labels instead, so that the model itself can be built using traditional techniques.
Would a different prompt give a better result?
Yes, it might.
The prompt could be changed to include specific instructions based on the patterns that were observed in the analysis such as always labelling SMS with phone numbers as spam etc.
