AI for sentiment analysis: Use cases, applications and development

Understanding your customers is key to providing exceptional services and improving your brand’s image. In this digital era, customers constantly express their opinions, feelings, and experiences with brands through various channels like social media, online reviews, and customer support interactions. The sheer volume of such unstructured data can be overwhelming and nearly impossible to process manually.

Enter AI for sentiment analysis—an innovative way to automatically decipher the emotional tone embedded in customers’ comments, giving businesses quick, real-time insights from vast sets of customer data. This advanced technique utilizes machine learning and natural language processing to transform how businesses interact with their customers.

In the dynamic world of customer service and sales, leaders strive to offer their customers a smooth, personalized experience. The challenge, however, lies in comprehending their emotions and sentiments, especially during real-time conversations across digital platforms. AI-powered sentiment analysis allows businesses to understand, engage, and serve their customers better, enhancing conversation performance, boosting sales momentum, and creating superior customer experiences.

With AI sentiment analysis tools readily available, businesses can swiftly dive into this innovative realm. This article explores AI for sentiment analysis, its value to your business, and the process of implementing it to help you unlock its full potential. Read on as we delve deeper into the captivating world of AI in sentiment analysis.

• What is sentiment analysis?
• Types of sentiment analysis
• How does AI in sentiment analysis work?
• Key applications of AI-based sentiment analysis
• Benefits of AI-based sentiment analysis
• Use cases of AI-enabled sentiment analysis across industry verticals
  • AI-driven sentiment analysis in retail
  • AI-based sentiment analysis in tourism and hospitality
  • AI for sentiment analysis in telecommunications
  • AI-driven sentiment analysis in healthcare
  • AI-powered sentiment analysis in banking
• Streamlining sentiment analysis workflow with GenAI
• LeewayHertz’s AI development services for sentiment analysis
  
• How to do sentiment analysis using LSTM?

What is sentiment analysis?

Sentiment analysis, also referred to as opinion mining, is a method to identify and assess sentiments expressed within a text. The primary purpose is to gauge whether the attitude towards a specific topic, product, or service is positive, negative, or neutral. This process utilizes AI and natural language processing (NLP) to interpret human language and its intricacies, allowing machines to understand and respond to our emotions.

Consider this process akin to mining a wealth of data, identifying hidden nuggets of sentiments in them that can provide actionable business insights. Rather than merely scratching the surface with rudimentary sentiment analysis, businesses can delve deeper using advanced AI techniques.

To illustrate, let’s consider three examples:

1. Suppose a customer writes an online review saying, “I love the friendly staff and quick service at this restaurant.” Sentiment analysis would categorize this feedback as positive due to the presence of words like ‘love,’ ‘friendly,’ and ‘quick’ in the feedback.
2. On the other hand, a comment such as “The laptop is good, but it overheats too quickly” contains mixed sentiments. While the customer appreciates some aspect of the product (‘good’), the negative sentiment is expressed in ‘overheats too quickly.’
3. A neutral sentiment could be something like “The book was received on time,” which neither praises nor criticizes the product or service.

Through sentiment analysis, businesses can go beyond simple count-based metrics, capturing high-value insights from customer conversations and social media streams. As a result, they can comprehensively understand customer sentiments, improve customer experiences, and ultimately enhance their product or service offerings.

Types of sentiment analysis

Sentiment analysis encompasses a variety of analytical methods that help understand the sentiment or emotional tone behind textual data. Here are four key types:

1. Aspect-based sentiment analysis: This approach allows us to understand the sentiment related to specific product or service aspects. Instead of merely determining whether the sentiment is positive or negative, it drills down to discover the sentiment about individual features or aspects. For example, in the case of a restaurant, aspects might include “food quality,” “ambiance,” or “service speed.”
2. Fine-grained sentiment analysis: This technique refines sentiment polarity by distinguishing degrees of sentiment. Instead of merely classifying sentiments as positive, negative, or neutral, it breaks them down into more specific categories such as “very positive,” “somewhat positive,” “neutral,” “somewhat negative,” or “very negative.” This is particularly useful for detailed analysis of reviews or ratings, offering a granular view of customers’ sentiments.
3. Emotion detection: This form of sentiment analysis identifies specific emotions within textual data. Instead of determining whether the sentiment is positive, neutral, or negative, it categorizes the emotions expressed in the text, like joy, surprise, anger, sadness, fear, etc. This offers a deeper understanding of the user’s emotional state.
4. Intent analysis: This type aims to understand the intention or goal behind a particular text. By identifying the underlying intent, organizations can gain insights into customer behavior, predict future actions, and adapt their strategies accordingly. It’s especially useful in scenarios like customer service, where predicting a customer’s behavior can help plan effective responses.

How does AI in sentiment analysis work?

Integrating AI into sentiment analysis involves various components to streamline the analysis of textual data, generate insights, and support decision-making. It goes beyond traditional sentiment analysis processes by incorporating powerful Large Language Models (LLMs) and connecting them with an organization’s unique knowledge base. This method facilitates a deeper understanding of sentiment by analyzing context, nuances, and underlying emotions within text data, empowering businesses to make informed decisions based on comprehensive sentiment insights.This LLM-based architecture utilizes various components to improve the sentiment analysis process, enabling businesses to fully comprehend sentiment trends and utilize this insight to drive strategic initiatives and enhance overall business performance. Here’s a detailed breakdown of the process:

1. Data sources: The process begins by gathering data from various sources relevant to the sentiment analysis. This data can include:

1. 1. Social media: Data retrieved from platforms like Twitter, Facebook, and Instagram, capturing user-generated content such as posts, comments, and messages to analyze public sentiment, opinions, and trends.
   2. Customer reviews: Written feedback and ratings provided by customers on platforms such as Amazon, Yelp, and Google Reviews, offering insights into customer satisfaction, preferences, and sentiments regarding products or services.
   3. Surveys and feedback forms: Responses collected from structured questionnaires, online surveys, or feedback forms designed to gather opinions, attitudes, and individual feedback regarding specific topics, products, or services.
   4. Call center transcripts, emails, and support tickets: Textual records of interactions between customers and customer support representatives, including transcripts of phone calls, emails, and support tickets, offering insights into customer concerns, issues, and sentiment towards a company’s products or services.
2. Data pipeline: The data gathered from the previous sources is subsequently channeled through data pipelines. These pipelines handle tasks such as data ingestion, cleaning, processing (including data transformations like filtering, masking, and aggregations), and structuring, preparing it for subsequent analysis.
3. Embedding model: The processed data is segmented into chunks and fed into an embedding model. This model converts textual data into numerical representations called vectors, enabling AI models to comprehend it effectively. Well-known models for this purpose are developed by OpenAI, Google, and Cohere.
4. Vector database: The resulting vectors are stored in a vector database, facilitating streamlined querying and retrieval processes. This database efficiently manages the storage, comparison, and retrieval of potentially billions of embeddings (i.e., vectors). Prominent examples of such vector databases include Pinecone, Weaviate, and PGvector.
5. APIs and plugins: APIs and plugins such as Serp, Zapier, and Wolfram are crucial in linking various components and facilitating additional functionalities, such as accessing additional data or executing specific tasks seamlessly.
6. Orchestration layer: The orchestrating layer is critical in managing the workflow. ZBrain is an example of this layer that simplifies prompt chaining, manages interactions with external APIs by determining when API calls are required, retrieves contextual data from vector databases, and maintains memory across multiple LLM calls. Ultimately, this layer generates a prompt or series of prompts that are submitted to a language model for processing. The role of this layer is to orchestrate the flow of data and tasks, ensuring seamless coordination across all the architectural components.
7. Query execution: The data retrieval and generation process initiates when the user submits a query to the sentiment analysis application. These queries can cover various aspects relevant to the user experience, including satisfaction levels, feature preferences, usability concerns, and overall sentiment towards the product/service.
8. LLM processing: Upon receiving the query, the application forwards it to the orchestration layer. This layer then retrieves pertinent data from the vector database and LLM cache before sending it to the suitable LLM for processing. The apt LLM is selected based on the query’s nature.
9. Output: The LLM produces an output based on the user query and the data it receives. This output can take various forms pertinent to analyzing sentiment toward a product/service, such as identifying key themes in customer feedback, generating sentiment reports, or summarizing sentiment across different user segments or demographics.
10. Sentiment analysis app: The verified output is then presented to users through the sentiment analysis app. This central platform integrates all gathered data, sentiment analysis results, and actionable insights, presenting them in an accessible format for decision-makers to understand customer sentiment trends and make informed strategic decisions.
11. Feedback loop: User feedback on the LLM’s output is another important aspect of this architecture. The feedback is used to improve the accuracy and relevance of the LLM output over time.
12. Agent: AI agents are critical in this architecture to address complex problems, interact with the external environment, and enhance learning through post-deployment experiences. They achieve this by employing advanced reasoning/planning, strategic tool utilization, and leveraging memory, recursion, and self-reflection.
13. LLM cache: Tools like Redis, SQLite, or GPTCache are used to cache frequently accessed information, speeding up the response time of the AI system.
14. Logging/LLMOps: Throughout this process, LLM operations (LLMOps) tools like Weights & Biases, MLflow, Helicone and Prompt Layer help log actions and monitor performance. This ensures the LLMs are functioning optimally and continuously improve through feedback loops.
15. Validation: A validation layer is employed to validate the LLM’s output. This is done through tools like Guardrails, Rebuff, Guidance, and LMQL to ensure the accuracy and reliability of the information provided.
16. LLM APIs and hosting: LLM APIs and hosting platforms are essential for executing sentiment analysis tasks and hosting the application. Depending on the requirements, developers can select from LLM APIs such as those offered by OpenAI and Anthropic or opt for open-source models. Similarly, they can choose hosting platforms from cloud providers like AWS, GCP, Azure, and Coreweave or opt for opinionated clouds like Databricks, Mosaic, and Anyscale. The choice of LLM APIs and cloud hosting platforms depends on the project’s needs.

This structured flow outlines how AI streamlines sentiment analysis tasks, utilizing diverse data sources and advanced tools to deliver precise and actionable insights on customer sentiments. By automating the sentiment analysis process, AI enhances operational efficiency and empowers stakeholders to conduct thorough sentiment assessments, leading to informed decision-making and a deeper understanding of customer sentiments.

Key applications of AI-based sentiment analysis

Sentiment analysis finds a broad range of applications across various domains. Here is how it is put to use in diverse areas:

• Unveiling and predicting market trends: Sentiment analysis can dissect vast amounts of market research data, identifying emerging trends and insights into consumer purchasing behaviors.
• Brand perception monitoring: Utilizing sentiment analysis allows businesses to delve into how consumers perceive their products or services. Businesses can provide development teams with substantial data by analyzing product-related discussions and sentiments, driving informed decisions about product improvements or modifications.
• Scrutinizing public and political sentiments: Sentiment analysis can be a potent tool in political science, allowing analysts to predict election outcomes by analyzing public sentiment towards candidates. A more accurate prediction can be made about the election’s outcome by sifting through news articles, social media posts, public opinions, and suggestions.
• Interpreting customer feedback: Customer feedback holds a wealth of insights. With sentiment analysis, businesses can interpret this data, identifying improvement areas. By extracting sentiment from customer feedback, businesses can shape strategies that enhance customer satisfaction and loyalty.
• Social media conversations analysis: Social media is a treasure trove of information about brand perception. Sentiment analysis allows businesses to decode these conversations, deriving meaningful insights about what consumers think about their brand. This can be crucial in shaping future business strategies, marketing campaigns, and customer service improvements.
• Mitigating employee turnover: By applying sentiment analysis to employee feedback, organizations can understand employee satisfaction levels and uncover potential areas of discontent. It can guide measures to boost employee morale and productivity, leading to a reduction in employee turnover and a happier, more productive workforce.
• Brand monitoring across platforms: Beyond social media, brand-related discussions occur on numerous platforms, such as blogs, news sites, forums, and product reviews. Sentiment analysis adds valuable context to these discussions, understanding the nuances of customer opinions. This can help assess the impact of a PR crisis on your brand and evaluate the effectiveness of measures taken to manage the situation.
• Market research and competitor analysis: Sentiment analysis can be a vital asset in market research. Analyzing sentiments around your competition can reveal their strengths and weaknesses, helping you strategize effectively. Also, tracking keywords or hashtags relevant to your industry can help detect market trends and gauge interest around specific topics, providing a competitive advantage.

Understanding employee feedback (Voice of employee): Organizations can gain actionable insights by employing sentiment analysis on employee feedback from various sources. It helps uncover how employees feel about different aspects of their job, like work-life balance, compensation, benefits, etc. Such insights can drive initiatives to boost employee engagement, improve communication, and attract new talent, ultimately creating a more positive and productive work environment.

Benefits of AI-based sentiment analysis

Understanding customer sentiments

Harnessing the power of AI for sentiment analysis can bring businesses closer to understanding their customers’ emotions during interactions. By evaluating the tone, vocabulary, and intensity of conversations, companies can detect if customers are delighted, vexed, or disappointed. Equipped with this knowledge, customer service representatives can tailor their responses to reflect customer emotions, cultivating a more empathetic and positive experience.

Boosting response efficiency

Sentiment analysis enhances the efficiency of responses to crucial customer interactions. By gauging the sentiment underlying customer communication, these tools can highlight urgent or significant concerns that demand immediate attention. This knowledge enables businesses to prioritize their responses and address the most pressing matters, such as a flaw in a product, thereby enhancing their response efficiency.

Tailoring customer experience

Sentiment analysis facilitates the personalization of customer experiences by comprehending customer emotions and inclinations, particularly during customer service interactions. Tools available today can discern a customer’s interests, likes, and challenges by analyzing the sentiment of customer service interactions. This knowledge enables businesses to customize their responses and suggestions to meet the customer’s unique needs, resulting in a highly tailored customer experience.

Increasing customer retention

Sentiment analysis can also be instrumental in bolstering customer retention by identifying and rectifying the root causes of customer dissatisfaction. By examining sentiments in customer communications, businesses can spot recurring issues and trends that lead to customer attrition. This knowledge allows businesses to adopt proactive measures to enhance customer satisfaction and loyalty.

Enhancing sales outcomes

Sentiment analysis can significantly improve sales results by analyzing sentiments in sales conversations and outreach. This analysis offers deep insights into customer preferences, pain points, and emotional reactions, as well as the performance of sales representatives. These insights enable sales teams to personalize their approach for each customer, fostering stronger relationships and increasing the chances of successful deals. Furthermore, sentiment analysis aids sales leaders in recognizing patterns in customer sentiment over time, facilitating data-driven decision-making on sales strategies and approaches. Ultimately, sentiment analysis empowers sales teams to conduct more effective and personalized conversations, leading to enhanced customer satisfaction and improved sales outcomes.

Use cases of AI-enabled sentiment analysis across industry verticals

AI-driven sentiment analysis in retail

The retail sector can harness the power of AI-driven sentiment analysis to gain a deeper understanding of customer perceptions about their brand and pinpoint upcoming market trends.

Here’s an outline of its use cases in retail:

• Understanding customer sentiments:  AI sentiment analysis tools can evaluate customer reviews, social media posts, and other user-generated content to understand customers’ feelings about a brand or its products. These insights can help retailers identify areas of success or areas that need improvement.
• Personalized marketing:  By understanding individual customer sentiments, retailers can tailor their marketing and advertising strategies to better resonate with their target audience. This personalized approach can increase engagement and improve overall marketing effectiveness.
• Customer service improvement: AI for sentiment analysis can identify negative sentiments in real time, allowing customer service teams to address complaints or issues promptly. This proactive approach can help improve customer satisfaction and brand perception.
• Product development:  Customer sentiment analysis can reveal what features customers like or dislike about a product. This valuable feedback can guide product development teams to modify to meet customer expectations better.
• Competitive analysis:  By applying sentiment analysis to public opinions about competitors, retailers can identify strengths and weaknesses in their rivals’ offerings. Such insights can help them position their products more effectively in the market.
• Trend forecasting:  AI for sentiment analysis can identify consumer attitudes and preferences shifts over time, providing early warnings of changing market trends. Retailers can use this information to stay ahead of the curve, adjust their strategies, and cater to evolving customer needs.

Overall, AI-based sentiment analysis offers a wealth of benefits to retailers, from enhancing customer satisfaction and personalizing marketing to improving products and anticipating market trends. By leveraging these insights, retailers can make more informed decisions, drive growth, and enhance their competitive edge.

AI based sentiment analysis in tourism and hospitality

The hospitality sector can greatly benefit from using AI for sentiment analysis in many ways. Here’s a snapshot of its use cases:

• Understanding guest feedback:  Hotels, restaurants, and other hospitality businesses receive massive amounts of feedback through online reviews, social media posts, and direct customer interactions. AI based sentiment analysis can help these businesses efficiently parse through these feedback data to understand customer sentiment, uncovering valuable insights about their services.
• Enhancing guest experiences:  Sentiment analysis can reveal specific aspects of the guest experience that are especially delightful or problematic. This information can help hospitality businesses tailor their services to meet guest expectations better, leading to increased customer satisfaction and loyalty.
• Real-time service recovery:  AI for sentiment analysis can identify negative sentiments in real time, allowing the hospitality business to intervene immediately and resolve the issue before it escalates. This can significantly improve the overall guest experience and prevent potential reputational damage.
• Strategic decision-making:  By gauging public sentiment about different aspects of their offerings, businesses in the hospitality sector can make data-driven decisions regarding their services. Whether revamping a restaurant’s menu or redesigning a hotel’s rooms, sentiment analysis can provide actionable insights that help enhance business strategy.
• Competitor analysis:  By applying sentiment analysis to reviews of competitor hotels or restaurants, businesses can identify their own relative strengths and weaknesses. This information can help them position their services more effectively in the market.
• Trend identification:  AI for sentiment analysis can help hospitality businesses identify emerging trends or changing guest preferences. This can be particularly useful in such a dynamic industry, where staying on top of trends can provide a significant competitive advantage.

AI for sentiment analysis in telecommunications

I-driven sentiment analysis plays a critical role in the telecom industry, providing a range of use cases to improve customer experience, business operations, and strategic decision-making. Here’s how it can be employed:

• Customer experience management:  The telecom sector often has to manage massive volumes of customer interactions across various channels, including call centers, social media, emails, and more. AI based sentiment analysis can process this data to understand customer sentiment and identify pain points, enabling proactive customer service and improved customer experience.
• Churn prediction and prevention:  Telecom providers can identify dissatisfied customers and predict potential churn by analyzing customer sentiment over time. This information can be leveraged to implement targeted strategies and offers to retain at-risk customers.
• Service improvement:  Sentiment analysis can reveal underlying issues with service quality, network coverage, or pricing causing customer dissatisfaction. Telecom providers can use these insights to improve these areas and enhance customer satisfaction.
• Competitive analysis:  Sentiment analysis can be applied to public discussions and reviews about competitors to understand their strengths and weaknesses from the customers’ perspective. This can help telecom providers position their services more effectively and identify opportunities for differentiation.
• Product development:  By understanding how customers feel about different features and services, telecom providers can align their product development efforts with customer needs and preferences.
• Marketing and sales:  Sentiment analysis can provide insights into how marketing and sales messages are resonating with the audience. This can inform the development of more effective marketing strategies and sales pitches.
• Real-time decision-making:  In an industry as dynamic as telecom, real-time sentiment analysis can help providers respond quickly to emerging issues or opportunities, leading to better decision-making and agility.

AI-driven sentiment analysis in healthcare

In the healthcare sector, AI sentiment analysis can be harnessed in a variety of ways to improve patient care, service delivery, and operational efficiency. Here are some potential use cases:

• Patient feedback analysis:  AI-driven sentiment analysis can be used to interpret patient feedback from various sources, such as social media, online reviews, and patient surveys. By identifying positive and negative sentiments, healthcare providers can better understand patient experiences, address concerns, and enhance service quality.
• Improving patient care:  Sentiment analysis can aid in interpreting patient feelings and emotions conveyed during consultations or written in patient records or communication. This can assist healthcare professionals in better understanding a patient’s mental and emotional state, potentially leading to more personalized and effective care.
• Pharmacovigilance:  Sentiment analysis can aid in drug safety surveillance, known as pharmacovigilance. Potential adverse drug reactions can be identified early by analyzing sentiments in patient forums, social media posts, and other online platforms.
• Clinical trial monitoring:  AI sentiment analysis can track participants’ experiences in clinical trials. Analyzing sentiments in patient reports can provide early indications of side effects, efficacy, or adherence issues.
• Healthcare marketing:  Using sentiment analysis, healthcare organizations can understand public perceptions and sentiments about their brand, services, or specific marketing campaigns. This information can guide marketing strategy and communication.
• Policy making:  Governmental and regulatory bodies can use sentiment analysis to gauge public sentiment towards health policies or public health issues, aiding in policy formulation and public health interventions.
• Mental health analysis:  Sentiment analysis could potentially help monitor patients’ mental health by analyzing their written or spoken communication for signs of negative sentiment or distress.

AI-powered sentiment analysis in banking

AI sentiment analysis plays a significant role in banking by enhancing customer experience, risk management, and brand perception. Here are some potential use cases:

• Customer service improvement:  AI in sentiment analysis can analyze customer feedback from various sources such as social media, online reviews, and customer surveys. By identifying and understanding the sentiments, banks can improve their services and address customer complaints more effectively.
• Risk management:  Sentiment analysis can assist in early warning signal detection for credit risk management. For instance, negative sentiments from a business’s customers can indicate potential financial distress, which can be factored into the bank’s risk assessment process.
• Product and service development:  By understanding customer sentiments towards various banking products and services, banks can gain valuable insights into what customers appreciate and what they don’t. This can guide the development of new products and the refinement of existing services.
• Brand perception:  Sentiment analysis can help banks understand the overall perception of their brand in the marketplace. They can identify positive and negative sentiments, understand trends over time, and compare their sentiment score with competitors.
• Marketing strategy:  By analyzing customer sentiment towards various marketing campaigns, banks can understand which messages resonate with customers and why. This can help guide future marketing strategies.
• Customer segmentation:  Sentiment analysis can categorize customers based on their sentiments. This segmentation can tailor communication and offers to different customer segments, enhancing personalization and customer satisfaction.
• Fraud detection: While not a direct application, sentiment analysis can indirectly support fraud detection. Unusual negative sentiments or sudden sentiment changes can indicate potential fraudulent activity, triggering further investigation.

Streamlining sentiment analysis workflow with GenAI

Generative AI has transitioned from a futuristic concept to a transformative force, poised to fundamentally reshape the landscape of sentiment analysis. From streamlining workflows to unlocking deeper insights, GenAI is driving a paradigm shift in how we understand and respond to customer emotions and opinions.

This sub-section discuses different personas involved and how those might align:

• Sentiment analysis business analyst: Translates business needs into sentiment analysis goals and interprets results for actionable insights.
• Marketing manager: Leverages sentiment insights to optimize marketing strategies, campaigns, and brand perception.
• Sentiment analysis project manager: Oversees the entire sentiment analysis lifecycle, from project scoping to optimization and communication.
• Sentiment analysis data scientist: Builds, refines, and optimizes the technical models that power sentiment analysis.
• Workflow integration engineer: Ensures seamless integration of sentiment analysis into larger data workflows and systems.

Here’s a closer look at how GenAI is transforming key stages of the sentiment analysis lifecycle:

1. Sentiment Analysis Request

2. Sentiment Analysis Review

3. Sentiment Analysis Feedback

4. Sentiment Analysis Optimization

In conclusion, Generative AI is poised to transform sentiment analysis across its entire lifecycle, from data collection and processing to model training and insight generation. The ability of GenAI to understand and generate human-like text unlocks unprecedented potential for automating tasks, enhancing accuracy, and uncovering deeper emotional insights. However, a symbiotic relationship between human analysts and AI will be crucial. While GenAI can automate tasks and augment capabilities, human expertise remains essential for interpreting nuanced sentiments, ensuring ethical considerations, and ultimately, deriving actionable business intelligence from the sea of data.

LeewayHertz’s AI development services for sentiment analysis

At LeewayHertz, we craft tailored AI solutions that cater to the unique requirements of sentiment analysis across various industries. We provide strategic AI/ML consulting that enables organizations to harness AI for enhanced sentiment detection, improved customer engagement, and optimized marketing strategies.

Our expertise in developing Proof of Concepts (PoCs) and Minimum Viable Products (MVPs) allows firms to preview the potential impacts of AI tools in real scenarios, ensuring that the solutions are both effective and tailored to specific needs.

Our work in generative AI also transforms routine tasks like sentiment report generation and data analysis, automating these processes to free up analysts for more strategic roles.

By fine-tuning large language models to understand the nuances of industry-specific terminology and customer interactions, LeewayHertz enhances the accuracy and relevance of AI-driven sentiment analysis.

Additionally, we ensure these AI systems integrate seamlessly with existing technological infrastructures, enhancing operational efficiency and decision-making in sentiment analysis applications.

Our AI solutions development expertise

AI solutions development for sentiment analysis typically involves creating systems that enhance decision-making, automate routine tasks, and personalize customer interactions. These solutions integrate key components such as data aggregation technologies, which compile and analyze feedback from diverse sources, including social media, reviews, and customer surveys. This comprehensive data foundation supports predictive analytics capabilities, allowing for the forecasting of customer sentiment trends that inform strategic decisions.

Additionally, machine learning algorithms are employed to tailor sentiment analysis to individual client profiles, ensuring that each client’s unique preferences and feedback patterns are considered. These solutions often cover areas like customer feedback analysis, market research, brand reputation management, and customer relationship management.

Overall, AI solutions in sentiment analysis aim to optimize customer insights, improve operational efficiency, and elevate the customer experience by providing accurate, timely, and actionable analysis of sentiments.

AI agent/copilot development for sentiment analysis

LeewayHertz builds custom AI agents and copilots that enhance various sentiment analysis operations, enabling companies to save time and resources while facilitating faster decision-making. Here is how they help:

Sentiment analysis:

• Performing comprehensive sentiment analysis on social media posts, reviews, and customer feedback.
• Identifying potential customer issues and opportunities based on predefined criteria or rules.
• Analyzing trends in customer sentiment by processing historical and real-time data, helping to predict future customer behavior.

Sentiment-driven customer engagement:

• Analyzing customer sentiment expressed in text, voice, or even facial expressions (if video is used). This allows them to tailor responses that are not only relevant but also empathetic to the customer’s emotional state.

• Identifying customers expressing negative sentiment early on. The agent can proactively reach out to offer solutions, provide reassurance, or simply acknowledge their concerns.

• By analyzing sentiment across different customer groups, AI agents can identify segments with specific emotional needs or preferences. This allows for targeted engagement strategies.

• Tracking customer sentiment over time and analyze its evolution in response to specific actions or campaigns. This provides valuable data for refining engagement strategies and improving the customer experience.

Sentiment-driven operations:

• Dynamically adjusting operations by continuously monitoring customer sentiment across channels (social media, reviews, surveys, etc.). Based on real-time sentiment trends, they can trigger adjustments to operational processes.

• Proactive customer engagement can identify customers expressing high levels of positive sentiment. It can then automatically engage these customers with tailored offers, loyalty programs, or personalized content to strengthen their relationship with the brand.

• Optimizing resource allocation by analyzing sentiment across different customer segments or product areas, AI can help companies prioritize resource allocation. For example, if sentiment is consistently negative for a particular product line, the AI agent can recommend shifting resources to product development or marketing efforts to address the issue.

• In industries like customer service, sentiment analysis can be used to predict potential issues or escalations. If negative sentiment escalates significantly around a specific product or service, AI agent can alert support teams to be prepared for an influx of inquiries and proactively address concerns.

Market analysis:

• Gathering and analyzing sentiment data from diverse sources, providing businesses with a holistic view of public opinion.
• Customizing marketing strategies based on identified sentiment trends and customer feedback.
• Providing real-time insights into market conditions and brand perception, supporting timely and informed decision-making.

Brand management:

• Recommending basic strategies for improving brand perception based on sentiment analysis.
• Identifying negative sentiment trends and suggesting actions to mitigate potential damage.
• Monitoring brand reputation and providing insights for proactive management.

Marketing and content generation:

• Analyzing customer sentiment across various channels (social media, reviews, surveys) to understand their preferences, pain points, and emotional responses. This information can then be used to generate personalized marketing messages and content that resonates with individual customers.
• Analyzing customer sentiment data to segment audiences based on emotional responses to different products, services, or campaigns. This enables more effective targeted marketing strategies.
• Analyzing the emotional tone of existing marketing materials and identify areas for improvement. The agent can then suggest language, imagery, or messaging that resonates better with target audiences based on their sentiment.
• Monitoring customer sentiment in real-time and adjust content strategies accordingly. If a campaign is generating negative sentiment, the AI agent can suggest adjustments to messaging, offers, or visual elements to improve customer response.

Customer segmentation and targeting:

• Analyzing client data to segment customers based on predefined criteria (e.g., demographics, sentiment profiles, preferences).
• Identifying potential cross-selling or upselling opportunities based on customer segments.

AI agents/copilots not only increase the efficiency of operational processes but also significantly enhance the quality of customer service and strategic decision-making. By integrating these advanced AI solutions into their existing infrastructure, businesses can achieve a significant competitive advantage, navigating the complex landscape with innovative, efficient, and reliable AI-driven tools and strategies.

How to do sentiment analysis using LSTM?

We will show how to do sentiment analysis on movie reviews using LSTMs. LSTM networks are enhanced versions of RNNs specifically designed to learn sequential data and its long-range dependencies more effectively than traditional RNNs. They find wide applications in areas of deep learning, such as predicting stock trends, recognizing speech, and processing natural language.

We will be using the IMDB movie review dataset that can be downloaded from this link – http://ai.stanford.edu/~amaas/data/sentiment/

Follow the undermentioned steps.

Import and display the data

Download the data from the given link and load it using the below code to view the data:

# read data from text files
with open('aclImdb/test/neg/10000_4.txt', 'r') as f:
 reviews = f.read()
with open('aclImdb/test/neg/10000_4.txt', 'r') as f:
 labels = f.read()
print(reviews[:50])
print()
print(labels[:26])
 
Output - This is an example of why the majority of action f
 
This is an example of why

Now convert the data into lowercase and remove the punctuation

reviews = reviews.lower()
from string import punctuation
print(punctuation)
all_text = ''.join([c for c in reviews if c not in punctuation])

Data processing: Generate a list of reviews

We have consolidated all the text into one large string. Now, our next step is to split this into individual reviews and store them as separate elements in a list, such as [review_1, review_2, review_3… review_n].

reviews_split = all_text.split('\n')
print ('Number of reviews :', len(reviews_split))
 
Output - Number of reviews : 1

Tokenization: Establish a dictionary mapping vocabulary to integers

For many Natural Language Processing (NLP) tasks, it’s common to construct a dictionary that maps indexes in a way that words appearing more frequently are assigned lower indexes. A popular approach for accomplishing this involves the use of the Counter method from the Collections library.

from collections import Counter
all_text2 = ' '.join(reviews_split)
# create a list of words
words = all_text2.split()
# Count all the words using Counter Method
count_words = Counter(words)
 
total_words = len(words)
sorted_words = count_words.most_common(total_words)

To establish a dictionary mapping vocabulary to integers, you would proceed as follows:

vocab_to_int = {w:i for i, (w,c) in enumerate(sorted_words)}

Here’s a minor detail to consider: In this mapping, indexing begins from 0, meaning ‘the’ would be mapped to 0. However, we will later need to add padding to shorter reviews, and the standard choice for padding is 0. Therefore, it’s necessary to start this indexing from 1.

vocab_to_int = {w:i+1 for i, (w,c) in enumerate(sorted_words)}

Tokenization: Convert words into an encoded form

Up until now, we have crafted a) a list of reviews and b) a dictionary mapping indexes using the vocabulary derived from all our reviews. The objective of these steps was to encode the reviews, replacing the words in our reviews with corresponding integers.

reviews_int = []
for review in reviews_split:
    r = [vocab_to_int[w] for w in review.split()]
    reviews_int.append(r)
    print (reviews_int[0:3])
 
Output - [[3, 8, 43, 44, 4, 20, 1, 45, 4, 21, 22, 9, 1, 10, 46, 2, 47, 48, 23, 49, 24, 25, 50, 5, 51, 52, 4, 1, 53, 54, 55, 4, 26, 2, 27, 28, 56, 57, 58, 59, 60, 61, 29, 62, 9, 63, 4, 11, 2, 11, 64, 65, 66, 30, 3, 31, 67, 6, 32, 68, 69, 70, 12, 33, 32, 71, 72, 13, 26, 12, 73, 74, 1, 75, 76, 77, 12, 78, 13, 27, 28, 2, 6, 1, 79, 80, 81, 82, 83, 84, 85, 86, 3, 7, 87, 14, 1, 88, 2, 89, 90, 91, 92, 1, 34, 35, 36, 15, 93, 37, 3, 7, 2, 20, 1, 34, 94, 95, 96, 97, 1, 38, 10, 16, 98, 39, 99, 100, 7, 101, 102, 30, 35, 36, 40, 103, 104, 1, 38, 10, 105, 16, 2, 14, 106, 37, 39, 107, 16, 108, 17, 41, 109, 110, 111, 112, 113, 3, 8, 114, 21, 115, 116, 9, 117, 18, 22, 19, 6, 2, 118, 119, 23, 120, 19, 6, 3, 31, 33, 121, 122, 17, 8, 123, 5, 124, 125, 42, 40, 18, 11, 2, 5, 18, 126, 1, 127, 128, 29, 41, 3, 14, 129, 24, 25, 15, 5, 130, 131, 132, 1, 133, 1, 134, 15, 135, 136, 17, 137, 138, 19, 139, 140, 13, 1, 141, 7, 142, 42, 143, 144, 145]]

Tokenization: Convert labels into encoded form

This step is straightforward as we have only two output labels. Accordingly, we will designate ‘positive’ with the number 1 and ‘negative’ with the number 0.

import numpy as np
labels_split = ['positive', 'negative', 'positive', 'positive', 'negative']
 
encoded_labels = [1 if label =='positive' else 0 for label in labels_split]
encoded_labels = np.array(encoded_labels)

Examine the lengths of reviews

import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
reviews_len = [len(x) for x in reviews_int]
pd.Series(reviews_len).hist()
plt.show()
pd.Series(reviews_len).describe()
 
Output
count      1.0
mean     232.0
std        NaN
min      232.0
25%      232.0
50%      232.0
75%      232.0
max      232.0
dtype: float64

The generated plot looks like the following:

Eliminating outliers: Disposing of excessively long or short reviews

reviews_int = [ reviews_int[i] for i, l in enumerate(reviews_len) if l>0 ]

Padding or truncating the remaining data

To manage both short and long reviews, we will adjust all our reviews to a specified length, which we will refer to as the Sequence Length. This length corresponds to the number of time steps for the LSTM layer.

For reviews shorter than the sequence length, we will add padding with 0s. Conversely, for reviews longer than the sequence length, we will truncate them to include only the first set of words up to the sequence length.

def pad_features(reviews_int, seq_length):
    ''' Return features of review_ints, where each review is padded with 0's or truncated to the input seq_length.
    '''
    features = np.zeros((len(reviews_int), seq_length), dtype = int)
 
    for i, review in enumerate(reviews_int):
        review_len = len(review)
 
        if review_len <= seq_length:             zeroes = list(np.zeros(seq_length-review_len))             new = zeroes+review         elif review_len > seq_length:
            new = review[0:seq_length]
 
        features[i,:] = np.array(new)
 
    return features

Splitting the dataset into training, validation, and test sets

After preprocessing our data into an appropriate format, we will divide it into training, validation, and test sets.

The distribution will be as follows: 80% for training, 10% for validation, and 10% for testing.

# Assuming you have defined the reviews_int variable with your data
# and you want to set the sequence length to 100
seq_length = 100
 
# Call the pad_features function with reviews_int and seq_length
features = pad_features(reviews_int, seq_length)
 
# The rest of the code remains the same
split_frac = 0.8
train_x = features[0:int(split_frac*len(reviews_int))]
train_y = encoded_labels[0:int(split_frac*len(reviews_int))]
remaining_x = features[int(split_frac*len(reviews_int)):]
remaining_y = encoded_labels[int(split_frac*len(reviews_int)):]
valid_x = remaining_x[0:int(len(remaining_x)*0.5)]
valid_y = remaining_y[0:int(len(remaining_y)*0.5)]
test_x = remaining_x[int(len(remaining_x)*0.5):]
test_y = remaining_y[int(len(remaining_y)*0.5):]

Loading data and organizing it into batches

Once we have separated our data into training, testing, and validation sets, the following step is to construct dataloaders for this data. Rather than using a generator function to batch our data, we will opt to use a TensorDataset.

import numpy as np
import torch
from torch.utils.data import DataLoader, TensorDataset
 
# Assuming you have some data in numpy arrays
train_x = np.random.rand(100, 10)  # 100 samples, 10 features
train_y = np.random.randint(0, 2, size=(100,))  # 100 labels (binary)
 
# Create Tensor datasets
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))
 
# DataLoader
batch_size = 50
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)

To procure a single batch of training data for visualization purposes, we will establish a data iterator.

# obtain one batch of training data
dataiter = iter(train_loader)
sample_x, sample_y = next(dataiter) # Use `next()` directly on the iter object
print('Sample input size: ', sample_x.size()) # batch_size, seq_length
print('Sample input: \n', sample_x)
print()
print('Sample label size: ', sample_y.size()) # batch_size
print('Sample label: \n', sample_y)

Sample input size:  torch.Size([50, 10])
Sample input:
 tensor([[0.7034, 0.4482, 0.8321, 0.6271, 0.6431, 0.6657, 0.8682, 0.8476, 0.3166,
         0.4798],
        [0.4986, 0.0200, 0.4595, 0.9254, 0.8145, 0.7975, 0.7027, 0.5315, 0.0633,
         0.3708],
        [0.2233, 0.8519, 0.2751, 0.1726, 0.0131, 0.9168, 0.4826, 0.2859, 0.8347,
         0.3571],
        [0.4834, 0.6181, 0.8814, 0.6987, 0.2538, 0.8455, 0.0361, 0.4932, 0.1716,
         0.6979],
        [0.3436, 0.2338, 0.9501, 0.2772, 0.7758, 0.3319, 0.4560, 0.0637, 0.3984,
         0.8067],
        [0.6528, 0.1932, 0.9580, 0.2130, 0.9449, 0.8794, 0.5506, 0.7223, 0.9259,
         0.3914],
        [0.1691, 0.2396, 0.1201, 0.0938, 0.6786, 0.4733, 0.2956, 0.0286, 0.2427,
         0.1807],
        [0.4079, 0.6814, 0.9060, 0.3632, 0.1305, 0.5444, 0.7909, 0.1413, 0.2312,
         0.9908],
        [0.0060, 0.9553, 0.6465, 0.8999, 0.4546, 0.5736, 0.0576, 0.6150, 0.7116,
         0.3232],
        [0.0087, 0.6969, 0.3452, 0.0889, 0.4847, 0.3626, 0.9049, 0.7747, 0.6826,
         0.4671],
        [0.3622, 0.2621, 0.0864, 0.4941, 0.5397, 0.5830, 0.5530, 0.5423, 0.5781,
         0.7268],
        [0.0919, 0.5907, 0.2638, 0.1189, 0.9791, 0.5678, 0.2447, 0.9174, 0.4527,
         0.2479],
        [0.1109, 0.1358, 0.4870, 0.2571, 0.0105, 0.8599, 0.5062, 0.3309, 0.1940,
         0.8816],
        [0.8237, 0.7502, 0.0300, 0.6941, 0.7874, 0.8147, 0.7814, 0.5930, 0.7264,
         0.2516],
        [0.4720, 0.8332, 0.7371, 0.5548, 0.3322, 0.1917, 0.2605, 0.6905, 0.0773,
         0.0194],
        [0.8260, 0.5284, 0.6566, 0.1853, 0.6731, 0.7663, 0.9669, 0.5206, 0.5350,
         0.5815],
        [0.0195, 0.9383, 0.4579, 0.4509, 0.5258, 0.4668, 0.0694, 0.7823, 0.5013,
         0.0329],
        [0.8956, 0.6812, 0.1840, 0.6256, 0.5740, 0.5031, 0.0395, 0.8363, 0.4525,
         0.3345],
        [0.5260, 0.8632, 0.0893, 0.1557, 0.1420, 0.2395, 0.2133, 0.7120, 0.4225,
         0.9701],
        [0.6108, 0.6718, 0.9251, 0.4928, 0.2130, 0.7672, 0.9414, 0.6192, 0.8826,
         0.1397],
        [0.1588, 0.8059, 0.1725, 0.1385, 0.8286, 0.5996, 0.9071, 0.1250, 0.0257,
         0.1717],
        [0.7239, 0.5421, 0.1405, 0.0718, 0.6385, 0.4447, 0.9896, 0.0021, 0.2930,
         0.3223],
        [0.2663, 0.8737, 0.4490, 0.8135, 0.3247, 0.9046, 0.9639, 0.8595, 0.9099,
         0.2215],
        [0.4407, 0.1478, 0.3036, 0.2252, 0.3201, 0.7197, 0.1164, 0.7817, 0.2695,
         0.1697],
        [0.1724, 0.6860, 0.8862, 0.1776, 0.5827, 0.7069, 0.9616, 0.6420, 0.4005,
         0.2195],
        [0.9335, 0.1254, 0.6823, 0.2451, 0.7582, 0.2520, 0.9394, 0.6173, 0.9073,
         0.5894],
        [0.7955, 0.4576, 0.9754, 0.7944, 0.4832, 0.2182, 0.1606, 0.4541, 0.7654,
         0.0996],
        [0.5673, 0.1580, 0.1877, 0.7124, 0.3510, 0.8210, 0.5903, 0.3984, 0.0172,
         0.9169],
        [0.8406, 0.3152, 0.4214, 0.8054, 0.3103, 0.8748, 0.5084, 0.7876, 0.2713,
         0.9026],
        [0.1041, 0.3078, 0.0334, 0.9537, 0.8232, 0.7124, 0.1294, 0.3954, 0.5099,
         0.0601],
        [0.9727, 0.0047, 0.3879, 0.5295, 0.8541, 0.5677, 0.8425, 0.9426, 0.8628,
         0.9001],
        [0.5560, 0.8635, 0.4567, 0.5668, 0.3350, 0.5839, 0.0938, 0.0450, 0.7188,
         0.7714],
        [0.9290, 0.2526, 0.5834, 0.6354, 0.0252, 0.6823, 0.6056, 0.7914, 0.7256,
         0.9863],
        [0.4679, 0.6593, 0.4330, 0.2957, 0.8998, 0.9524, 0.8147, 0.0711, 0.0199,
         0.0072],
        [0.6717, 0.0452, 0.5355, 0.0869, 0.9485, 0.9067, 0.9364, 0.6426, 0.4703,
         0.1123],
        [0.1588, 0.2142, 0.3126, 0.5052, 0.1097, 0.5796, 0.9278, 0.2355, 0.3715,
         0.1877],
        [0.9141, 0.9462, 0.3081, 0.6432, 0.7723, 0.9893, 0.5752, 0.8356, 0.8120,
         0.8874],
        [0.5130, 0.8690, 0.8283, 0.6539, 0.0724, 0.3174, 0.7202, 0.2355, 0.2533,
         0.1136],
        [0.7707, 0.9367, 0.5900, 0.0450, 0.0664, 0.9667, 0.4897, 0.2897, 0.5269,
         0.0057],
        [0.4037, 0.1897, 0.8003, 0.9956, 0.9417, 0.7785, 0.1825, 0.3463, 0.7151,
         0.6947],
        [0.6328, 0.0575, 0.2658, 0.9864, 0.7538, 0.7379, 0.8595, 0.4063, 0.2231,
         0.8980],
        [0.5761, 0.4596, 0.4098, 0.7361, 0.2366, 0.5256, 0.4925, 0.4653, 0.4855,
         0.5675],
        [0.3936, 0.6751, 0.6700, 0.1433, 0.0736, 0.5637, 0.7160, 0.8280, 0.7033,
         0.9736],
        [0.9990, 0.1377, 0.2928, 0.8615, 0.1747, 0.0418, 0.0603, 0.3587, 0.6106,
         0.2169],
        [0.2078, 0.0172, 0.5001, 0.5338, 0.8021, 0.6420, 0.1756, 0.9057, 0.9402,
         0.9377],
        [0.9235, 0.1980, 0.1743, 0.4629, 0.3510, 0.5150, 0.2793, 0.7308, 0.4083,
         0.2104],
        [0.2590, 0.6794, 0.2936, 0.8421, 0.0090, 0.7507, 0.0512, 0.6522, 0.8703,
         0.3848],
        [0.9626, 0.6640, 0.1349, 0.9443, 0.9004, 0.5377, 0.3468, 0.2036, 0.4505,
         0.2096],
        [0.8717, 0.1118, 0.1710, 0.2745, 0.0250, 0.3887, 0.0661, 0.2466, 0.9903,
         0.8031],
        [0.4419, 0.4513, 0.9796, 0.0894, 0.0739, 0.5816, 0.8260, 0.9002, 0.4198,
         0.3433]], dtype=torch.float64)
 
Sample label size:  torch.Size([50])
Sample label:
 tensor([1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0,
        0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1,
        0, 1])

Next we need to feed the data into the LSTM network

Establishing the model class

import torch.nn as nn
 
  class SentimentLSTM(nn.Module):
      """
      The RNN model that will be used to perform Sentiment analysis.
      """
 
      def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
          """
          Initialize the model by setting up the layers.
          """
          super().__init__()
 
          self.output_size = output_size
          self.n_layers = n_layers
          self.hidden_dim = hidden_dim
 
          # embedding and LSTM layers
          self.embedding = nn.Embedding(vocab_size, embedding_dim)
          self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers,
                              dropout=drop_prob, batch_first=True)
 
          # dropout layer
          self.dropout = nn.Dropout(0.3)
 
          # linear and sigmoid layers
          self.fc = nn.Linear(hidden_dim, output_size)
          self.sig = nn.Sigmoid()
 
 
      def forward(self, x, hidden):
          """
          Perform a forward pass of our model on some input and hidden state.
          """
          batch_size = x.size(0)
 
          # embeddings and lstm_out
          embeds = self.embedding(x)
          lstm_out, hidden = self.lstm(embeds, hidden)
 
          # stack up lstm outputs
          lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
 
          # dropout and fully-connected layer
          out = self.dropout(lstm_out)
          out = self.fc(out)
          # sigmoid function
          sig_out = self.sig(out)
 
          # reshape to be batch_size first
          sig_out = sig_out.view(batch_size, -1)
          sig_out = sig_out[:, -1] # get last batch of labels
 
          # return last sigmoid output and hidden state
          return sig_out, hidden
 
 
      def init_hidden(self, batch_size):
          ''' Initializes hidden state '''
          # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
          # initialized to zero, for hidden state and cell state of LSTM
          weight = next(self.parameters()).data
 
          if (train_on_gpu):
              hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
                    weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())
          else:
              hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                        weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())
 
          return hidden

Train the network

Instantiate the network

# Instantiate the model w/ hyperparams
vocab_size = len(vocab_to_int) + 1  # +1 for the 0 padding
output_size = 1
embedding_dim = 400
hidden_dim = 256
n_layers = 2
net = SentimentLSTM(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)
print(net)
 
Output
SentimentLSTM(
  (embedding): Embedding(146, 400)
  (lstm): LSTM(400, 256, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.3, inplace=False)
  (fc): Linear(in_features=256, out_features=1, bias=True)
  (sig): Sigmoid()
)

Train the loop

# loss and optimization functions
lr=0.001
 
criterion = nn.BCELoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
 
train_on_gpu = torch.cuda.is_available()
# training params
 
epochs = 4 # 3-4 is approx where I noticed the validation loss stop decreasing
 
counter = 0
print_every = 100
clip=5 # gradient clipping
 
# move model to GPU, if available
if(train_on_gpu):
    net.cuda()
 
net.train()
# train for some number of epochs
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)
 
    # batch loop
    for inputs, labels in train_loader:
        counter += 1
 
        if(train_on_gpu):
            inputs, labels = inputs.cuda(), labels.cuda()
 
        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])
 
        # zero accumulated gradients
        net.zero_grad()
 
        # get the output from the model
        inputs = inputs.type(torch.LongTensor)
        output, h = net(inputs, h)
 
        # calculate the loss and perform backprop
        loss = criterion(output.squeeze(), labels.float())
        loss.backward()
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()
 
        # loss stats
        if counter % print_every == 0:
            # Get validation loss
            val_h = net.init_hidden(batch_size)
            val_losses = []
            net.eval()
            for inputs, labels in valid_loader:
 
                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                val_h = tuple([each.data for each in val_h])
 
                if(train_on_gpu):
                    inputs, labels = inputs.cuda(), labels.cuda()
 
                inputs = inputs.type(torch.LongTensor)
                output, val_h = net(inputs, val_h)
                val_loss = criterion(output.squeeze(), labels.float())
 
                val_losses.append(val_loss.item())
 
            net.train()
            print("Epoch: {}/{}...".format(e+1, epochs),
                  "Step: {}...".format(counter),
                  "Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(np.mean(val_losses)))

Testing

On test data

test_losses = [] # track loss
num_correct = 0
 
# init hidden state
h = net.init_hidden(batch_size)
 
net.eval()
 
# Convert lists to NumPy arrays
test_x = np.array(test_x)
test_y = np.array(test_y)
 
# Create Tensor datasets for test data
test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))
 
# DataLoader for test data
test_loader = DataLoader(test_data, shuffle=False, batch_size=batch_size)
 
# Move this line outside the loop to initialize the hidden state only once
val_h = net.init_hidden(1)  # Initialize for batch size 1
 
# iterate over test data
for inputs, labels in test_loader:
    # Move this line inside the loop to create a new hidden state for each batch
    h = val_h
    h = tuple([each.data for each in h])
 
    if train_on_gpu:
        inputs, labels = inputs.cuda(), labels.cuda()
 
    # get predicted outputs
    inputs = inputs.type(torch.LongTensor)
    output, h = net(inputs, h)
 
    # Calculate loss using BCEWithLogitsLoss
    test_loss = criterion(output, labels.float())
    test_losses.append(test_loss.item())
 
    # convert output probabilities to predicted class (0 or 1)
    pred = torch.round(output.squeeze())  # rounds to the nearest integer
 
    # compare predictions to true label
    correct_tensor = pred.eq(labels.float().view_as(pred))
    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())
    num_correct += np.sum(correct)
 
# -- stats! -- ##
# avg test loss
print("Test loss: {:.3f}".format(np.mean(test_losses)))
 
# accuracy over all test data
test_acc = num_correct / len(test_loader.dataset)
print("Test accuracy: {:.3f}".format(test_acc))

Generated output

Test loss: 0.695 Test accuracy: 0.000

On user-generated data

Initially, we will define a function called ‘tokenize’ that handles the pre-processing tasks. Subsequently, we will create a ‘predict’ function that delivers the final output after analyzing the user-supplied review.

import numpy as np
import torch
from string import punctuation
 
# Define the SentimentLSTM class and other necessary variables here
 
# Define the negative review text you want to test
test_review_neg = "This movie was really disappointing. The acting was bad and the plot made no sense."
 
# Tokenization and preprocessing
def tokenize_review(test_review):
    test_review = test_review.lower()
    test_text = ''.join([c for c in test_review if c not in punctuation])
    test_words = test_text.split()
    test_ints = [vocab_to_int[word] for word in test_words if word in vocab_to_int]
    return test_ints
 
# Define the sequence length
seq_length = 200
 
# Tokenize and preprocess the review
test_ints = tokenize_review(test_review_neg)
 
# Padding
features = pad_features([test_ints], seq_length)
 
# Convert to tensor
feature_tensor = torch.from_numpy(features)
 
# Model prediction
def predict(net, test_feature_tensor):
    net.eval()
    batch_size = test_feature_tensor.size(0)
    h = net.init_hidden(batch_size)
 
    if train_on_gpu:
        test_feature_tensor = test_feature_tensor.cuda()
 
    output, h = net(test_feature_tensor, h)
    pred = torch.round(output.squeeze())
    return pred.item()
 
# Predict the sentiment
prediction = predict(net, feature_tensor)
 
# Print the prediction
if prediction == 1:
    print("Positive review detected!")
else:
    print("Negative review detected.")

Generated output

Negative review detected.

Endnote

The influence of sentiment analysis, propelled by advanced AI techniques, is already exerting a transformative effect on global customer interactions. Despite its current impact, we have merely scratched the surface of its potential benefits. The thriving landscape of sentiment analysis as a service offers diverse options for businesses seeking to integrate this powerful tool into their operations. While its advantages may vary across industries, there is a consensus that sentiment analysis will play a pivotal role in elevating customer experience and engagement for enterprises in the foreseeable future.

Nevertheless, it’s important to recognize that the capabilities of AI do not replace the need for genuine customer comprehension. Rather, they equip us with more refined instruments for understanding customers more profoundly. These insights, facilitated by AI-driven sentiment analysis, empower businesses to not only decipher customer sentiments but also to translate those insights into tangible enhancements. By bridging the gap between technology and empathy, sentiment analysis stands as a cornerstone in the evolution of customer-centric strategies, guiding businesses toward more meaningful connections and enduring success.

Unlock deeper customer insights with our AI-powered sentiment analysis tools. Embrace the power of AI for effective decision-making with LeewayHertz’s AI experts!