• News
  • Columns
  • Interviews
  • BW Communities
  • Events
  • BW TV
  • Subscribe to Print
BW Businessworld

When Traditional Algorithms Fail As Trendlines Are Taken Over By Outliers

Demand estimate outcomes from the model are continuously assessed against the actual sales values which emerge over time

Photo Credit :

The COVID 19 scenario

As is very obvious now that COVID would be a long and painful battle for most of the business. Businesses would have to learn to live with reduced demand and an uncertain business environment.  

In this article, we discuss how to use modern data analytics to improve your demand forecasting and estimation processes. The months of February, March, April and May 2020 have become outliers in the sales value for almost all businesses. This may vary as per your region but for most companies, the business has come to a zero,  while for few others it is significantly down, and for some essential categories, it has even grown in this period. 

Imagine a real-life situation of a company operating in travel retail space. Their daily sales fluctuate a lot in retail depending on seasonality, trend and other factors. COVID crisis hit in the middle of March and sales dropped sharply and in a matter of days came to zero. This would now stay at zero till travel opens up again. 

But what now? in the near future?

Traditional time series forecasting software use algorithms which essentially looks at the historical sales and project the future. Post-COVID, when the businesses are projecting sales, these algorithms would consider these few months as outliers and exclude them from consideration set. This would result in incorrect forecasting leading to wastage of working capital and loss of sales due to low fulfilment.  

How can businesses project demand for the next 10 to 15 weeks as the time series models are now totally non-applicable? There is an urgent need for newer advanced approach & algorithms, which can help demand planners in this new world. This brings us to the nascent art of demand sensing.

Traditional perspective of Demand Sensing

Demand sensing has developed over the last few years as a tool for assisting demand planners in making demand projections more useful to the businesses. It is a tool for fine-grained break-down of projected demand into small units of time and space.  

In simple business language, if projected demand for the next 3 months for Brand X in geography P is 100, 120 and 110 respectively, demand sensing algorithms would break this down by week, days of the week or even hours each day for each of the subcategory. This aids in steering a business in planning their ordering, production and logistics over the short term. 

Demand Sensing takes demand projection as an input variable. It then adds other available information like holiday calendar, yesterday’s sales, customer research data, secondary sales data and gives a breakdown of projection into finer grains of time and space. Demand forecasting relies on time series and therefore can see only historical information while demand sensing can add many more intelligent variables along with historical information.

In the process, it also enhances the accuracy of the projection. It provides short terms demand forecast with an accuracy which goes beyond core trend and seasonality that might have been assessed from the past data analysis. In reality, this becomes far too complicated, given thousands of such factors and hundreds of SKUs. This is, therefore, all built into regression-based, ML-powered algorithms which perpetually enhance their accuracy over time.

Need to build Sentiment Analytics into Demand Sensing

The Demand Sensing models would begin with two basic inputs. These would be

Environmental factors (e.g., when will markets/stores/malls/airports etc. open up, what alternates consumers will have for your product/services)

Internal business factors (e.g., planned marketing or promotion activities)

Primarily it was lockdown which brought the industry down to its knees.  First, the retail stores were closed, then production units closed down affecting the total supply chains. Currently, even as lockdowns are being eased out, the customers are indoors and are worried about the pandemic. The undeniable fact, therefore, is that consumer sentiment is the main factor behind demand depression in the current scenario. 

A comprehensive consumer sentiment analysis is required to understand a) what the consumer is going through, b) what products and services will the consumer come back into the market? and c) when would he come back for which specific products and services?

Customer sentiment analytics, therefore, becomes the third pillar in the Demand Sensing construct we have seen earlier.  Therefore, in the current scenario, given below is how demand sensing constructs have to be modelled.

Sentiment Analytics for Demand Sensing algorithms

There are 3 pillars of sentiment analytics on which demand sense will rest in the post COVID scenario – relating to the consumer, relating to the category and lastly relating to specific brand sentiments.

These pillars of sentiment analytics are inter-connected and also often impact each other. Therefore, it is critical to understand them better. We are looking at 2 aspects of sentiment analytics in this article

1) How do these factors impact the demand for product/category/brand? 

2) How to measure/assess them to build them into Demand sensing modelling? 

The Core of Sentiment Analytics today is about understanding customer’s world view and his/her product categories view. The three parts of the Sentiment which needs to be measured are  

a) Consumer Anxiety Index

The proportion of consumers who are very worried are less likely to step out for what they consider less critical. Therefore, we need to assess anxiety quotient for the target audience and track the proportion of very worried, somewhat worried and not worried. The impact of anxiety on consumer behaviour, however, is not a standalone variable but interacts with category criticality. The behaviour with respect to a category considered critical may not get affected despite extreme anxiety and on the other hand behaviour with respect to a category considered non-critical would get severely affected at a moderate anxiety level. 

b) Category Disposition Index

Category disposition index is a mean of 2 distinct uncorrelated aspects :

Criticality Index – how essential does one consider a specific category to be. This simply means that categories considered more essential are less affected by consumer anxiety.

Risk assessment with product/service. This captures the associated risk with the product or service. For instance, going to a movie hall has high-risk perception while going to the golf course has a much lower risk.

These two aspects are uncorrelated and have a simple additive relationship, therefore for a category Disposition index is assessed as a mean of the two measures.

An anxious consumer is likely to avoid air travel for pleasure while having no second thoughts about buying regular groceries. Therefore, it is of utmost importance to build co-interactions of consumer anxiety and category disposition into Demand Sensing models. 

c) Brand Affinity Index

In the times of COVID, consumer’s relationship with brands may have taken a shift. This could be assumed to be of no importance if we are sure that there has been no reason for consumer-brand relationship shift and, in those cases, the index is assumed to be 1.

However, some brands could see a significant shift on this for reasons like a patriotic fervour to support domestic corporations or avoidance of certain corporations due to their origins or may have repositioned themselves because of their contribution to consumer, society, country,  or humanity in these troubled times.  In such instances, the relationship index could range from 0 to 2 where 1 stands for the status quo. This is then built into the demand estimates for the brand in the post COVID scenario

How to measure Sentiment Analytics

The consumer anxiety index, as well as category disposition index, can be measured on a continuous basis through an online survey. The survey could be instituted for specific categories and data collected by using online channels or through SMS based campaigns on clients’ existing databases. 

Revisiting the problem case of travel retail client in post COVID scenario

We have mentioned above the case of a company operating in travel retail space. The overall sales dropped sharply when COVID crisis hit and in a matter of days came to zero and stayed at zero for a few weeks. We need to use Demand Sensing modelling to assess its total likely demand when the market place opens. This has been assessed below using the revised Demand Sensing Model for 7 days after the market place reopens.


Estimated customer inflow – e.g. 60% of usual capacity. Data can be collected from the number of operating flights

Consumer Anxiety – e.g. 55%. Data can be collected from the online survey with a relevant geographic reach

Category Disposition Index  – e.g. 30%. Data can be collected from the online survey with a relevant geographic reach

Brand Affinity Index – e.g. 1.  Qualitatively assessed. Unless there is a strong reason, this would be 1

Base values – taken from projections done on historical data from pre-COVID times for this time period 

Demand estimate outcomes from the model are continuously assessed against the actual sales values which emerge over time. The model will use machine learning algorithms to continuously evolve and enhance the accuracy of daily projections. 


This model is an example of advanced tools which can be used for improving the demand forecasting estimates. These can be used for all types of retail stores, spanning across products and services, travel and tourism, and FMCG products. The model would initially be used for short term projection into near-future time spans. However, once we have established clear trends of consumer sentiments over time, it will also be used for long term projections also. It would immensely help businesses to improve their forecasting and conserving working capital by saving inventory investment in an uncertain business environment.

Disclaimer: The views expressed in the article above are those of the authors' and do not necessarily represent or reflect the views of this publishing house. Unless otherwise noted, the author is writing in his/her personal capacity. They are not intended and should not be thought to represent official ideas, attitudes, or policies of any agency or institution.

Tags assigned to this article:

Shubhra Misra

The author is the Head of Research at Voiceback Analytics- a Bengaluru based Data Analytics startup. She has worked as a data scientist for the last 25 years. She is an expert in the field of market research and structured & unstructured data analytics. She has worked with corporates like Biocon, Pfizer, Eveready, Mylan, Abbott, etc. on multiple projects using primary, secondary research, and advanced big data analytics. Shubhra Misra is an MBA from IIM Lucknow, India.

More From The Author >>

Rishi Sanwal

The author is Visiting Faculty at ISB Hyderabad & IIM Kashipur. He has been a management consultant in Supply Chain & Operations area since 2001. He has worked at Accenture and IBM Business Consulting, advising large multinational and Indian firms like PepsiCo, Unilever, Heinz, Maruti, Bajaj Auto, L&T, and many others in improving their supply chain performances. His areas of interest are Forecasting, Optimization, Supply Chain Analytics, and Project Management. He is a graduate from IIT Bombay, India & MBA from IIM Ahmedabad, India

More From The Author >>