• News
  • Columns
  • Interviews
  • BW Communities
  • Events
  • BW TV
  • Subscribe to Print
  • Editorial Calendar 19-20
BW Businessworld

Just When It Happens

Photo Credit :

 Every minute, every second, the Internet generates huge amounts of data. This data is truly a goldmine, but almost no one knows where the gold lies or how to extract it. Businesses whose life depends on data have not figured it out either. The reason why this cannot be done is that online data changes its nature far too quickly, and companies often find that they are finding patterns that are no longer useful. Nevertheless, some companies are trying to make a business out of this problem.

Take analytics firm Truviso. Like so often in the Silicon Valley, this startup was born out of a doctoral thesis. Its founder Sailesh Krishnamurthy had worked out a way to analyse online data as soon as it was formed, before it entered relational databases. It turned out that this technique was useful to many firms, and most of all to advertising firms. Advertising firms target their online ads by mining consumer behaviour data, but not usually in real time — the actual time that it takes a process to occur. So, their analysis is usually old, and often a waste of time.
For example, the number of unique visitors to a site is a useful bit of information for advertising companies. This data is used to understand and analyse consumer behaviour, and then to target online advertisements through predictive models. But there is a glitch. Since information is processed in batches, with definite time lags between them, advertisements are often delivered to the wrong consumer. “Models change fast because consumer behaviour is variable,” says Krishnamurthy, founder and vice-president of technology at Truviso. The company’s method, which it calls continuous analytics, reduces this time lag.
Click here to view enlarged image
Continuous analytics is useful not just in advertising. Retail companies, for example, find that sales are intimately related to real-time online consumer behaviour, and often depend on chatter outside their own websites. Financial companies need to act on streaming data from capital markets. Telecom service providers need to analyse different kinds of data constantly in order to come up with innovative pricing models and new services. A new breed of real-time data analytics companies let many businesses do this task that seemed impossible till recently.
Catch On The Go
A few simple statistics will explain how useful real-time data analysis can be. The conventional way to deal with data is to clean and then put them in data warehouses, to be mined at leisure. But this poses a problem even if the subsequent analysis turns out to be useful. Data in warehouses are increasing at the rate of 173 per cent a year. This rate is faster than the processing power improvements of computers, which doubles in 18 months. This means it takes more and more resources to process data meaningfully. The problem is getting worse when you come to network data, which increases at the rate of 10 times a year. Sending all this data to a warehouse is not a sensible thing to do, as many companies have found out in the past few years.
As companies realise the value of the real-time Internet, companies are sprouting that can help them use it in many ways. StreamBase Systems, based in Lexington, Massachusetts, develops software that helps financial companies analyse capital market data in real time. Qlipso, based in Los Angeles, lets users share videos in real time, and also play games with each other. Hot Potato, based in New York, lets people 
create and share content around events. Qwisk, based in the Silicon Valley, enables users share content as they browse without a plug-in or download. And so on.
Of course, it is not just the startups who are interested in the real-time Web. Google had launched its real-time search two months ago. So did Microsoft. Both companies had deals with Twitter to enable tweets to be part of their search results. Of course, the real-time Internet was itself fuelled by the success of Twitter and Facebook, but the new breed of startups is trying to take it to a new level by adding significance to search results even as they reduce time lag.
Click here to view enlarged image
Wowd, for example, is a company based in the Silicon Valley, with a development team in Serbia. Wowd is trying to achieve a few new things, using a new approach to Web search. Traditional search engines use a technique called spidering. A spider, or a Web crawler, is an automated program that finds and indexes Web pages. This does not work well for the real-time Internet, as crawling can take a long time, and many pages change by the time they are indexed. Search engines get around this problem in many different ways, but Wowd thinks it has one of the best methods; one that does not use crawlers at all.
Instead of using crawlers, Wowd uses the signals that users generate to discover and rank websites. For example, tweets with weblinks are a strong indication of the quality of a Web page because someone has already discovered it and recommended it in some way. Tweets are explicit indications, but one could also use browsing behaviour to get powerful clues as to the quality of Web pages. You could also use the amount of time people spend on a page as an input for ranking.
This idea is not totally new, but Wowd is one of the few search engines that use it exclusively to search for Web content. Wowd hopes to provide more real-time results on a global scale. Says Mark Drummond, CEO of Wowd: “We hope to help in real-time discovery of topics across the Web, and not just a few popular sites.”