Showing posts with label analytics. Show all posts
Showing posts with label analytics. Show all posts

Sunday, December 23, 2012

Web Content Management: Opportunity to integrate with Big Data Analytics and Machine Learning technology!

Web Content Management (WCM) space has come a long way in terms of maturity from mid 90s to where it is today. Many companies, specially the ones who rely on their online business for revenue generation, are probably going through third or fourth iteration of overhauling their WCM solution. There are hundreds of commercial and open source offerings in this space catering to all sizes and different needs.The WCM vendors have also done a good job in keeping pace with the customers' demands. In fact, WCM has graduated to being part of a Web Experience Management (WEM) or Customer Experience Management (CXM) where WCM is one of the aspects of the solution along with  web analytics, optimization, targeting, A/B testing, social media integration, personalization and marketing integration. Overall, the space is matured and all major players from HP, Adobe, Oracle, Opentext, SDL, IBM, Microsoft and hundreds of others are playing in it. All of them are trying hard to find their own niche and differentiation as it is a crowded space.

CMS tools have also become more business friendly and no longer require the level of involvement of IT for every small task. Now the business has better ability to do tasks like content creation, presentation management and workflow without IT intervention. These tools continue to offer advanced features like in-context editing, multi-site management, multi-channel delivery, social media integration, globalization etc.. There is definitely room for improvement in the areas of multi-channel delivery, globalization, social media etc.. All leading products support standards like J2EE/.Net and have a cloud offering so that is no longer a competitive differentiation. Mobile is definitely a must-have thing now and everyone, CMS vendors as well as businesses, is trying to keep pace with the increasing number of varied mobile devices in the market. Definitely, there are more channels and platforms to worry about now. Content is no longer though of as a web page now! It doesn't mean because of these advancements in tools and technology, all projects run smoothly. They don't and the main issue is lack of CMS experience, inability to structure CMS projects correctly from timeline perspective and in some cases because of disconnect between business and IT. I also wonder where do we go from here as far as capabilities of CMS tool is concerned?

The focus on building an integrated digital marketing experience by vendors where content is at the core is the right thing but one important aspect is still lacking. That aspect is ability to have recommendation and deep personalization as part of the experience. The bar is set by Amazon.com and it is the model  everyone can look upto. It is not an easy thing to build but CMS vendors need to think of ways to offer templates which can enable it. They also need to think of  innovative ways to integrate with all the work that is happening in Big data Analytics world. In my experience, companies have a very good understanding about the level of personalization and recommendation they should provide to their end users. They just don't know how! They are just constrained by the limitations of their IT departments to deliver this functionality. IT also has a big challenge as they don't have access to out-of-the-box tools or an easier path to provide such functionality. Analytics and machine learning, technology behind deep personalization and recommendations, are not understood very well by many IT departments. It is also not easy to build as an  investment like an amazon.com in recommendation technology is not easily affordable. Something to keep in mind for CMS tool vendors!

Thursday, September 8, 2011

Big Data : Do we need more use cases?

Today, we can comfortably say that Big Data is an accepted term or a concept - most of the people who work in information technology have at least some definition about it. Well, everyone understands that data is growing at exponential speed; it will continue to do so for decades and there is no reason for it to stop. New analytical needs, not well suited to existing data warehouses, and growing volumes of source data are one of the biggest drivers to this relatively new space. Exponential drop in the cost of bandwidth, storage and computing makes big data applications possible and economical. The wider acceptance of cloud computing has made the business case for big data even stronger. And as companies move from proccess-driven business to analytics-driven business, the interest in big data is only going to multiply in the coming years. So overall its great news for all enterpreneurs, technologists and vendors who want to offer products and services in the big data space.

Almost every second day there is some interesting news in the big data world - last Tuesday Fujitsu announced that they will offer a cloud platform to leverage big data and MapR, an apache Hadoop distributor, has just secured a twenty million dollar funding led by Redpoint ventures. There are numerous others like couch.io, 10gen, Cloudera, Neo Technology, Loggly, Hypertable etc. who got funded earlier. Today, market seems to be most receptive to Apache's Hadoop-based distributions which includes offerings from companies like Cloudera, IBM and EMC.

But big data adoption has still not reached tipping point in enterprises even though you will find many pilot projects. The primary reason being that there is still confusion around which problems are big data problems. What are the right questions to ask? Basically, how do you start and structure a project? What is the scope of a project? Cloudera has done a great job in raising the awareness but still the focus is very technical. The story is very different if you consider pioneers like Google, Yahoo and early adopters like Linkedin, Facebook etc.. Big data is critical part of their business and their needs are very different. Startups in the web space like Bitly, Foursquare and many others have similar needs - maybe not in terms of scale but at least in terms of flavor. Also, not every company is going to have a petabyte problem - infact, most of the companies have terabytes problem. The ecosystem of tools in this space is maturing very quickly, though still immature as compared to other fields. The majority of the existing use cases are around click stream analysis, log analysis, marketing analytics, text processing etc.. For e.g. NTT Communications built a log analysis system for marketing using hadoop, which explore the internet users’ interests or feedback about specified products or themes from access log, query. CBS interactive used Hadoop as the web analytics platform, processing one billion weblogs daily from hundreds of web site properties at CBS Interactive. Right now, targeted advertising is the biggest market for big data applications and there is lots of interest in using social media analytics from big data. While the problem of  finding unexpected patterns in unstructured data makes sense but there is lots of work to be done for structured data. Because not all problems in the enterprise is going to be around unstructured data specially if you consider industries like financial services. The schemaless and flexible aspect of nosql technologies used to solve big data problems is begining to be viewed as very attractive specifically for ETL kind of operations.

In the end, the promise of big data is about building models, cause and effect relationships and predicting outcomes. The expectations are high as it is even considered critical path to building personalized medicines in future. But we need to take various baby steps before we get there. Not having the right set of skillset to focus on big data is another problem and having a steep learning curve to understand technologies like Hadoop/Mapreduce doesn't make it easy also. Abstractions over Mapreduce like cascading, pig and hive are good approaches but are still relatively new. Market for use case driven applications is least understood and not tapped. Using public cloud for big data seems no brainer from cost perspective but there are practical problems like how do you move large amounts of big data to/from the cloud. Scalability problem is sorted out using technology like Hadoop but building analytical applications is where the challenge is. Big data is here to stay and the space has become very interesting but many unknowns are still there before it starts becoming mainstream.

Thursday, November 11, 2010

Predictive Analytics for social data: Is there a role for Semantics?

Recently, Gartner, the leading technology analyst company, came out with its predictions for key technologies for 2011. The list comprises of cloud computing, mobile applications, social collaboration, next generation analytics, social analytics and many others - shouldn’t surprise you if work in information technology. The only thing which was not clear to me that why next generation analytics and social analytics are in two different categories when social collaboration is already emphasized as a part of a roadmap for large enterprises. I am sure they must be having their own valid reasons to do so. But the point is that overall it is getting a bit confusing about the various terms which are being used in context of analytics. What I mean here is what is the difference between just analytics versus predictive analytics versus forecasting versus predictive modeling versus optimization versus data mining versus advanced analytics. To many people, it sounds same! Also, it will depend upon who you ask this question. If you are a vendor or a consultant then try explaining it to a decision maker in an enterprise - basically, try not to get into that conversation. Many of these disciplines are more than a decade old as something like predictive modeling has been used in credit scoring for years. Also, academically, there is not much difference between classic techniques used in data mining and in statistics. Though, data mining has evolved to deal better with real life messy data. Unfortunately, unlike analytics, statistics could never become a hot topic but maybe it is about to change. In short, analytics or predictive analytics is the umbrella term or the new term - maybe the buzz word. It seems there is new surge of interest in predictive analytics because it is about the future outcomes in context of business intelligence. There is a difference between insights and gaining foresight!

You can always question that companies were always worried about the future outcomes so what has changed now if many of the methods were available before also. Probably, more data is available now, and there has been advancement and simplification of  tools/techniques - you can hire a good business analyst to do the job instead of someone with a doctorate in statistics. Predictive analytics enables you to develop mathematical models to help you better understand the variables driving success. Predictive analytics relies on formulas that compare past successes and failures, and then uses those formulas to predict future outcomes. Also, if you consider the fact that IBM has spent almost eleven billion dollar in the last five years acquiring software companies, like SPSS and Unica, for its analytics consulting organization then it starts making more sense.

On top of that, social analytics is a new kid on the block and there is new buzz that it is going to play a significant role in predictive analytics. I do believe that it will become true gradually but it is  not going to happen as quickly as we are being made to believe. What are the challenges in it? From the process perspective, predictive analytics is about understanding the prediction variables to the business problem, selecting the relevant statistical technique, validating the model with the test data and finally applying/adjusting the model iteratively with the production data. Do you think that it should be very different in context of social data? First of all, just because a company is doing brand monitoring (there are just too many companies), it doesn't make it a social analytics company. What I mean here is that if they are just following converations about an entity and don’t have much semantic intelligence in their software. If you look at most of the common examples of so called social media helping in predictions, they are about topics about election predictions (as recently claimed by Facebook political team) or about how a new product like iPad is perceived - the outcome is most of the time boolean i.e success or failure. In my opinion, these are interesting examples but much more is expected to do predictive analytics from social data. Maybe, if you consider social analytics and all associated prediction with it as a seperate or standalone discipline then it is good enough - but then you don't have an integrated view from an enterprise perspective. Maybe, in some cases, you don't need to integrate social data with enterprise data and still can get some value. But, you still need to build a repeatable predictive model using the social media data. And before you build the predictive model, you need to do true semantic analysis of the social data. Build some kind of normalized social data model to work with enterprise data for predictive analytics. IBM has come out with a new offerring where it claims to have enhanced SPSS software with social analytics and it can do predictive analytics for your business needs. It also offers semantic network analysis of the social data. I am not aware how well it works or if it can work across different business domains without too much business analysis or customization.

It is not an easy task to build a repeatable predictive analytics with a everchanging large volume of social data. The quality of meaningful data is also very important in this context. I see the ambiguity of social data as one of the biggest hurdle. You will have to do lots of preprocessing and deal with many new attributes being added in the social context. Every company is unique and we will see predictive analytics manifesting itself differently for each of them. Though, I do believe that we will see many companies building very high number of predictive models which will take social data into consideration for their business needs - maybe a predictive model per product and total turn around time of a week or less from problem definition to scoring. That brings up another question - even if real-time social data is present, can you take real time actions? If not then what is the true value of real time data in context of predictive analytics? You really need a very different level of infrastruture to take advantage of it.
 
This whole integration of social media analytics with predictive analytics should be owned by business - not by IT.  Infact, in most of the cases, it should be owned by marketing departments because they value/understand social data more than any other department and they are also most qualified to define prediction goals from a social context, predictive behaviors, pragmatic tradeoffs and evaluating results.

It seems there are some good opportunities in this space. We still have a long way to go. A lot needs to be understood before making any claims in the industry. There are definetely some good examples in the research community like SOMA, a forecasting model developed by a researcher at the University of Maryland, Terror Organization Portal. It analyses a wide range of information about politics, business and society in Lebanon to predict, with surprising accuracy, rocket attacks by the country’s Hizbullah militia on Israel. By the middle of 2010 SOMA was sucking up data from more than 200 sources, many of them newspaper websites. There is another example of two researchers at HP Labs who have established that they can use tweets to predict how well a movie will do - the results turned out to be fairly accurate. What we don't understand that can these examples be generalized for the indusry adaption? It will be good to know more examples from the industry.