Priyank Mohan

Data Without Borders: Data can be a burden if it is not set free!

2011-09-29T22:17:00.001-04:00

Wikipedia describes philanthropy etymologically means "the love of humanity"— love in the sense of caring for, nourishing, developing, or enhancing. Historically, philanthropy has always been associated with giving generous donations of money. It will continue to be associated with giving generous donations but some professionals can make bigger impact by donating their skill set than money. Yes, I am talking about the skill set of data science! Data science is relatively a newly coined term and probably originated from data geeks working on hard data problems in the companies like Linkedin, Facebook and other technology companies who needed these experts to make sense and insights from the vast amount of data being produced everyday. Data Without Borders, a newly founded organization, seeks to match non-profits in need of data analysis with freelance and pro bono data scientists who can work to help them with data collection, analysis, visualization, or decision support. The concept is brilliant and makes sense!

There are various initiatives out there where technology is being leveraged creatively to help the non-profit organizations. The Bill and Melinda Gates Foundation has recently funded a new digital-media hub call ViewChange.org. The hub uses semantic technology to create a platform that combines the video sharing power of YouTube with the open information of Wikipedia and the mission of your favorite advocacy organization. I had written about it in more detail in one of my posts titled - Philanthropy goes Semantic. Ushahidi, initially started as a simple web site to map reports of violence in Kenya, is another non-profit tech company that specializes in developing free and open source software for information collection, visualizing and interactive mapping. To my knowledge, Hans Rosling, a medical doctor and a statistician with decades of work studying outbreaks in Africa, is probably the first data science philanthropist. He co-founded Gapminder foundation which developed the Trendalyzer software, acquired by Google, that converts international statistics into moving, interactive graphics. His TED presentation about his best stats you have ever seen is worth watching.

The genesis of the idea of "Data without Borders" is to match the NGOs, who are sitting on lots of data with nobody to look at because of resource and budget constraints, with data scientists who have the energy, time and passion to make sense of this data. Timing of this initiative couldn't be better because data scientists can now have a common and noble cause to rally behind! It is the beginning of a powerful vision but it will surely have its own challenges. Having some experience with an NGO myself, I can say that sustaining the enthusiasm and commitment of data scientist for a long-term can be challenging. We are all aware that data scientists are going to be one of the most sought after, busiest and highly paid professionals in the next decade! So I will go for a good data scientist with more commitment over a rock star data scientist in this context. Also, a weekend of data hackathon in this context will probably won't be enough because data Science is an iterative process and will require an ongoing engagement. It is still not clear to me that why there are not initiatives like open government data in case of NGOs to build powerful data mashups. I am aware of new standards like IATI but its more about aid spending by governments. In this context, I believe that too much data can be a burden if it is not set free and used effectively. Ideally, in case of NGOs, open data shouldn't have political or privacy barriers. In the end, the co-founders of "Data without Borders" will need all possible support, structure and maybe funding, to be successful in their mission. Winston Churchill, rightly said, "We make a living by what we get, but we make a life by what we give."

Big Data : Do we need more use cases?

2011-09-08T11:28:00.000-04:00

Today, we can comfortably say that Big Data is an accepted term or a concept - most of the people who work in information technology have at least some definition about it. Well, everyone understands that data is growing at exponential speed; it will continue to do so for decades and there is no reason for it to stop. New analytical needs, not well suited to existing data warehouses, and growing volumes of source data are one of the biggest drivers to this relatively new space. Exponential drop in the cost of bandwidth, storage and computing makes big data applications possible and economical. The wider acceptance of cloud computing has made the business case for big data even stronger. And as companies move from proccess-driven business to analytics-driven business, the interest in big data is only going to multiply in the coming years. So overall its great news for all enterpreneurs, technologists and vendors who want to offer products and services in the big data space.

Almost every second day there is some interesting news in the big data world - last Tuesday Fujitsu announced that they will offer a cloud platform to leverage big data and MapR, an apache Hadoop distributor, has just secured a twenty million dollar funding led by Redpoint ventures. There are numerous others like couch.io, 10gen, Cloudera, Neo Technology, Loggly, Hypertable etc. who got funded earlier. Today, market seems to be most receptive to Apache's Hadoop-based distributions which includes offerings from companies like Cloudera, IBM and EMC.

But big data adoption has still not reached tipping point in enterprises even though you will find many pilot projects. The primary reason being that there is still confusion around which problems are big data problems. What are the right questions to ask? Basically, how do you start and structure a project? What is the scope of a project? Cloudera has done a great job in raising the awareness but still the focus is very technical. The story is very different if you consider pioneers like Google, Yahoo and early adopters like Linkedin, Facebook etc.. Big data is critical part of their business and their needs are very different. Startups in the web space like Bitly, Foursquare and many others have similar needs - maybe not in terms of scale but at least in terms of flavor. Also, not every company is going to have a petabyte problem - infact, most of the companies have terabytes problem. The ecosystem of tools in this space is maturing very quickly, though still immature as compared to other fields. The majority of the existing use cases are around click stream analysis, log analysis, marketing analytics, text processing etc.. For e.g. NTT Communications built a log analysis system for marketing using hadoop, which explore the internet users’ interests or feedback about specified products or themes from access log, query. CBS interactive used Hadoop as the web analytics platform, processing one billion weblogs daily from hundreds of web site properties at CBS Interactive. Right now, targeted advertising is the biggest market for big data applications and there is lots of interest in using social media analytics from big data. While the problem of finding unexpected patterns in unstructured data makes sense but there is lots of work to be done for structured data. Because not all problems in the enterprise is going to be around unstructured data specially if you consider industries like financial services. The schemaless and flexible aspect of nosql technologies used to solve big data problems is begining to be viewed as very attractive specifically for ETL kind of operations.

In the end, the promise of big data is about building models, cause and effect relationships and predicting outcomes. The expectations are high as it is even considered critical path to building personalized medicines in future. But we need to take various baby steps before we get there. Not having the right set of skillset to focus on big data is another problem and having a steep learning curve to understand technologies like Hadoop/Mapreduce doesn't make it easy also. Abstractions over Mapreduce like cascading, pig and hive are good approaches but are still relatively new. Market for use case driven applications is least understood and not tapped. Using public cloud for big data seems no brainer from cost perspective but there are practical problems like how do you move large amounts of big data to/from the cloud. Scalability problem is sorted out using technology like Hadoop but building analytical applications is where the challenge is. Big data is here to stay and the space has become very interesting but many unknowns are still there before it starts becoming mainstream.

Predictive Analytics Conference!

2011-07-27T12:07:00.002-04:00

In my previous post, I had talked about "Debunking Myths about Watson" and now I am going to talk about Predictive Analytics World. So is there a connection? Well, one connection is Dr. David Ferruci who is an IBM Fellow and the principal investigator for the Watson/Jeopardy is one of the keynote speaker at Predictive Analytics conference. He also led the team who developed UIMA. Also, the other connection is that I had the opportunity to know Eric Siegal for some time, who is an expert in predictive analytics, founding chair of the Predictive Analytics World and is also a former computer science professor at Columbia University. Eric is very knowledgeable, practical and articulate about this area and really knows how to simplify this topic. The last reason is that line is blurring between so many of these technologies and we will continue to see overlap -- the effect will be that we will also see more and more reorganization in future enterprises. There is a always a difference between how vendors build categories for different technologies and how businesses view them.

Predictive Analytics by definition is a business intelligence technology that produces a predictive score for each customer or a prospect. Some people in the industry consider analytics as a different discipline than business intelligence -- because conventionally BI is more about what happened in the past or what is happening now but analytics is why it happened and what will happen in future. Predictive analytics (one of the key areas of analytics among areas like data mining, forecasting, optimization, text analytics etc.) is becoming increasingly important as marketing is the main business driver behind this discipline. It was already being used for many years in the applications for fraud detection, credit scoring and insurance pricing. But now there is almost an explosion of this discipline in the areas of direct marketing, customer retention, product recommendations, behavior-based advertising, email targeting and leads scoring. The academic term for predictive analytics is "machine learning". As Mark Twain said, "The art of prophesy is very difficult, especially with respect to future" but the good news is that predictive analytics doesn't need to be very accurate to provide value. It can help you answer questions like "People who buy life insurance are probably more likely to buy a luxury sedan." Probably, we know this already but it matters when you are dealing with large volumes of data about customers and their interactions with products and services. Also, if you know in advance, which of your customers are likely to leave you, you can take measures to hit only those customers with right campaigns to retain them. Business Intelligence doesn't get more actionable than that! Forrester research expects the growth to double within five years as ROI is very high.

You might consider going to Predictive Analytics World, October 16-21 in New York City (pawcon.com/nyc) which will give you deep dive in the Predictive Analytics. There is also a new conference Text Analytics World (tawgo.com/nyc), co-located with PAW NYC. You can get a 15% discount on the 2 Day Conference Pass by using this code: PMNY11

Debunking Myths about IBM's Watson!

2011-06-08T09:33:00.000-04:00

There are many instances of televised technological feats of human race which have managed to leave a lasting impression on us - In my opinion, probably, Apollo 11 landing on Moon on July 20th 1969 will always retain the number one spot - in terms of impact value at a given time. I don’t know who is the close second but IBM Watson’s televised Jeopardy challenge when it bested Brad Rutter, the biggest all-time money winner on Jeopardy!, and Ken Jennings, the record holder for the longest championship streak, will always have a place in history. Organizing these kind of challenges is not new to IBM - who can forget when Deep Blue, a chess playing computer developed by IBM, beat world champion Gary Kasparov in a controversial match on 11th May 1997.

A lot of great things have been written about Watson, named after the IBM's first President but also influenced by the name of Sherlock Holmes' assistant Dr. Watson, since the televised challenge. Though, at the same time, you will also find many tweets ridiculing, hopefully in a good humour, that it needed lessons in geography after the (in)famous "Toronto" answer. But if you understand even little bit of information retrieval, natural language processing, machine learning, knoweldge representation etc.. then you would have realized what an amazing accomplishment this is. Recently, I had the opportunity to meet Aditya Kalyanpur, one of the team members of the DeepQA project which built Watson. Rome wasn't built in a day! Similarly it took more than four years and roughly twenty five brilliant technologists to build Watson. There are many unknown facts which we will come to know in due course of time. for e.g. Did you know that from September 2010 through December 2010 Watson played 55 games against Tournament of Champion Jeopardy! players and won 71% of the games. These players represent some of the best Jeopardy! players in the world.

Aditya managed to debunk few myths about Watson and also highlighted the approach taken by the team to develop the software. I would like to share them with you:

Three prominent myths:

• Watson answers a question by changing the query to structured query; then it queries a structured knowledge base. This is not true at all. This approach is taken by traditional QA systems and is very brittle with poor domain coverage.

• Watson identifies the question type and generates candidates from precompiled list of instances. This is false. Watson relies on a radically novel open domain type “coercion” technique.

• It uses either structured or unstructured data analytics. This is not true either. Watson integrates information from both unstructured and structured data analytics.

Some of the notable things about the DeepQA project:

• It does deep analysis of a question by breaking it down into relevant components like the focus, key entities and relationships, classifying it into broad classes, requirements of special handling etc..In one such strategy the system identifies independent facts within a given question, poses these as new questions to the underlying QA system, and generates answer candidates supported by the facts. Candidates that have support from multiple independent facts reinforce each other to boost system confidence in them

• Hypotheses, which in the QA case, are potential answers to the question, are generated by the Search component, which retrieves content relevant to a given question from the large volume of local knowledge resources Watson can access. The sources of information for Watson include encyclopedias, dictionaries, thesauri, newswire articles, and literary works. Watson also used databases, taxonomies, and ontologies, for example, DBPedia, YAGO, WordNet.

• Candidate Generation component identifies potential answers to the question from the retrieved content. A variety of answer scoring algorithms are then applied using DeepQA’s pervasive probabilistic framework. Other than linguistic processing, taxonomic, geospatial, temporal, popularity and source reliability are some of the evidence dimensions used by the scorers to constrain or support the right answer.

• UIMA-AS was used for orchestrating the overall processing and an in-memory implementation of Sesame was used for storing RDF data. Initially, it used to take 1-2 hours to answer a question but using a massively parallel architecture and exploiting more than 2800 P7 cores, QA time was reduced to a few seconds.

• Machine learning and Monte Carlo methods used by the strategy components to estimate and optimize the win probabilities for the various players in a particular game state

Elsevier Challenge: Another Big Step towards Open Data Movement

2011-04-07T13:31:00.001-04:00

Nobody was happy after hearing that Data.gov, along with a number of other data-related sites of the government such as USAspending.gov and Apps.gov, are slated to be shut down due to budget cuts. The current annual budget of $37 million will be reduced to $2 million. It wasn't long ago when I had written very enthusiastically about the open data movement. In general, despite the fate of data.gov, the open data initiative is still going strong. Today, we have almost 25 cities in US who have opendata. Sanfrancisco's datasf.org is another success story which has almost 60 applications built by the developers. What we really need, in this context, is more participation from the commercial world!

On a similar note - Today, I was contacted by Elsevier, one of the largest publisher of medical and scientific literature in the world about their open data initiative. I am more than happy to write about this great initiative from a well known commercial enterprise.They just announced its first worldwide challenge called “Apps for Science,” powered by ChallengePost. The challenge is designed to bring together developers and a community of 15M researchers to collaborate more efficiently via new and innovative apps.

This database comprises of more than 25% of world’s academic and scientific articles for the challenge. Now, developers can access Elsevier’s data catalog and APIs from its SciVerse Suite, a content discovery platform + developer network w/ 10M+ articles, an abstract database with 41,000,000 records and more. This video explains SciVerse Suite better:

Why should we care for it?

If you care about a cure for cancer, AIDS or any other deadly disease then it is an important step. We need to help scientists and support them with the best tools and information possible because better communication is equal to greater knowledge share which will result into innovation breakthroughs.
It will free up approximately 12 hours per week previously spent on collecting and organizing research (according to a 2007 Outsell survey of 6,300 knowledge workers).

What’s in it for developers?

You can build & host tools (free or fee-based) for a captive audience of 15M researchers & 10k research institutions on Elsevier’s Application Marketplace.

Judges include known names like Jeff Jonas and James Handler among others.

Hakia's SenseNews: Can it really tell you when to buy and sell stocks?

2011-01-23T09:40:00.013-05:00

There is a always a big difference between an exciting technology and an exciting business. And understanding those differences is the diffference between success and failure. Hakia always had an exciting technology if you ever cared to go to their web site and tried to evaluate their semantic search results. Unfortunately, they never had a succesful brand name like Google, Yahoo etc. which is probably true for most of the startups in the semantic search space. As a result, they could never attract too many visitors to their website. What got my attention is their recent announcement about Sensenews which is a service to make buy and sell recommendations for any stock. Sensenews reads news sources (over 30,000 news sources), blogs (over 1 million), Twitter and performs an advanced computation to make buy/sell recommendations to you.

Stock movement is always correlated with news - good or bad. An example in the fall of 2008 shows how unexpected news can impact prices dramatically. At 1:37 AM, September 7, 2008, Google's newsbots picked up 2002 story about United Airlines filing for bankruptcy. The story had a cascading affect and it was picked up by a person, who obviously failed to notice the date on the story, and put it on Bloomberg. The stock droped 76 percent in six minutes with a huge spike in volume.

The issue with news is that it is not mathematical in nature. It is very hard to make machines understand news. Various semantic technology, text extraction, natural langauage and statistical techniques are used to maker sense out of vast amount of news for a particular stock, currency or commodity. Hakia is not the first one who is trying this as:

Monitor 110, funded with 20 million dollars, was one of the first high profile failure in the capital markets and Hedge fund space. They had signed up almost hundred trial clients. They closed the shop in July 2008.
Relegance (a company out of Israel), one of the last acquistions (between 50 million to 100 million US dollars) of AOL, was also in this space. AOL is often accused of not leveraging Relegence’s technology effectively. But then AOL is accused of doing lot of things not in a right way in last few years. Relegance's technology is being used for AOL Money & Finance and WalletPop.
BNP Paribas developed their own indicator that reads the news looking for words that imply strength or weakness, and decides how to trade depending upon the number of times various words are used.
German exchange Deutsche Borse acquired Need to Know News, a Washington D.C.-based provider of machine-readable news for automated trading engines.
Reuter's NewsScope and Sentiment Analysis product is designed for use in algorithmic trading, where computer programs make all of the trading decisions including the entry and exit of any trades. More than half of the trading is now automated, much more for US stocks.
Stocktwits - The service takes financial related data from tweets and structures it by stock, user, reputation, etc.

This is a very hard and known problem which is becoming more painful because of the information overload. The whole idea is to:

Exploit market inefficiencies by showing the unrealized value of a stock
Manage event risk by reacting to breaking developments

Probably, not even 2% of the firms in the capital markets are doing this. There are hardly known successful cases - there is always a secrecy aspect also. But I don't know any vendor who has truly succeeded in making a successful business out of it. Semantic technologies are sometimes viewed as a technology which is looking for a business problem to solve but this case is different. At some level, the business problem is defined (yes, the requirements of every trading desk or traders in different firms will vary), but the technological limitations are exposed by its inability to semantically filter the right content, duplicate removals, understanding the right sentiment, categorizing it correctly, getting rid of marginal news, irrelevant stories and many other things. I also think that it is very hard to understand the social media manipulations as it is an opportunity for someone to make lots of money by fooling so called intelligent programs.

What is interesting in the case of Hakia is that they have gone a step ahead by building a recommendation model on top of semantic filtering of the content. This is really bold and interesting! Their delivery model is subscription/consumer based which is different from most of the ones who have tried this before or trying now as they were/are more enterprise focused with the exception of stocktwits. I do think that they should have offered thirty day free trial period for this service from marketing perspective. It is not clear what level of fundamental and technical analysis is done. I also don't understand the profile of the trader they are targetting. It also doesn't seem to be realtime in nature and won't work correctly for stock splits. But if they can deliver based on what they are claiming then it is commendable. Only time will tell whether they will be successful in this new business where no company has truly succeeded yet. But one thing is for sure that they have understood that future of semantic search - that it is very much related to being some kind of content clearing house i.e. clean, curate, collate and categorize content.

Predictive Analytics for social data: Is there a role for Semantics?

2010-11-11T15:00:00.000-05:00

Recently, Gartner, the leading technology analyst company, came out with its predictions for key technologies for 2011. The list comprises of cloud computing, mobile applications, social collaboration, next generation analytics, social analytics and many others - shouldn’t surprise you if work in information technology. The only thing which was not clear to me that why next generation analytics and social analytics are in two different categories when social collaboration is already emphasized as a part of a roadmap for large enterprises. I am sure they must be having their own valid reasons to do so. But the point is that overall it is getting a bit confusing about the various terms which are being used in context of analytics. What I mean here is what is the difference between just analytics versus predictive analytics versus forecasting versus predictive modeling versus optimization versus data mining versus advanced analytics. To many people, it sounds same! Also, it will depend upon who you ask this question. If you are a vendor or a consultant then try explaining it to a decision maker in an enterprise - basically, try not to get into that conversation. Many of these disciplines are more than a decade old as something like predictive modeling has been used in credit scoring for years. Also, academically, there is not much difference between classic techniques used in data mining and in statistics. Though, data mining has evolved to deal better with real life messy data. Unfortunately, unlike analytics, statistics could never become a hot topic but maybe it is about to change. In short, analytics or predictive analytics is the umbrella term or the new term - maybe the buzz word. It seems there is new surge of interest in predictive analytics because it is about the future outcomes in context of business intelligence. There is a difference between insights and gaining foresight!

You can always question that companies were always worried about the future outcomes so what has changed now if many of the methods were available before also. Probably, more data is available now, and there has been advancement and simplification of tools/techniques - you can hire a good business analyst to do the job instead of someone with a doctorate in statistics. Predictive analytics enables you to develop mathematical models to help you better understand the variables driving success. Predictive analytics relies on formulas that compare past successes and failures, and then uses those formulas to predict future outcomes. Also, if you consider the fact that IBM has spent almost eleven billion dollar in the last five years acquiring software companies, like SPSS and Unica, for its analytics consulting organization then it starts making more sense.

On top of that, social analytics is a new kid on the block and there is new buzz that it is going to play a significant role in predictive analytics. I do believe that it will become true gradually but it is not going to happen as quickly as we are being made to believe. What are the challenges in it? From the process perspective, predictive analytics is about understanding the prediction variables to the business problem, selecting the relevant statistical technique, validating the model with the test data and finally applying/adjusting the model iteratively with the production data. Do you think that it should be very different in context of social data? First of all, just because a company is doing brand monitoring (there are just too many companies), it doesn't make it a social analytics company. What I mean here is that if they are just following converations about an entity and don’t have much semantic intelligence in their software. If you look at most of the common examples of so called social media helping in predictions, they are about topics about election predictions (as recently claimed by Facebook political team) or about how a new product like iPad is perceived - the outcome is most of the time boolean i.e success or failure. In my opinion, these are interesting examples but much more is expected to do predictive analytics from social data. Maybe, if you consider social analytics and all associated prediction with it as a seperate or standalone discipline then it is good enough - but then you don't have an integrated view from an enterprise perspective. Maybe, in some cases, you don't need to integrate social data with enterprise data and still can get some value. But, you still need to build a repeatable predictive model using the social media data. And before you build the predictive model, you need to do true semantic analysis of the social data. Build some kind of normalized social data model to work with enterprise data for predictive analytics. IBM has come out with a new offerring where it claims to have enhanced SPSS software with social analytics and it can do predictive analytics for your business needs. It also offers semantic network analysis of the social data. I am not aware how well it works or if it can work across different business domains without too much business analysis or customization.

It is not an easy task to build a repeatable predictive analytics with a everchanging large volume of social data. The quality of meaningful data is also very important in this context. I see the ambiguity of social data as one of the biggest hurdle. You will have to do lots of preprocessing and deal with many new attributes being added in the social context. Every company is unique and we will see predictive analytics manifesting itself differently for each of them. Though, I do believe that we will see many companies building very high number of predictive models which will take social data into consideration for their business needs - maybe a predictive model per product and total turn around time of a week or less from problem definition to scoring. That brings up another question - even if real-time social data is present, can you take real time actions? If not then what is the true value of real time data in context of predictive analytics? You really need a very different level of infrastruture to take advantage of it.

This whole integration of social media analytics with predictive analytics should be owned by business - not by IT. Infact, in most of the cases, it should be owned by marketing departments because they value/understand social data more than any other department and they are also most qualified to define prediction goals from a social context, predictive behaviors, pragmatic tradeoffs and evaluating results.

It seems there are some good opportunities in this space. We still have a long way to go. A lot needs to be understood before making any claims in the industry. There are definetely some good examples in the research community like SOMA, a forecasting model developed by a researcher at the University of Maryland, Terror Organization Portal. It analyses a wide range of information about politics, business and society in Lebanon to predict, with surprising accuracy, rocket attacks by the country’s Hizbullah militia on Israel. By the middle of 2010 SOMA was sucking up data from more than 200 sources, many of them newspaper websites. There is another example of two researchers at HP Labs who have established that they can use tweets to predict how well a movie will do - the results turned out to be fairly accurate. What we don't understand that can these examples be generalized for the indusry adaption? It will be good to know more examples from the industry.

Google Instant: What does it mean to Semantic Search?

2010-09-16T16:38:00.000-04:00

After the recent introduction of Google Instant, the technology blogosphere has been lit up with all kinds of speculations about the impact of this feature. My first reaction was whats the big deal? It is just another feature and why everyone is talking about it! After testing it with many permutations and combinations of different queries, I have to admit that I am just impressed with sheer speed and elegance of this implementation. Performance is as important as relevance when it comes down to search! It works very well in most generic cases. Again, a lesson for many companies about how to leverage your assets (The asset in this case for Google is a database of user queries built over a decade) and market it effectively. From positioning standpoint, Google already owns the term "search" in consumers mind and it is going to consolidate that position more. It will take some time to get used to this feature and I have heard few people saying that they are already annoyed with it. If you don't want to use this then all you need to do is to go to your google search settings and turn it off. Overall, it is also a good marketing ploy from Google as it will eventually lead to more loyal consumers. Bing and Yahoo might claim that they have already done something similar - if it is true then they did a poor job in marketing it.

I do believe in following things:

I agree, with all other opinions, that SEO community is going to be impacted. Sites which are optimized for more generic key words will have clear benefits. And those who have been relying only on the long-tail of key words will have to rethink about their SEO strategy
Bigger brands will benefit more in the begining. In general, they are more optimized for generic key words. In any case, everyone should monitor their traffic aggressively post launch of Google Instant
I also think that this can be a challenging news for pure-play semantic search engines who are trying to compete with Google. They might argue that it is still a key word based implementation. I agree that Google or 'Google Instant' still can't understand similar concepts and differentiate between words which can have same meaning or are related heirarchically. Understanding query intent is big part of semantic search engines' strategy and consumers will start perceiving Google as someone who is also becoming very smart in this area. Consumers don't understand difference between search and 'semantic search' and it is waste of time if you are trying to convince them about the difference by explaining it to them. They just have to make sure users have a better search experience. Just think of other creative ways to win the mind share. Infact, they can think of implementing similar feature using some kind of semantic suggestion assuming they don't have a database like Google. Also, we need to accept the fact that people are much more forgiving to Google than to any other engine.They are willing to ask a query in 5 different ways to Google to get the relevant results but probably give one or two chances to any other search engine.
I am still unclear about the impact of Google Instant on SEO++. Basically, a data driven marketing approach! What I mean here that is there more incentive now for websites/brands, large or small, to adapt RDFa, microformats or GoodRelations ontology as companies like Yelp and Best Buy have done. Please read how Best Buy implemented Semantic technology to get more hits to understand it better. It needs to be tried out before anything concrete can be said as Google is still a blackbox in that aspect.

If you rely on SEO for your business in any way then you shouldn't leave any stones unturned. You should experiment with all aspects of SEO in this context including SEO++ to get that exra edge. Also, you don't know how your competitors will take advantage of this!

Sentiment Analysis: Can you get it right by just automating it?

2010-04-28T12:26:00.002-04:00

Sentiment analysis or opinion mining, which is relatively a new area of research is based on text extraction, natural language processing, semantic technology, statistical and various hybrid approaches, has become a topic of great interest for many companies. The purpose is to extract opinions and sentiments from text and determine whether a product or a service is viewed positively or negatively. Basically, this area has gained lot of prominence in last three to four years - thanks to all aspects of social media like Twitter, Facebook, blogs, newsfeeds and customer review sites as they have provided enough data to do the sentiment analysis. Has it really arrived for mainstream adaption? Can you really trust the automation of sentiments and get actionable insights? How many sectors and companies are really using it? And how? Is it mainly about the accuracy of the technology to get it right? What level of human analysis is required along with the automation? Is there money for vendors in this space? There are just too many questions out there if you ask most of the people outside the vendor and research community.

Now, we even have a symposium dedicated to it which was recently held at New York city with a good representation from vendors and researchers in this space. There are many companies who are in this space like Lexalytics, Brandwatch, Attensity, Scoutlabs, Istrategy labs, Sentiment 360, Saplo, Serendio and many others from text analytics and natural language processing space who are positioning themselves to take advantage of this promising market. Despite various claims, it is not an easy task to determine whose algorithm is better and how good is their machine learning and language processing modules. Right now, there are hardly any standards out there to compare them. I doubt that we will see any standards in near future also. Till this date, I have seen sentiment analysis being applied more in context of reputation management or understanding voice of customer like:

Which conversations are relevant and active
Which products are most talked about
Sentiments behind each product
Intentions to buy product
Top issues, cries for help
product suggestions
Early warning of issues

Even the most optimistic of researchers and vendors agree that it is not an exact science and according to them even seventy five percent of accuracy is hard to get - basically, law of diminishing returns sets in if you want to stretch beyond it. Linguistic nuances or ambiguity of the language, culture and getting the right context is often cited as the reason which leads to lack of accuracy. Overall, it is still worth it to do the sentiment analysis when you have to do deal with piles of content in this web 2.0 and beyond world. Other than issue of accuracy there are many more challenges like:

How do you do segmentation of the audience? I have a hard time understanding many sentiment analysis graph which doesn't tell me anything about the profile of people whose sentiment is being tracked.
You do get a broader idea about the polarity of a sentiment but it is very difficult to understand the degree of emotion
The industry is realizing that the sentiment analysis has to be used in conjunction with other research techniques though it is still not clear or defined about how it will be done in a repeatable manner?
How do you track and interpret sentiments about a product on global basis? It is just not about supporting international languages. There are various instances where a product has done well in US but has failed in a different country or viceversa.
Do you apply the same criteria of sentiment analysis whether it is a product, person, service, social or international issue?
Technical Integration with CRM or business intelligence will not be an issue. But what should be the criteria to assign weightage to a sentiment?
Sentiments can vary considerably if you measure them in different durations. In this article from BBC, it is much easier to measure the positive and negative sentiments associated with Gordon Brown in the UK elections debate during two hours of duration. It seems there were negative comments when he expressed his views about immigration. It just proves that there is a component of time dimension also in an opinion about a famous personality. Though, a sentiment about a product can't vary within hours. That too from the same person.

Mid-March brought a turn in public sentiment towards Obama from "majority approving" to "majority disapproving" in Gallup : http://www.gallup.com/poll/113980/Gallup-Daily-Obama-Job-Approval.aspx. Gallup tracks daily the percentage of Americans who approve or disapprove of the job Barack Obama is doing as president. Results are based on telephone interviews with approximately 1,500 national adults. And, if you compare these results with TipTop, a sentiment engine based on twitter : http://feeltiptop.com/obama.php, the results are not way off. It does surprise you but it tells me that sentiment analysis in case of public opinion about a burning social issue or a famous personality is relatively easier. It is also easier to get sentiment about a popular consumer product like iPad or iPhone. Probably, you get lots of data and strong opinions. On the other hand, if you read this article, it tells you that results produced by machine in a certain scenario were crap after the humans analyzed it.So there are definetely sceptics out there who will question the value of this technology.

Overall, the risk is that this discipline shouldn't be oversold and needs to go beyond technologists. It is still at an early stage but it is going to stay because the business case is solid. It needs to be positioned as an aid to human analysis rather than a stand alone discipline. Someone rightly said about sentiment analysis - "I think of these tools like a metal detector, sure it beeps when it finds something, but it’s still up to you to expend the energy to dig in up to your elbows and find the nugget." For it to become mainstream, marketers, product management and analysts needs to work in tandem with technologists and develop methods to measure its effectiveness in the long run. For mainstream adaption, more customized use cases have to be thought of and this technology has to be applied in a very pragmatic and realistic way to succeed.

In my opinion, voice of a customer and reputation management is just too crowded and new areas like financial services seems promising. Though, there are already products like Reuters Newscope who is playing in the financial services market. Pharma is another area but I have heard that there are still open regulatory issues out there which needs to be sorted out. I often think that at some point in future, sentiment analysis can also be integrated with early warning systems or some flavor of predictive analytics.

The topic of sentiment analysis always reminds me of saying from Bertrand Russel, the founder of analytic philosphy, that "The fact that an opinion has been widely held is no evidence whatever that is not utterly absurd."

XBRL Implementation Status: Doing a Reality Check

2010-03-23T23:22:00.007-04:00

It was only few months back when I wrote the post - Semantic Technology, Financial Reporting and Toxic Assets. The post was more about how XBRL and the Semantic technology can play a big role in addressing issues related with toxic assets in the world of financial derivatives. It is becoming more clear now that XBRL is almost a movement which is going to have deep impact on how information between businesses, regulators and investors across the Internet will be communicated in the next decade. XBRL is not only considered the most revolutionary change in financial reporting since the first general ledger but also one of the most successful semantic web format. So what is the status of XBRL implementations? You can check this interview with Eric E. Cohen, co-founder XBRL, about the recent updates. You can also find some good information at Charles Hoffman's blog or http://xbrlplanet.org/. I also liked the section on XBRL in David Siegel's Pull where he talks about the history of XBRL.

Some of the points which I found interesting are:

XBRL is almost ten years old but the real adaption has started in the last few years only
Regulators are one of the first group to adopt it. Regulators who are adopting XBRL are : Capital market regulators, tax offices, banking regulators, national statistic offices, corporate registrars
Corporate America has already been complying with a mandate from the Securities and Exchange Commission for nearly a year to “tag” financial data in XBRL
There have been issues in many projects but no significanlty failed XBRL project has been reported
June 15th, 500 public companies did XBRL filings. Another 50 have started doing this even though they didn't have to file. 2009 taxonomies came late so 2008 taxonomies were used
June 15th 2010, another 1500 will start filing in XBRL
June 15th , 2011, another 10,000 will do
One pending bill in Congress would direct all federal agencies to adopt XBRL for all requests for government bailout funds and all required reports on how those funds are used
Edgar Online gets $12 million from Bain Capital for XBRL efforts
Using XBRL, FDIC reduced the time to report information from 45 days to 2 days
European Parliament is the largest government body who has expressed interest in XBRL
The Securities and Exchange Commission launched its XBRL information portal, which can be found at http://xbrl.sec.gov/
The investment in person hours that it took to create either the IFRS or the US GAAP taxonomies dwarfed the total hours needed to create XBRL itself
Over 90% of Spanish banks now report in XBRL
Holland and Newzealand are already accepting tax returns and other government required documents in XBRL
Nevada is one of the first US state who is trying to use XBRL in many of its operations
XBRL taxonomies may not be interoperable. For e.g, the US GAAP and the IFRS taxonomies are all used for financial reporting but are significantly different
MIX (Microfinance Information Exchange) which collects information from more than thousand microfinance institutions is using XBRL. It is the first non-profit to use XBRL
Data aggregators and distributors have not embraced XBRL in any significant way
Taxonomy extensions is one of the hardest issue to solve as it reduces standardization and make interoperability very difficult
You can find some common errors in XBRL implementations in this excellent article
It is observed that few technologists have accounting domain knowledge and few accounting experts have technology domain knowledge in XBRL projects
XBRL International, Inc. (XII), has released “XBRL: Towards a Diverse Ecosystem," a discussion document seeking public comment on the future business requirements and technical roadmap for the XBRL business information standard. The document may be downloaded here.

I am sure that there will be many more highlights but it gives good idea that things are looking good for XBRL despite some initial challenges. It is still in early stages of adoption on Wall Street. But the progress is being made and it looks like that the financial reporting and analysis landscape is going to be a very different one in the coming years!

Semantic Search momentum continues: Netbase gets 9 million dollar funding

2010-03-02T12:49:00.000-05:00

It was only six months back when Netbase, a semantic search company focused on Health and Consumer segments, went through lot of negative publicity after Techcrunch published this story titled "Netbase thinks you can get rid of Jews with Alcohol and Salt." NetBase Solutions’ HealthBase, a semantic search engine that aggregates medical content from millions of authoritative health sites including WebMD, Wikipedia, and PubMed was the center of this story when many users found some glaring glitches. It wasn't over for Netbase despite heavy criticism as it is evident from this news which say that it has raised $9 million in venture capital funding, completing its Series C round with Altos Ventures and Thomvest Ventures Inc --obviously, the VCs are seeing something which the media couldn't. In my opinion, these technologies are still a long way from being perfect and we will continue to see 20% to 40% errors - in many cases, it is dependent on the domain/content and the technology needs to be tuned for that.

The marketing message of Netbase is not very different from what you hear from many companies in the semantic search, natural language or text analytics space but they do have global brands like PandG and Elsevier as customers. The company also claims that by April about sixty percent of nurses around the world will be looking up information in its resources with the help of NetBase’s HealthBase Insight Discovery tool.

Overall, it is good news for semantic search companies who are trying to raise money as VCs believe that they have a good future and demand will only grow. I am sure they are aware of imperfections but can still see the value it can deliver to solve certain problems in a particular domain.

Bill and Melinda Gates Foundation: Philanthropy goes Semantic

2010-02-26T16:32:00.005-05:00

Bill and Melinda Gates Foundation is undoubtedly one of the most transparent and largest privately owned philanthropic organization in the world. The foundation's work ranges from providing vaccines to prevent childhood diseases to cutting edge research to improve agricultural yields and prevent malaria, among other things. Though, philanthropy is philanthropy and there is no big and small in that world as anybody who is doing it genuinely is a unique entity - you only need to travel outside US, Western Europe and few other developed nations to understand how desperately it is needed in rest of the world. Still, I believe that this particular foundation is very visionary in all its pursuits and focus areas if you follow them closely. Today, I was pleasantly surprised when I was contacted by Fenton communications who handles communications for them to write about this interesting press release on my blog. This is the bare minimum I can do as I understand the importance of what they are trying to accomplish - I have also been fortunate to work with few philanthropic organizations and I understand how valuable this concept can be for them.

The Bill and Melinda Gates Foundation is funding a new digital-media hub call ViewChange.org. The hub will use semantic technology to create a platform that combines the video sharing power of YouTube with the open information of Wikipedia and the mission of your favorite advocacy organization.

ViewChange.org is being created by a social change organization, making it one of the first time a non-profit is on the leading edge when it comes to technological innovation. They’re partnering with Zemanta, FreeBase and OpenCalais - the three well known companies in the semantics world.

Actor Danny Glover announced the launch of the project today via email and in a video.

ViewChange.org is using the power of semantic technology to make videos, articles, blogs, and actions readily available to people working in global development. While watching high-impact video stories on the site, viewers can choose to dig deeper by exploring up-to-date details on which organizations are involved, links to related content, and lists of relevant actions they can take.

For example, imagine you are watching a short documentary about clean water issues in India. As the video plays, adjacent windows will dynamically generate links to actions and media directly related to each scene. These could include organizations involved in clean water and sanitation, action campaigns related to water issues, relevant videos from YouTube, articles from research organizations, and the latest updates from news services and blogs.

This is a great cause and it will be immense help to anybody who is doing anything noble for any society. Please spread the word in any way you can.

Where does my money go? A begining for an open government in US, UK and rest of the world

2010-02-25T23:47:00.006-05:00

Once James Madison, the political philospher and the fourth president of America, said "If men were angels then no government would be necessary." Though, there are no clear historical records available about the first official government, democracy or parliament, despite various claims by few old civilizations, but I guess that men figured out much before James Madison that they can never be angels and they will always need a goverment to live happily and prosper. I am sure people just didn't want any government but also hoped and craved for a smarter, open and transparent government. But it seems nobody could define it clearly in last so many generations what openness and transparency really means for a government. The good news is that all of it is changing! Surprisingly, for the first time "data" is taking the lead in defining an open government - maybe because it is measurable and never lies.

The two big initiatives, data.gov and data.gov.uk (still in Beta), were launched by US and UK government respectively in May 2009 and January 2010. Infact, Prime minister Gordon Brown of UK asked Berners-Lee to look at access to government data in June, after Barack Obama's administration launched an open source data site. Well, UK already ranks number three in the OECD (Organisation for Economic Cooperation and Development) study behind Austria and Portugal in the sophistication of its e-services so making it data available was the next logical step. At high level, the goals of both the initiatives are same as they want to make the government data available online to general public for improved access; creative use of that data outside the walls of the government; public participation, collaboration and feedback; identify unexpected and insightful data relationships - insights that would normally take several decades and hundreds or thousands of brilliant socialist scientists, statisticians, psychologists, focus groups and public policy experts to simply suspect. Both these initiatives are great because of the intention behind them but lets take a closer look at the state of the initiatives.

The data.gov started with forty seven data sets but already has thousands of datasets from eighty-one US agencies. It links directly to data files in various formats including CSV, XML, Excel, and KML. A lot seems to be lacking though:

It makes little effort to highlight or promote any projects that uses the data from the site
The focus is more on a repository
What you do with the data is not very clear
The website needs lot of work in terms of clarity and user experience
It is still not developer friendly and needs to develop an ecosystem
There's no basic demographic data like population from the Census Bureau
Browse and search functionality seems to be missing

The Sunlight labs, a DC based non-profit organization, is working on some projects to take advantage of this data but it seems you need to have larger developer community doing the same. I have also seen some good applications from the team of James Hendler of Rensselaer Polytechnic Institute, USA. His team is converting the data sets into RDF and taking advantage of semantic technolgies to build few applications. For e.g:

One of the application is about the amount of money received by the government for corporate and personal income taxes projecte through 2014 - click on the link to access it.
If you want to know how knowledgeable is your state - click on the link to access it.

I am still not sure why Linked data (Semantic Technology) approach was not taken from the begining. Overall, data.gov in US has still a long way to go before its goals are met. Ideally, it will be great to see more impressive applications which uses data from different sources and gives you an insight about a specific problem. Nevertheless, it is moving forward - it is also understandable that managing and simplifying the process of publishing humungous data from so many agencies is a herculean task.

Now, when I look at data.gov.uk then I have to say that I am just simply impressed considering the progress they have made in six-seven months. Kudos to the team, along with Sir Tim Berners Lee, who has been working on it. They just used the semantic technology, basically linked data, approach from the begining. It also has a modern design with a very developer friendly approach. Combining data and creating mashups from different sources in this context is not an easy task but semantic technologies have made it possible. Overall, data.gov.uk's approach is simple and clear - they have used open standards, open source and open data. The website has quality and elegance written all over it even though it is in beta. They also need to figure out many things like modelling various datasets behind the scenes, encourage more participation and many other things you can think of in a project of this complexity. But the results are showing! The top ten application as rated in this telegraph article are impressive. You can see the screen shot of one of the application called "Where does my money go." Or click here to access the prototype.

If you are really keen to go deeper in the approach of UK govenment in this implementation then I will encourage you to read this document - Putting the frontline first:Smarter Government. UK might have followed US, Australia, New Zealand in implementing open and smarter government but it seems at least now that their template will be followed by rest of the world. It will be interesting to see if/when countries like India, China and Russia will follow this trend!

"This is very much the beginning. Hopefully, this is the tip of the iceberg. There is a whole lot more to do." But what a beginning! These were the words of Sir Tim Berners Lee when he launched the beta site for data.gov.uk.

After Best Buy, now Tesco adopts Semantic Technology!

2010-02-22T14:19:00.003-05:00

It wasn't longtime back when I wrote how Best Buy is using Semantic technology to define a new trend. Now, it is Tesco , largest British retailer, who is following similar steps. Simply put, Tesco is to UK the same way Walmart is to US. Though, both of them get into each other's territory as it was almost an year back when Tesco launched Fresh and Easy,a chain of 10,000 square foot convenience stores in US to compete with Walmart.

Tesco has also been known as a company with a unique ability to manage vast reams of data and translate it into sales. It also uses information gathered from Dunnhumby, a British data mining firm of which it has majority control, to manage every aspect of its business, from creating new shop formats to arranging store layouts to developing private-label products and targeted sales promotions.

Tesco has always known to be pioneer in e-commerce and it is one of the most visited online supermarket sites in UK. They have started experimenting with RDFa in their website which seems like a logical next step. I won't be surprised if they adapt Goodrelations ontology also in near future.

Cognition Technologies to power Microsoft's Bing now!

2010-02-18T13:29:00.003-05:00

According to this news, Bing will also be powered by Cognition Technologies as Microsoft has licensed its technology for its search application. It is an interesting news because Congnition Technologies was always compared with Powerset who Microsoft acquired for $100 million plus in June of 2008. It is not replacing Powerset technology but adding more power to its capabilities. What is the relevance behind this non-exclusive licensing?

Though, Bing has made lot of improvements in the last year, to become more relevant in the search space and grab more share of the search market, it still needed to add more semantic capabilities in its arsenal. Bing is perceived to be good in following things:

Clean interface and a good user experience
Excellent information aggregation capability
Extracting concepts and summarization
Very useful for advanced search on Wikipedia and Freebase

Precision and Recall (the two criteria to measure the accuracy/relevancy of search results) is always a debatable topic if you ask any search company. Eveybody claims different results! As far as capability to undersatnd semantics/meaning of query is concerned, that is also debatable as you get mixed results if you test different queries with different search engines. Different techniques are used to really understand the meaning of a query by all semantic engines. But in case of Google, it gets its context from the humungous amount of data it indexes - they are processing so much data that they have lot of context around things like acronyms etc.. Suddenly, the search engine seems smart, like it achieved that semantic understanding, but it hasn't really. There have been instances where I have seen google nailing it quite accurately! So what is left to measure the effectiveness of search engine? Probably, this is the reason, the site traffic ends up becoming the only crietria for effectiveness or success. But, it takes years to build traffic and every improvement in the search engine counts.

I believe that Congition's real value propostion in this licensing is its advanced semantic map (think of it as a combination of dictionary, thesaurus, ontology etc.) which has millions of semantic connections that are comprised of semantic contexts, meaning representations, taxonomy and word meaning distinctions. Bing should benefit from this in making its query parsing more powerful and in improving relevance!

HP lists Semantic Technology among the top 10 BI trends for 2010!

2010-02-16T20:56:00.000-05:00

It is always good to know that HP has listed Semantic Technology among the top 10 BI trends for 2010 in this new white paper. Ideally, the technology enthusiasts would have liked more details about the semantic technolgy as a trend in this white paper but please keep in mind that this is a business white paper. Meant for executives who will probably scan through it! It is great to have it atleast mentioned there! The key to acceptance of any technology by business is always simplified marketing message and validation by a brand like HP.

It is also good to have it categorized under BI umbrella by a global leader! HP has become a serious player in the IT services sector after its $13.2 billion acquisition of EDS. So far, the acquisition has been working out very well as it grew its 4th quarter revenue of 2009 by 8% to $8.9 billion. Even though HP services only represents 30% of total company revenues, services created 40% of the total operating profit dollars. When technology markets mature, the revenue and margins start becoming more service-centric. Software is another high-margin business for HP. Semantic Technology market is not matured so it will have both product and services opportunity in the long run. Hopefully, HP will proactively help in its widespread acceptance. Maybe, there will be more opportunity for tools and products vendors in semantic technology space who can develop alliances with HP.

Semantic Technology is not new to HP - the most significant thing that HP has done is that it provided Jena. It’s an open source framework for building semantic Web applications. It incorporates RDF and owl APIs; it also includes a rules-based inference engine, it includes in-memory and persistent storage for the data. It includes the SPARQL query engine, and it’s by far the most popularly used framework for developing applications.

In the end, it is always difficult to say whether semantic technology will become a trend in 2010 or later because most of the companies buy/adapt products and services based on their business cycles, not on the vendors' products and services roadmap.

New Patent from IBM for Semantic Web!

2010-02-09T13:25:00.001-05:00

IBM has always been a leader in patent filing. Do you know that just in 2008, it filed more than 4000 patents which is three times more than its nearest rival HP. Recently, IBM filed a patent to improve traditional tag clouds by using semantic technology. Basically, the idea is that since tags are single words and users can't have description and context with a tag, it is very limiting and value of tag diminishes as the tag space grows. For e.g. picture tagged as "dog" will not show up when the user searches for content associated with tag "puppy." You can think of hundreds of similar examples. So, with the help of ontologies and associations, you can have more meaningful, descriptive and understandable tags. For example, once this tag cloud is represented in an ontology form then a "German Shepherd" can be classified as a type of dog with attributes like eye color, fur color etc. and relationships like "owned by". You can also specify that Puppy is a yound dog in this context.

The method, explained in the attached filing, comprises of receiving a tag cloud which includes tags that hyperlink to web content. It will seperate the tag into different linguistics categories, assigning a weight to each tag, and grouping the tags into clusters, whereas tags in a cluster are associated with a context. The server will have components like linguistic analyzer, semantic domain analyzer, taxonomy builder, attribute analyzer, relationship analyzer and ontology generator. Like any onology generator, the process will be iterative in nature. As a result of this, it will eventually lead to more accurate searches for the content you are looking for.

All of it makes sense to me but I wonder if there are risks associated with companies in future, who might try to accomplish similar goals using different flavors of technology, without infringing on this patent. I will let patent lawyers figure this out in future.

Your new Virtual Assistant on iphone: Can it deliver?

2010-02-06T13:16:00.004-05:00

It always surprises me how little is known about Siri, the virtual assistant, outside the Semantic Technology circle which is still very small community. Intuitively, everyone understands what a virtual assistant can do or deliver but most of the people still think that these technologies are not really ready for serious use. For those of you who don't know, Siri was born out of SRI's CALO Project, the largest Artificial Intelligence project in U.S. history. (CALO stands for Cognitive Assistant that Learns and Organizes). Made possible by a $150 million DARPA (Defense Advanced Research Projects Agency) investment, the CALO Project included 25 research organizations and institutions and spanned 5 years. Siri is bringing the benefits of this technology to the public in the first mainstream consumer application of a virtual personal assistant. The Siri application is just released for iphones and you can download it for free.

Siri - The Personal Assistant in your Phone from Tom Gruber on Vimeo.

It uses Semantic Technology for intelligent mash-ups that automatically make connections, takes action and communicates information based on dimensions such as personal data, theme or task awareness, time and location awareness much the same way a real live virtual assistant working the Internet could. It is almost like a very personalized semantic search focused on concierge-oriented tasks but can also take voice as input.

I downloaded the application on my iphone and it did manage to surprise me pleasantly. This technology has come a long way! I did face few issues as sometimes it struggled to understand what I am trying to say. Actually, it uses Nuance technology for voice recognition.I don't know how it will work outside United States - basically, lets say if you are in India, China or Russia then can this virtual assistant deliver the same help as it can do in US?

All of us need virtual assistant at some point in our professional and personal life - more when we think that time spent on some tasks is not worth our time. Ofcourse, we won't mind as long as this virtual assistant is working free for us and giving us some extra input. Will we trust it enough to do high-value transactions just based on its advice? Maybe buying a movie ticket is not a big risk and the prices are almost standard. But beyond that? It really comes down to confidence we need to develop gradually in our virtual assistant so that we can start delegating more and more tasks to Siri or other similar virtual assistants in future. Nothing different from level of complexity of tasks which we assign to out real life assistant - the trust needs to be earned everyday.

Release of Siri on iphone is still a very important milestone in a new era of AI and semantic technology-based applications.Can Siri build a formidable user base and become synonymous with the consumer Internet? It still has a long way to go and company will have to learn many things from its user base. Can it make enough money by just charging its affiliate network and giving the application to consumers for free? Time will tell this. I still think that the company can eventually make more money by making it more focused for business-oriented professionals. It just needs to think more about applying its technology to solve unique use cases. In the business scenario, it will still have to compete with other flavors of virtual assistants likeTimesvr, which is pronounced Time Saver. It is virtual assistant of different kind - an offshore-based online service which provides a task based (as opposed to assistant based) solution, where each task goes into a queue and is handled by whatever assistant (a real person) is available and qualified. It works surprisingly nicely and economically for few. Well, you can argue, it is not really fair to compare a top-notch AI/Semantic Tech. startup with an offshore-based service whose business model is basically labor arbitrage. But this is the reality of this flat world!You have to compete at every level and in all the fronts to make business sense!

Measuring Semantics!

2010-02-03T14:03:00.005-05:00

A lot has been written and discussed about the power and benefits of semantic technology. But still when it comes down to quantifying the benefits of semantic technology, you will find very few case studies where it is measured or some kind of ROI analysis has been done. It is always good to know about this case study by Telefonica which is one of the largest fixed-line and mobile telecommunications companies in the world: third largest in terms of number of customers only behind China Mobile and Vodafone, and in the top five in market value. They used semantic technology for their tariff calculation which is a very complex acitivity for an operator of their size. They could achieve 80% reduction in working hours and 75% reduction in errors for activities related with tariff calculation. If Semantic Technology has to become mainstream in the enterprises then we will continue to need more case studies like this.

Semantic technologies for the enterprise. Use case

Semantic Search: Finding Stuff and Creating more Businesses in this Flat World!

2010-01-25T14:34:00.009-05:00

If you want more good jobs then spawn more Steve Jobs" says Thomas Friedman, the author of "The World is Flat " in this new article in NY times. Not everyone is a genius like Steve Jobs, can put 10,000 hours or is a part of 1955 club which were the key characterstics of these successful enterpreneurs as pointed by Malcom Galdwell in his interesting book "Outliers." Not every company becomes Apple also. Well, what US really needs is more of small businesses than ever which are one of the driving force behind its economy. There are some real facts about the small businesses in US:

Employ just over half of the country’s private sector workforce
Hire 40 percent of high tech workers, such as scientists, engineers and computer workers
Include 52 percent home-based businesses and two percent franchises
Represent 97.3 percent of all the exporters of goods
Represent 99.7 percent of all employer firms
Generate a majority of the innovations that come from United States companies

We need more enterpeneurs than ever who can identify opportunities worldwide and develop it into profitable ventures. Where do you start? How do you get the information? How do I know about the gaps in the markets for particular products and services? What should be the focus area? Has it been done before? Who can partner with me? And there are so many questions you can think of. I undertand that you don't start every business just by searching on the web as there are other important things like personal contacts, capital, your network, your own experience, trade associations etc. etc.. But the search on the web is increasingly becoming the major starting point for many of these activities. It is more relevant than ever to do the research because everything you want to do can be outsourced; can be imported; maybe already exists somewhere; or demand is going to go away and you are blissfully unaware. Not that it is easy to figure this out but atleast we should have more resources than just the big three search engines - Google, Yahoo and Bing.

Try to find or research that information on these engines and you will know it is so hard. All three of them have done good jobs in the last many years but they can't continue to be all things to all people in all the contexts. Too much of emphasis has been on ranking also. I personally like google but it definetely falls short as far as exploratory and interactivity aspect is concerned. Bing, which calls itself decision engine, has shown some very good improvements in last one year. Also, somehow the big three have ended up promoting a marketing view of search on the web and they thrive on the tension created between SEO consultants/advertisers and them. It seems that media and analyst community are also too concerned with glorifying every percentage gain by Bing over Yahoo as you can see in this news and may others. Maybe, "its just not about search, its about business" - probably, Michael Corleane (from movie Godfather) would have said if he worked for Google in this era.

In general, I have seen a very narrow view of the search on the web from a school of thought which believes that whatever could be done in search will be just confined to these three as far as web is concerned - they think that new entrants will make some noise initially and then go away quietly. I completey disagree. In my opinion, it is no different from the view in eighteenth century when there was a school of thought which believed that whatever human beings can think of or can invent has already been done - there is no scope of anything new. Sounds ridiculous if you evaluate the progress mankind has made since then!

A lot has been written about the benefits about the Semantic Search and how it is better than the key-word based search. I have also written in one of my previous article about "semantics" in semantic search. Recently, Seth Grimes also compiled a very good article about types of semantic search. So I am not going to talk about what semantic search is but more about the opportunities for the new breed of semantic search engines.

Occassionally, you do see articles like "semantic search engines which will change the world which lists new breed of semantic search engines - some call them google killers. I always wonder why these semantic search have not been able to make measurable impact yet. Some of these engines have very good technology also though the list doesn't include many others which are out there. The internal details of most of these engines are still proprietary and they combine a natural-language processing with various flavours of semantics. One more semantic engine which I like is TipTop which mines Twitter database and does sentiment analysis also. But there are more than six Twitter search-based products in the market as you can see in this review - product from Tiptop is not even mentioned here while the other products may not be doing semantic search. Now, even Bing has a product which searches Twitter. So what can be the next step for these new semantic search engines?

In my opinion, the big issue is the lack of focus for some of these "semantic search" startups and their obsession to boil the ocean. Many of them waste lot of time comparing themselves with google. Some of them are also trying to do very similar things. Market will continue to get crowded with semantic search engines in next few years but there is a risk that many companies with excellent technologies will get lost in the crowd. Very soon, every search engine will start calling itself a semantic search engine, the same way every SAAS offering is a Cloud offering nowadays. The other issue is lack of understanding from business and users about what "semanticity" in search engines really means. Ideally, Semantic search engines should have some aspects of natural language, contextual (focus on disambiguating queries), ontologies and reasoning. The hard part is always developing the understanding how much of work is required to customize the technology to incorporate all these aspects so that it is relevant for a particular domain.

There is a great opportunity for these new breed of semantic search engines to rethink about their strategy. They should also not try to be all things to all people. They really need to carve out a space for themselves in specific segments. If they go after enterprises, they will face stiff competion from the big three in the enterprise - Sharepoint/Fast, Autonomy and Endeca who have customers in hundreds and have evolved over the years. Among them, Autonomy has done a great job in e-discovery space by following a vertical strategy. Even companies like Marklogic with its powerful XML server can solve many search related problems for unstructured content. In my opinion, semantic search startups can always continue to tweek their algorithms and enhance semantics but simultaneously the focus should be on verticalization, branding, strategy, positioning etc.. Application-centric or vertical strategy will be better for them as opposed to platform-centric strategy. They can also think about merging what is there on the web with the enterprise data/content to give extended BI inxights which is still a new area to develop powerful applications. Though I can count few small companies in this area also and even big ones like Business Objects after acquistion of Inxight. Still , there is ample oportunity to think creatively and develop useful analytics-based applications for enterprise.

Recently, Financial times launched Newssift (still in Beta) which is a business news semantic search engine which indexes thousands of news sources worldwide. They have used Endeca technology for faceted search and sentiment analysis is provided by Lexalytics. It can be a useful tool but more can be done in this area. If any of these new breed of semantic search engines can correlate data from historical sources and the one which is acquired from multiple sources to: identify patterns and indicate important events then it can be a killer application on Wall Street. Unstructured data is already being leveraged in electronic trading strategies but the adaption is not so fast. Generating alpha from the stream of unstructured data is not an easy task but a great opportunity.

Another set of innovative companies I want to mention in this context are Bintro ,Trialx and Echo Nest. Bintro matches you to what you are looking for like employment, partnerships, investment and joint ventures. TrialX is a free service that matches participants to relevant clinical trials based on their personal health information. TrialX uses a comprehensive database of 25,000+ clinical trials approved by the Food and Drug Administration (FDA) in the United States. Echo Nest helps you find your audience with targetted music production. It claims that it can understand every music writer on the web (bloggers, review sites etc.) and helps you find the writer most likely to review your music. Again, very useful way to leverage semantic technology to solve problems in a particular domain!

In the end, I believe that there is a scope for hundreds of similar applications for semantic search engines and they can happily coexist with Google, Yahoo and Bing. We will also see very interesting changes once the data web evolves and semantic markup starts becoming more prevalent.

Christmas Bomber: Connecting The Dots!

2010-01-12T18:31:00.005-05:00

It is now official that it was the fault of software which almost got 289 people killed in the bungled Christmas day bombing. Couple of facts have emerged:

The suspect, Umar Farouk Abdulmutallab, was added to a catch-all terrorism-related database when his father reported concerns about his son's radicalizations and associations. Though his name was not on flight watchlists
A misspelling of Mr. Abdulmutallab's name initially resulted in the State Department believing he did not have a valid U.S. visa. It seems that his visa could not be revoked earlier because of it
According to remarks by the President Obama:

The intelligence community did not agrresively follow up on and priortize stream of intelligence related to possible attack.
a failure to connect the dots of intelligence that existed across our intelligence community and which, together, could have revealed that Abdulmutallab was planning an attack.
In sum, the U.S. government had the information -- scattered throughout the system -- to potentially uncover this plot and disrupt the attack. Rather than a failure to collect or share intelligence, this was a failure to connect and understand the intelligence that we already had.

Image via Wikipedia

There will be many corrective steps which will be taken after this but the key point is that the suspect went through the same screening as other passengers and a a metal detector can't detect the kind of explosives that were sewn into his clothes. All of us know that billions have been spent on homeland security and the aviation security before this happened. And this has been one of the top priority since President Bush's days. It seems that it wasn't enough and we need to take a step back and reassess our strategy.

It is useless to play the blaming game at this stage as written in this Economist article but what it means that we have to rely more on the intelligence of the software before a terrorist, with all valid documents, tries to board the plane. Yes, there are ways to detect the device at the airport as explained in the Scientific American article - adding unpredictable or layered security screening in future but that will always be a very costly solution if we have to implement in all the airports in this world. I am still not clear about this report released by National Research Council in 2008 which says that data mining is not the most effective way to smoke out terrorists. Yes, there can be issues of false positives which really means that a non-match can be declared as a match but it is always not the case as evident in the Christmas bomber's case.

To me, what really stands out is how/why we fail to connect the dots." as the suspect's name was in an international database indicating "a significant terrorist connection". It is clear that there is a strong need for superior knowledge discovery, database integration, cross-database search and the ability to correalte biographic information with terrorism-related information. I can't imagine doing any of these things without taking semantic technology into consideration. Infact, it should be one of the biggest drivers for any new initiatives in this context!

You might have heard story of David Headley (whose earlier name was Daood Gilani) - he is named as the key architect behind the Mumbai/India attacks in Nomberber 2008 in which 173 people died and 308 were injured. The residents of Mumbai were not as lucky as the passengers on the flight with Umar Farouk Abdulmutallab. David Headley is an American with a Pakistani father and also served as an agent for the Drug Enforcement Agency after being caught twice doing drug dealings. He was an operative of the Pakistan-based terror group Lashkar-e-Taiba. After his arrest by U.S. authorities, Indian officials discovered that he was given a long-term business visa for India. It is also alleged that he was already on a watch list which Indian authorties were not aware of. Indian authorities also say Headley traveled seamlessly between borders and stayed in various hotels in the same city while scouting for targets. What is more shocking is that he came back to India after the Mumbai attacks! This is another case of big failure to connect the dots! There can be many of these in the future also!

Semantic technologies can really help in connecting these dots because OWL/RDF can help build views in a more natural data graph format that is highly expressive and strongly deterministic. It is also more applicable in scenarios like this which places more premium on adaptiveness, agility, flexibility and grounded unambiguous level of truth. It is very useful when you really care to see end-to-end picture of how things are logicaly connected. The consistency can still be maintained while changing and asserting new facts! Inferencing is also a powerful capability which can unearth many new facts.

It is understandable that there are very complex protocols and policies involved in sharing of databases between various agencies around the world but not having the right technology shouldn't be an excuse. Because Semantic technology can be a very good solution to this problem.

I would really like to know your opinion about this. If you have new ideas/thoughts or you are aware of existing work being done in this area then please comment or write directly to me.

"Pull" by David Siegel: Book Review

2010-01-07T00:43:00.015-05:00

I just finished the final pages of "Pull" while watching the People's choice awards on TV. Who could have thought few years back that there will be a category for most popular "Web Celebrity" award which will be won by Ashton Kutcher for his more than one million followers on Twitter. Such is the power of web technology! Are you curious about the power of Semantic Web technology? Read "Pull" by David Siegel.

First of all, I would like to acknowledge the courage of David Siegel to write a business book on a difficult topic like this. The book is more about the power of Semantic Web to transform your business and is meant for business managers and enterpreneurs. It is not easy to write a "business book" on a topic like Semantic Web which has more sceptics than believers. In general, I have found that most of the good business books are more about analyzing the past and there are very few which are visionary or predict the future. In this complex world, it is so hard to see the future even beyond five years from now! So don't expect perfection! David makes a very good attempt in this direction.

Other than explaining the benefits of pull vs the push approach as practiced in most of the businesses, you will find some very useful information and statistics. The book is more conceptual in nature and talks about the recent developments in the world of semantic web and also takes you to the decade between 2020 to 2030. Some of you might question his timing also, if you are in a mood to question everything, but that is not that important from my perspective. Eventually the market forces dictate everything so why should we worry about it? You can always argue that he has become over enthusiastic about certain topics and is almost Utopian in its approach at some places but keep in mind that many of the concepts in this book are about a distant future. You may wish for more details or hope that he should have covered more domains but then he had to draw a line somewhere to make it readable for everyone. Overall, he has done a good job in doing gap analysis between the present state and the future state across various verticals but don't expect that you are going to get the perfect technology and process roadmap to achieve future state - the book is not about making incremental improvements but to find a completely new learning curve.

In the end, this is a thought provoking book which you should read with an open mind. It will definetely make you think! Without giving away too much about this book, I just want you to know that I really enjoyed reading it. It is a passionate work by someone who has put two years of life writing about a subject he believes in. The level of effort he has put in research also shows. Apart from that, for $18.45 on Amazon.com, this book is value for money. A real bargain! I will recommend all of you to read it.

Mobile Internet report: some interesting insights

2009-12-23T00:47:00.001-05:00

In this age of information overload, it is very easy to miss this comprehensive report by Mary Meeker of Morgan Stanley. It is a solid and very detailed work! Basically, the theme of the report is about the growth of mobile which will be much bigger than any of us can imagine. This new technology cycle is compared with what Windows 3 did for the PC in 1990 and Netscape browser did for desktop internet in 1995. And how the mobile internet has potential to create/destroy more wealth than prior computing cycles!

The Mobile Internet Report Key Themes

The report is well supported by solid research and numbers to back the analysis and forecasts. It took me some time to go through it but it was worth it as it gave me some interesting insights. In general, if it comes to mobile, we don't need to read a report to predict the future - we just have to look around and see what/how everyone is using these smart devices. I will skip the obvious like the spark created by iphone, 100k+ apps etc. but there are still few interesting facts such as:

5 trends converging - 3G + social networking + video+ VOIP + impressive mobile devices
57 m iphone +163% growth, 125k developers worldwide, 2B+ downloads
Many consumers are finding that their online usage rises dramatically when they have 24*7 mobile access to cloud based stuff
Growth/monetization roadmap for mobile is provided by Japan
Physical products are gaining share in mobile ecommerce - 20% in Japan but less than 1% in US
China leads world in virtual good monetization
Google (Android) has the best chance to to serve as a more open counter-balance to apple. Opera leading transformation of mobile browsers
Professional content repository is still open - amazon, iphone, netflix, hulu?
Open mobile web potentially more attractive to developers - Apple may believe its platform management is prudent way to ensure high quality content but it also runs the risk of stifling development and innovation
Clouds will be providing real infrastructure for mobile applications
Location aware ads will be better targetted
People are more willing to pay for content on mobile than desktop
AT&T - 50* mobile traffic growth in last three years despite only 40% subscriber growth over the same period
Shift in type of applications from games, lifestyle, utilities, enterptainment etc. to business oriented though among the top 100 applications on iphone, business oriented applications are just five
Daily usage for productivity based applications is probably less than 5-6 %

I wish there was little more information about the user profile but I still consider it the best work by Mary Meeker after her famous Internet report in 1994. Since new business models are often created during technology changes so there is an opportunity for eveyone inlcuding Semantic Technology to take advantage of this booming market because:

Browser still remains the weakest link in these devices. There is no integrated/personalized experience.
The principles of end user interaction have not been established
Semantic search will become more relevant as users will have less tolerance for too many results.
Semantic and location specific advertising will have a role to play
Cloud computing will be one of the biggest enabler of writing semantic applications for mobile devices as computing power will no longer be an issue for even smaller shops
Semantic web powered commerce can also be triggered
Agent technology should find better business cases

Obviously, any analyst report will not be specific about the type of future applications/services which will reside in these mobile devices. That job is for innovators to figure out!

Online retail : How Best Buy is using Semantic Technology to define a new trend

2009-12-10T21:11:00.004-05:00

This downturn has changed the behavior of consumers considerably in the retail sector. Though, the retail industry, just in US, has been more than four trillion dollars in last few years but it has been affected by this economy in last two years. Despite of this trend, 80% of retailers feel that the online retail channel continues to be better suited to withstand an economic slowdown better than other channels. It is validated by recent observations in the retail sector:

Black Friday spending rose 0.5%, ($54 million), to $10.7 billion, this year from last year.
Online sales have been up 17% (Thurs. to Sun.) over the same period last year.
Cyber Monday has been up 11%, more than they did a year ago
Online retail has been growing more than 20-25% in countries like Germany, Italy, UK
Forrester Research projects that online retail sales will increase by 8 percent to $44.7 billion this holiday season

The channel is slowly maturing and with many of the easy wins now maximised, further progress will be much slower. In industries where consumer shifts as small as 1 percent can severely dent the profitability of brand, retailers now need to think more strategically about maximising revenue online. Textbook theory tells you that changes in the relationship between how much consumers are willing to pay, on the one hand, and their perception of the value they are receiving, on the other, underpins behavioral changes. There is also a strong trend where consumere are learning to live without expensive products and they are no longer willing to pay easily for premium brands. According to this article, the retailers need to be aware of following key trends:

As acquiring new customers becomes more of a challenge, retailers should switch more marketing budget to maintaining existing customers and driving repeat business
They must clearly communicate why customers should shop with them, and what extra benefits can be gained from doing so
Providing clear, accurate and detailed information on products, prices and additional charges is a key
Deep knowledge of your competitors’ online offerings coupled with sophisticated testing of different customer acquisition strategies will be crucial to stay ahead of the market

Till now, SEO, Search Engine Optimization, has been a successful strategy applied by online retailers to increase the traffic to their website but it has its own limitations - the links to the website doesn't communicate clearly why customers should shop with them and also doesn't provide clear information accurate and detailed information on products, prices and many other relevant things. GoodRelations ontology, which is just an year old, can fulfill those gaps and give retailers that extra advantage. It is a standardized vocabulary for product, price, and company data that can (1) be embedded into existing static and dynamic Web pages and that (2) can be processed by other computers. This increases the visibility of your products and services in the latest generation of search engines. I have explained some of the concepts of microformats and RDFa in one of my previous blogs but I will highly recommend you to visit website maintained by Martin Hepp, the creator of Good Relations Ontology. Martin has done a great job in coming up with a practical application of Semantic Technology which can deliver value. The best part is that adaption will eventually increase as the learning curve is so simple.

In his talk at the Search Engine Strategies 2009 conference in Chicago, Jay Myers, Lead Web Development Engineer for Best Buy, Co., Inc., reported very surprising effects of adding GoodRelations and RDFa to their products pages:

1. GoodRelations + RDFa improved the rank of the respective pages in Google tremendously. In fact, if you try the query "BestBuy Ferris Bueller" on Google, then the page comes on rank # 1 ahead of the much more established page . This indicates a strong effect of GoodRelations + RDFa on Google's appreciation of a page. It is particularly surprising since the age of a domain has now a huge influence on ranking in Google - older ones get a much higher ranking. In this case, the semantically augmented one is just eight weeks old but it is still ranked higher!

2. Jay also reported a 30 % percent (!) increase in traffic on the BestBuy stores pages, e.g. http://stores.bestbuy.com/1895

3. Yahoo observes a 15% increase in the Click-through-Rate (CTR). Nick Cox from Yahoo also recently reported that augmented search results, e.g. those with GoodRelations / RDFa in Yahoo get a 15 % higher Click-through-Rate (CTR).

So in short: While better visibility in traditional search engines is of course not the main intended effect by adding GoodRelations & RDFa rich mark-up, I think these findings are so substantial that any SEO / SEM consultant should apply it - now!

Kudos to Best Buy for showing leadership in early adaption of this technology! I don't know whether it is a tipping point or not but I do recognize that it is a very positive step in adaption of semantic technology by the retail industry! In the end, the most important thing is to give SEO experts and those who pay them an incentive to add rich meta-data now. I hope to see many other online retailers to join the bandwagon and take full advantage of this simple technology.

Microsoft Semantic Engine!

2009-12-08T11:29:00.003-05:00

One of the highlights of the PDC09, an event focused on the technical strategy of the Microsoft developer platform, was the overview and demonstration of the Microsoft Semantic Engine. The Semantic Engine unifies search, structured querying, and analytics over structured and unstructured data.You can read some more details about PDC at the CTO's blog. It goes beyond existing components like Lucene by supporting both text and non-text, such as audio, video, and images. The key points are:

Micorsoft has been working and investing heavily on this technology for the last two years
The "Microsoft Semantic Engine" name is just a place holder
It is not a W3C SemanticWeb(tm) approach but one which melds the unique capabilities of unsupervised machine learning (hierarchical clustering), information retrieval models (higher-dimensional vector spaces), pluggable and trainable classifiers (SVMs, Naive Bayesian, Maximum Entropy, Decision Tree, etc.), and personalized filtering and ranking.
One of the goals is to make search semantically enhanced. Clustering the results based on Semantics is a key differentiator.
You can expect to see the Microsoft Semantic Engine in one of the upcoming SQL Server Betas.

This demo/presentation will explain the approach in more detail. My initial thoughts are:

This is a good move by Microsoft and was long overdue because there is a solid business case for integrating this technology in the enterprise.
I view it more as a flavor of text extraction/analytics technology - infact, Msft has said it very clearly that it is not using the semantic web technology approach
Having Microsoft in the Semantic technology space is a good thing for the Semantic Technology indutsry. This is a great validation from one of the most successful leaders in the software space
They are not the only one who is trying this approach. As far as I understand, the goal behind Inxight's acquisition by Business Objects (now SAP) was the same one. I am not aware exactly how that integration has worked out
I am sure Micosroft will come out with a clear message regarding the "Semantic Engine's" positioning in the enterprise in comparison to Fast Search Engine (Now part of Sharepoint division).
It will intersting to understand if Powerset (now Bing) technology, Microsoft's semantic search, was used in this effort
More details are needed to understand how it will work with disparate data sources
Microsoft has been underestimated for too long as far as their search strategy is concerned. I agree that Google has a big lead in terms of number of users on the web but I really think that they have made all the right moves, both for web as well as enterprises,at least in last 2-3 years as far as search space is concerned. There are three great moves:

Acquistion of Fast search and integrating it with Sharepoint (MOSS)
Acquistion of Powerset and launch of Bing
Plan to introduce Semantic Engine and integrating it with SQL serve

Probably, services and product-based companies in the Semantic space, need to go back and revise their marketing message.
In the end, it just validates my initial thoughts that Semantic technology is a superset of Semantic Web Technology. Microsoft's approach or Semantic Web technology (W3C) approach are two different approaches to solve Semantic related problems in the enterprise. This is the reason, I named my blog as Semantic Technology blog.

Semantic Technology and iPhone!

2009-12-03T22:05:00.002-05:00

Recently, I came across this intersting article about "Inside the Apple Economy" - http://bit.ly/2FnBPt . Please read about some of the unknown names who have made a killing in last two years just by exploting this new ecosystem. I believe that Semantic web, linked data and Semantic technologies in general for the enteprise will also be creating an ecosystem which will be exploited by many people. But I only realized this Thanksgiving weekend during a NY city visit along with few friends of mine that there can be a corelation between iphone and Semantic Technology.

We had not made any dinner reservations as we were not sure about timings and were confident of finding an interesting cusine in the NY city, practically a paradise for your palate, which would work for everyone in the group. I was also counting on my Zagat membership which I have always found to be a very trusted source for restaurant reviews. We soon realized that everytime we agreed on a particular cuisine and restaurant after looking up ratings on Zagat on my iphone, the restuarant was always few miles away and it was very difficult to find two cabs on a crowded weekend. So we decided to use "Around Me" - an interesting application on iphone which can list critical services around you — banks, coffee shops, bars, gas stations, hospitals, movie theaters, restaurants and so on. Using geolocation, the app orders each service by its proximity to you — how many yards away — and, like other apps aimed at the traveler, maps out a route from here to there, if requested.

We kept looking up restaurants in "around me" but we couldn't find ratings and reviews for those restaurants. So I ended up toggling between these two applications, during our long walk, for a while till everybody agreed on a Turkish cusine.

I was wondering if these two services shared there data using Semantic Technology, basically Linked data, then life would have been so simple. All I had to do was to ask - "Give me all restaurants with minimum 4 star rating within 400 yards of my position." Hopefully, some food for thought for enterpreneurs who want to exploit the iphone economy. As somebody rightly said, "The food that enters the mind must be watched as closely as the food that enters the body."

Finally, a National Semantic Technology Roadmap!

2009-11-24T23:42:00.002-05:00

Probably, you might have assumed it that I am talking about the US but actually it is envisioned by the Malaysian government on their National Information technology Council (NITC) web site. More details would have helped but I will still give lot of credit to Malaysian government to start thinking in this direction and having the courage to recognize Semantic Technology on their official technology web site under the Ministry of Science, Technology and Innovation. Among other great things, the Malaysian governement also has the credit of creating one of the tallest towers in the world called Petrona Twin Towers. You have to be there to believe it and these towers have also become the epitome of astonishing growth in the last two decades in Malaysia.

While in US, sometimes, we still have to run sessions in conferences like - "Is Semantic Technology for Real? "

The Death of Taxonomies! Is there a role for Semantic Technologies?

2009-11-20T12:40:00.002-05:00

Image by justgrimes via Flickr

There is a very nice article by CMSwatch about how the practice of building taxonomies is dying - this discussion is more oriented towards content/document-centric world. It basically highlights few trends:

Since social computing is being widely adapted by companies so there is no fixed way of categorizing things i.e days of single, heirarchical taxonomies are far behind. People will tag content in any way they want.
You should be able to get to information the way you want, which may be different from your colleague's approach.
Text mining and auto-tagging software is gradually improving, and extracted terms can be applied as metadata. Metadata needs to be very fluid - cloud like. See the fig. above
Metadata architects should really understand the domain

I believe there are some solid arguments as I have seen this trend myself. I have only few things to add:

What will be the future of existing large taxonomies?
Many companies have already invested in more than one content management system like Sharepoint, Documentum or Opentext. There is a good opportunity for semantic web technologies, basically OWL and RDF, which can unify the taxonomies of different content management systems and provide a single data model to retrieve the content through.
We should always be very clear about what the word "Metadata" means in any context. It means different things to different people and has been overused. A metadata in a document-centric world is entirely different from say a metadata in a word processing program.
Semantic Web is really an umbrella language for metadata. More thought needs to be given to how this new trend of adding tags/metadata on the fly can be leveraged to add more value to the semantic web or linked data cloud.

Information is Beautiful! What about Semantics?

2009-11-16T19:24:00.005-05:00

I got this information from a site called Information-is-Beautiful and decided to share it with you. The site is really about ideas, issues, knowledge, data - visualized! This creative picture captures it all and should make everyone think twice when next time they think about designing "anything" which is interesting, functional, has a form and has plenty of integrity! And yes, what about "Semantics?" Probably, this creative author is also taking "Semantics for granted," at least in this picture, like tens of thousands of businesses worldwide!

Semantics, Metal Reserves and how long will they last!

2009-11-12T16:52:00.003-05:00

The picture below tells you about the state of planet earth's metal reserves!

Basically, this diagram says that there are fifty eight years left when all the metal resources will be exhausted if the world continues to consume at today's rate. There is nothing to worry about because I am sure that geologists on this planet will figure something out much before that. I am saying this because many of us worried about the state of oil reserves also in last two decades after many pessimistic predictions - more than half of the time people were "Crying Wolf." In any case, the point I want to talk more about is the role of semantic technology in helping the geologists around the world.

I came across an interesting paper in this context. The key points explaining why semantics is becoming important in geology are:

Geology, perhaps more than any other science, has advanced with the aid of pictures. The pictures, which are geological maps shows the distribution of different rock types. The current efforts to improve them have spawned a global e-science initiative to review the semantics of geology.

One example of issue of semantics in context of geology is - The US department of agriculture and Forest service and the British Columbia Ministry of Lands, Parks and Environment will describe and classify the same geological phenomenan in a completely different way

When working at relatively large scales, geologists are always dependent on data and information gathered and reported by other geologists regarding smaller scale features. Semantics is very critical here also

One of the consistent problems in geology is the omni-present "map boundary fault" - an apparent geological continuity along the border of adjoining maps which is nothing more than difference in nomenclature or semantics

Research is being done to prototype vocabularies by developing web services which can be used to store, compare, and rank on similarity, decsriptions of models (concepts) and instances (physical entitites and events) of mineral deposits, landslides and landslide hazards.

To go through the full research papaer please go here.

Semantic Advertising or not : Diamonds are forever!

2009-11-12T10:53:00.007-05:00

Most of us know the origins of the "De Beers", one of the largest diamond company out of South Africa, and how its founder got his start by renting water pumps to miners during the diamond rush. It is also believed that the founders merged their interests in various diamond mines into a single entity that would be powerful enough to control production and perpetuate the illusion of scarcity of diamonds. Rest is history! One thing at least I didn't know was how De Beers is used as an example, as shown below, of "Bad" ad placements. Basically, it highlights the limitation of existing ad technologies like contextual advertising and behavioral targetting in understanding the meaning of a web page.

Semantic Advertising or Semantic Targeting, which is more about targeting the relevant ad based on the meaning and sentiment of the page has gained lot of traction in last few years. I will recommend everyone to read this article by Scott Brinker which explains four different kinds of Semantic advertising in a very clear way.

In theory, the value of semantic advertising is great and companies like peer39, expert system and few others have gained lots of visibility. The opportunity is large as publishing industry is going through the biggest turmoil in decades because of this new generation who doesn't believe in paying for the content. So a relevant or highly targeted advertisement will end up being a major source of revenue for publishers.

In the end, semantic advertising is still a niche space and not a whole lot is understood by the industry about the effectiveness of this approach. There are not enough demos to show the value of semantic advertisement and publishers still need to be convinced more about the benefits of switching to semantic advertising. It will probably take some more time for semantic advertising to become completetely mainstream but I don't think that it will have any bearing on future of De Beers despite the "flaw" in ad placement - as there is a saying that "it is better to have a diamond with a flaw then to have a pebble without one." "Diamonds are forever" and they will always remain a scarce and sought after resource at least in next few decades.

Linked Open Data : Can we learn anything from failure of many B2B Exchanges?

2009-11-05T14:37:00.021-05:00

"You can't stop an idea whose time has come!" Linked data seems to be that idea in the grand vision of Semantic Web! The growth of Linked data cloud always reminds you that we are in exponential times! The Semantic technology community, including me, believes that Linked Data is the best thing happened to the Semantic Web vision. It makes sense and it is the next step for web and can also contribute significantly, if done right, to the evolution of this civilization. The possibilities are endless! We also have one of the best brains behind this initiative. Whenever I hear Metcalfe's law (by Bob Metcalfe) , in context of Linked data, which states that "the value of the network is proportional to the square of the connected users of the system" then it always reminds me of those early B2B days during late 90s when the world was about to change. Metcalfe law was very popular during those times also! B2B exchanges were considered the pillars of the new economy and their valuations made us lose our sleep. It seemed to all of us that everything was going to be re-defined and we wanted to be part of it. Well, it didn't happen exactly as we were led to believe! Despite the differences between B2B and Linkedata like one is trasactional and other is knowledge oriented, privately owned vs free, different technologies, different era etc., there are some similarities in both of them. Lets do some introspection so that we don't repeat some of those mistakes!

B2B (Business to Business) exchanges is an entity which brings multiple buyers and sellers to a marketplace where all kinds of commodity, financial instruments, intellectual property and various other goods can be eletronically traded - web is used as the medium in most of the cases. It has following characterstics:

The perceived value followed Metcalfe's law
Enterprenuers could fundamentally re-invent how work gets done
No longer comparison between big-small and so on
Shift of power from producers to consumers who are in control of everything
Standardized marketplace and standardized contracts
Markets operated at fraction of physical word cost
Global reach and one stop shopping
Neutrality, transparency, self-regulation, market efficiency, confidentiality and anonymity were other virtues of this marketplace
The winner, of a particluar vetical B2B , takes all

It did have some good principles but a very large number of the B2B exchanges failed within few years. There were few fundamental reasons for their failure:

The internet enterpreneurs didn't really understand their place in the overall marketplace
What they offered to business was something that already existed - at least in some form
Companies had always done business with other businesses and most of these businesses negotiate to lower the price. Adapting those existing processes to an internet format didn't really create anything new or different in the field. Unfortunately, many companies felt that switching to an internet-based sites controlled by third parties was a riskier bet than staying with the current partners and vendors

Few of these exchanges figured it out early and survived by either addressing these issues or by being creative about it - like some of them just focused on small businesses. ECN (Electronic Communication Network) are another success story in the world of B2B exchanges as they allowed a more efficient price discovery mechanism for stocks and currencies. In the end, B2B exchanges were all about attracting buyers and sellers or producers and consumers which they couldn't do. In a similar way, the success of Linked data cloud will depend on creating a marketplace which should be able to attract producers and consumers or buyers or sellers. The technical design prinicples by Sir Tim Berners Lee are just great and well thought of. We are also aware of the existing issues in the Linked data Cloud like quality of data, what is out there, disambiguations issues, trust of the source, frequency of the update, how to get started, should we always publish in RDF, how do you erase inaccuracy and many others things like this. We had very similar issues, maybe in a different flavor, when Web 1.0 was developing and we have come a long way. We might say that HTML was much easier to adapt then RDF but the success or wide adaption of the original web was not only simplicity of HTML and HTTP protocol but because it was a great sales, marketing and ecommerce tool. In context of Linked data cloud, lets accept the fact that most of the businesses will not be interested in "the future," they will always be interested in "their future." Their first responsibility will always be their share holders.

Companies need very strong incentives or business case to publish their data to this "Cloud" and write applications to consume the collective intelligence of the nodes in this cloud. Initiatives like exposing government data (UK and US at this point), dbpedia, scientific, map oriented data, music, individual research projects and so many other examples which can be good and interesting from exploration standpoint are already underway. We will also see more invovlement from non-profits and intelligence communities at some point. It is a great effort but still not enough! The Cloud will continue to grow and will add billions of more RDF triples but we need to proactively involve the corporate world. Technology enterpreneurs for Semantic Web or Linked data can't do it alone. They have to partner with the business community and the enterprise. The value proposition of Linked data needs to be articulated to the CXO community and their participation needs to be encouraged. We have to start having more conversations like the "benefits of publishing data to the cloud for the enterprise." Basically, much more on business development, business case analysis, education, marketing then technology alone - technology will always remain important and core. I had written an earlier blog about a new category name for Semantic technology which talks about some aspects. Once we have their attention and buy in then the ecosystem of new tools, applications, security infrastructure, new architecture, developers, APIs, SEO, creative sales and marketing ideas, consulting, outsourcing, selling and buying and many either good things will start maturing. It will be very important to do so in next few years to sustain this momentum of Linked data cloud. It will also attract significant investments in this effort! Don't we think that there are dollars to be spent on the Linked data cloud in the worldwide IT spending of more than three trillion dollars in 2009. We should be thinking or advocating about the "guidelines" for publishing to the Linked data Cloud like:

The benefits of Linked data and possible use cases.
How are they impacting their supply chain?
What are the advantages in reciprocity of links?
What should be the process of identifying the data which needs to be published
How will it help their revenue and help in their relationships with partners and customers?
What are the security issues? What kind of security tools or approaches which are out there?
How will it mitigate risks for the company?
What will be the possible compliance and legal issues for them?
What data access choices can they have - free access, partial access, sign up , qualification, payment?
What are the data packing options like - depth, breath, granularity, freshness etc..
Can they have handle on who is accessing this data, frequency of access, redistribution issues?
Can they create virtual networks of data clouds for their partners and business customers?
Can they promote their data?
Can they increase the findability of their data?
How can they do it in a phased manner?
If they use RDFa then are they part of Linked data?
Are they accountable for consistency, organization, correctness of their data?

I can think of so many more questions. A very clear policies and procedures aproach needs to be articulated which will lead to a governance model for publishing to Linked data cloud within an enterprise. Probably, we will have to learn more from the experience of present initiatives like publishing data from the UK government to the cloud. During the early days of web, if any company didn't have a web presence then it was perceived that they were missing the action, opportunity or future revenue. At some point, today's businesses should start looking at Linked data cloud with the same lens.

Book of Odds: A Semantic Knowledgebase of Probability

2009-10-28T22:46:00.009-04:00

When I think about quotes related to probability, I can't decide which one of these is more profound or I like more:
1) "Probability is the very guide of life" by Marcus Tullius Cicero
2) “Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.” by Sherlock Holmes
3) "Million to one odds happen eight times a day in New York" by Penn Jillette

I am always tempted to choose number three because it is more funny and if you have lived, worked or visited NewYork city then you can relate to it. But this blog is not about quotes related to probability! It is about Book of Odds: World's first reference on the odds of everyday life or a Knowledgebase of Probability built using Semantic Technologies. It is not another search engine or "yet another knowledgbase" as I explained in one of my earlier blogs. It tells you odds about things which you care about like "odds of surviving in a plane crash" or "odds of having a heart attack if you are of certain age and ethnicity". Odds, Probability and Chances are used interchangeably in this context -- it is basically the ratio of favourable outcomes to total outcomes. The world had references like google, wikipedia, Webster Dictionary, encyclopedias and so many other for almost everyhting but didn't have "Book Of Odds" to understand the probability of an event or an occurence - so the founders had a good business case to start this initiative.

It seems that the project started almost three years back and millions of data points were collected by the team of very talented researchers. Once data have been collected, analyzed, and the odds have been computed, every Odds Statement then undergoes a rigorous quality review, is modeled ontologically (or in simple words by using semantic technologies explained throughout this blog), and finds its place in underlying semantic database of probabilities. The Semantic middleware is provided by Cambridge Semantics, a Semantic Technology company, mentioned in one of my earlier blogs, which structures the data, adds metadata and other information that helps users find the relevant odds statements using simple search. I consider it a very creative use of applying semantic technologies to the problems which could not have been solved using traditional database management systems. This proves again that a typical DBMS is not the answer to all the storage and query related problems! The semantic technology also allows it to find interesting and unexpected connections. It is not only a fun site but aims to be a respected new reference source to increase the tolerance for uncertainity and help people to deal with tough decisions.

To start using Book of Odds, you can browse by topics like health and illness; accidents and death; relationship and society; and daily life and activities. It is understood that it will always remain work in progress as this planet is becoming more complex and eventful everyday. There will always be new areas to add and grow this reference.

My limited intearction with the website tells me that content is of high quality. I will recommend readers to at least give it a try and encourage this team. The most important thing for "Book of Odds Inc." is to have more people registered and start using this knowledgebase. I will "not" keep on pressurizing the founders at this stage by asking about business models and revenue stream.What business model did google have for many years? I am sure that they know that these questions are going to come up repeatedly after an year of their launch. If they will keep on producing high quality content or probabilistic ratios in this case and keep it meaningful for people then good things will happen. They do need to think creatively about advertising and marketing campaigns. They shouldn't be perceived as another google killer! Atleast they were smart enough to not use "engine" in their name. A business model will follow if they can attract more users and keep them interested. My only recommendation to the founders will be to start thinking about combining their work or knowledgebase with various forms of predictive modeling or business analytics in the business world. Who knows better than companies in this economy that "By looking at the past we can come to a greater understanding of the world and what is forever an uncertain future!"

How Semantic Technology and Text Extraction can help in listening to "Voice of Customer!"

2009-10-23T13:19:00.015-04:00

Businesses succeed by getting, keeping, and growing customers. Customers are the only reason you build factories, hire employees, schedule meetings, lay fibre-optic lines, stock inventory, write software or engage in any kind of business activity. "Without customers, you don't have a business. You have a hobby." These are not my words but they are explained very nicely by Don Peppers and Martha Rogers in their famous book - Return on Customer. The book is basically about developing a business metric around your customers that can drive better management and higher stock price.

We know that it is becoming increasingly hard to keep this customer as they are no longer hungry for what a business has to offer. They have more choices than ever as far as developed and emerging economies are concerned! At the same time, Social media tools have made it possible for hundreds of millions of people on this planet to express their opinion. And everybody has an opinion about something! If that "something" is not a person or social/political/economic issue than chances are that it is likely to be a product or a service! It is becoming very important for businesses to understand what their customers are thinking and saying about their products and services. They know that they will lose this customer as customers' shift loyalties very quickly if they are not heard. Businesses call it "Voice of Customer" and have been trying various ways in last few decades like polls, surveys, incentives and some other ways to understand that. There have been many issues with most of the earlier approaches as it is not an exact science and new regulation and privacy issues are making it harder. Marketing and support personnel have been trying hard to keep tabs on real-time conversations regarding their products, services and brands in blogs, Web forums, wikis, microblogging services including Twitter and Facebook. The advancement in text analytics is bringing a complete new approach to hear VOC.

In a nutshell, "text analytics" is about processing unstructured information in text and transforming it into relevant information. It is a highly sophisticated technology which uses concepts like clustering, categorization, advanced pattern recognition, machine learning, AI, language identification, semantics, linguistics and relationship extraction . The text processing can extract entities, concepts, events and sentiments from the text and marry it with structured data in databases and data warehouses to get useful business insights. Pharma and intelligence community have been the earliest adapters of this technology as they have compelling need and ROI is very high. In of my earlier blogs, I have also explained how Open Calais, using text analytics, is helping Semantic Technology.

Attensity Group, one of the remaining leaders, after Clearforest and Inxight were acquireed by Reuters and Business Objects respectively, text analytics space, has announced a very interesting product called Attensity Cloud. They built this product by combining their patented text analytic technology with Radian6 platform. Radian6 provides a platform to develop a dashboard, metrics, filtering and segmentation for social media data. According to Attensity Group, the Cloud plaform also uses sematic web technology along with text analytics technology. The Attentisty Cloud can tell things like:

Which conversations are relevant and active
Which products are most talked about e.g hamburger vs. salads in case of Mcdonald
Sentiments behind each product
Intentions to buy product
Top issues, cries for help
product suggestions
Eearly warning of issues
Actionable facts

Morever, you can integrate this platform with CRM and e-service to get value across all channels. I think that it is a very good move as it is a very smart idea to tie text analytics technology with semantic technology and applying it to solve a business problem. It can allow organizations to monitor, analyze and respond to social media in more effective and faster way.

Since, the platform is introduced very recently so there are no real customers. It will take a while to understand the power and usefulness of this plaform and how it is working out for Attensity's customers. Nonetheless, the concept is good because it will probably help in customer retention if businesses can integrate, configure and deploy Attensity Cloud successfully along with CRM systems. Because, In the end, any customer needs just few basic things:

They want you to know who they are
They want you to know what they need
They want a reliable way to communicate with you
They want you understand their definition of value
They want you to know what they want next
They want you to support them if they trust you and buy from you
And they don't want you to ask the same questions

I believe, despite doing all of these things, it is still a challenge to retain a customer if you don't innovate. Do you know the number of customers Apple stole from its competitors, who thought they were doing everything right to hear VOC, by introducing iphone? But thats a different discussion for a future blog!

Do we need to define a "Category" for Semantic Technology in the IT Stack?

2009-10-21T13:44:00.009-04:00

I started thinking about 'having a category for semantic technology' more after having a conversation with a friend of mine who is a Vice President of Product development for a software product company. His exact words were : "I believe in Semantic Technology and it is very promising. We will also see lots of innovation in next few years. But it is hard to sell product and services in this space" He really meant it more in context of an enterprise than the WWW. He is probably right somewhere despite his limited exposure to Semantic Technology world.

Even though there are many semantic technology-based implementations that are happening in the IT industry but it is still a tiny fraction of what is going on with Databases, ERP, CRM, BI, Portals, ECM, SAAS, SOA, Collaboration, Applications Servers, Security, legacy application modernization and so many others disciplines. Semantic Technology is not among top ten priority for any CIO for 2009 or even 2010! Actually, it is not even perceived as a category. When will it become a priority? Do we need to create a category for semantic technology for the enterprise? Maybe! Selling any product and services in this space can be hard sell if semantic technology industry doesn't do a good job in coming up with a nice and understandable category like ERP, BI, CRM etc.. which is already defined clearly in the minds of business and IT in any company. At least, they don't argue about what it really means. At the same time, we have seen the fate of "knowledge management" category which till this date means different things to different people. Selling it as "Semantic Technology" has the same risk as selling SOA (service oriented arhchitecture) which is always a hard sell by IT to business. It has to sound like a solution to the business who doesn't want to buy another piece of technology. The Semantic Technology community now has to wear their marketing hats.

I really admire at least four instances where defining a category played a huge role in success of one or more companies in that space and subsequently many innovative products and solutions were introduced in the marketplace.

SAP hijacking MRP and rebranding it as ERP (Enterprise Resource Planning). Later on other disciplines like sales and distribution, warehousing, financial etc. became a part of ERP.
Siebel, through acquistion, coalesces SFA (Sales Force automation), CCA (Contact Center Application) and marketing automation tools into CRM category.
BI was coined by Gartner and they re-named catgeories like DSS (Decision Support System) or EIS (Executive Information System)
The industry went through the confusion of defining a category for many years for web content mangement, imaging, document management, records management, digital asset management, collaboration etc.. Finally, everybody has settled on ECM (Enterprise Content Management) which is broad and relevant enough. The ECM industry has been on a great growth trajectory since then.

Also, these four categories are already multi-billion dollar industry now. Having a broader concept and having more players as part of it always helps when anyone is sizing up a category. In matter of months, from small player who is struggling to make few millions on its own, you suddenly look like a part of a growth industry which is already making hundreds of millions. It also makes a CIOs as well as VC community very comfortable.

Defining a right category which sticks and resonates with both business and IT is one of the most important steps for growth and to foster innovation. We really need better positioning for semantic technology! Having a right category doesn't guarantee success but it is the first step towards long-term success. It will definetely accelerate adaption! From timing perspective, the Semantic Technology Industry has to do it now. As Sir Tim Berners Lee says, "We have all the tools to make it happen" - he probably meant web in this context but we know that all the technologies are equally relevant and game changing in the enterprise context also.

Things can quickly change in theIT industry if you take the examples of SAAS (software as a service model). It was nowhere among the priorties for CIOs 3-4 years back and has already made it into the top 10 list for many of them. There are already more than six hundred vendors who are offering SAAS. Virtualization and Cloud computing also comes to mind in this context as they have also made it into top priorties in a very short span of time. These examples should be enough inspiration for companies who are planning to build products or offer services as Semantic Technology discipline is based on eually good foundation of solid research and development as any of these disciplines mentioned above. Again, right categorization is the key here.

We really need an "Umbrella Term" for Semantic Technology so that companies in this space like Franz (database), Metatomix (platform), TopQudrant(tool), offering ontology services and many others can live peacefully under it. It will also make life so easy for sales, marketing, channels and future enterpreneurs in the vendor community.

"Business Semantics Management" has been used earlier in a similar but smaller context. Though, personally I think that it has a very broad meaning and can be one of the candidates. It does resonate with business but this name can have its own limitations also. It can be perceived more as just in context of ontology and vocabulary whereas Semantic Tchnology can give you business insights also. It can multiply your ROI on your investments on BI, data management,integration, ECM, search etc.. There can be many meaningful and relevant names for this category which will require more dialogue in the industry. Once we agree on a category name then it needs to be marketed and socialized relentlessly.

We really need a body in the semantic technology like what AIIM is to ECM industry or TDWI is to Business Intelligence to help in this. W3C is doing great work but its focus is more web, research and standards. We need an organization who can work closely with corporatations, vendor communiy and maybe research to facilitate conversation and ideas.It will be lot of work for this organization if they have to follow the model of AIIM who has done great job in ECM market in education, industry advocacy, marketing, consulting, professional development, peer networking, benchmarking, market intelligence, developing local chapters, best practices, help in developing policies and procedures etc..

Is it GE or Google as proxy for US economy? The role of Search in being a leading indicator!

2009-10-16T14:08:00.010-04:00

We know that the best way to understand the state of US economy is by looking at GDP . It wasn't long ago, probably two to three years back, when GE was unanimously considered as proxy for US economy. It was always most diversified company as its businesses spanned from entertainment, medical devices, energy, locomotives, equipment, infrastructure services to finance. It always remained among the leaders in revenue and market capitalization. Unfortunately, it has fallen out of favor in last two years because of its average performance across the board in most of its businesses. It is no longer a leader in market cap. also. The result of all of this is that GE is not considered as a proxy for US economy or their sales pipeline as one of the leading indicator of economy.
I will leave it to economists to debate what is a true economic indicator but recently, Google has started claiming that its search patterns are saying that economy is recovering. Basically, Google's chief economist says that he can tell the direction of economy from American's search habits.

For e.g Google is seeing following trends:

Decrease in searches for unemployment benefits
Increase in searches for homes and real estate agents
They also showed an early spike in government "cash for clunkers" program.

There can be million other search trends which they might have analyzed or modeled! It seems lot of people are already listening! Well, the part of the reason can be that it wasn't long ago when Google's market cap. became more than GE!
You can always disagree with Google's claim and argue that a search for "property prices in a neighbourhood of Las Vegas" shouldn't be interpreted as recovery sign. Maybe, the person is trying to to sell the house! In the end, it is very difficult to interpet the true motive from few key words in the search. After all, economic forecast is a serious business and econometric modeling it is a very complex thing!

I think that we will continue to see this trend in future where "search on the web" will continue to indicate at least partially where the economy is heading. As search on the web moves from "keyword search" to more advanced "semantic search", we will have better idea about the intention of the query.

If we can analyze the "query intent" then we can build better models also. We also have to take into consideration the search patterns from Yahoo, Bing and other search engines to have a complete picture. I will still not dismiss off leading indicators from GE as it remains a great company despite a plunge in its stock value.

Semantic Technology, Financial Reporting and the Toxic Assets!

2009-10-15T12:02:00.003-04:00

Financial markets, traditionally the earliest adapter of any new technology relative to other industries, has been a laggard as far as Semantic Technology is concerned. It seems that the turmoil in the capital markets in last two years has managed to dampen enthusiasm for new technologies in capital markets and banking industry. All of it is about to change! The two obvious reasons are: we are coming out of recession and new regulations regarding financial reporting in XBRL. I believe that the third reason is the inherent limitation of XBRL as far as Semantics is concerned!

As we know, XBRL, solves two significant problems for companies who prepare financial statements along with analysts, investors, regulators, financial publishers and data aggregators:

The first problem is that preparing a financial statement for printing, for a Web site, and for filing today means that a company could typically enter information three times
The second problem is that today (if the report is not in XBRL), extracting specified detailed information from a financial statement - for e.g we still can't ask questions like "Give me depreciation expense from 2003 a financial report."

The basic idea behind XBRL is to provide grammer and syntax behind financial reporting so that it can be extracted, analyzed and queried. Although XBRL has been around for 10 years now, the adoption and acceptance has only begun to significantly accelerate during 2007 with the support of SEC. Since the year 2009, the filing has become mandatory for largest five hundred US corporations and other companies will follow in a phased manner from 2010 onwards. Market is already flooded with XBRL products , services and tools. Most of these products and tools help in one or more of following things : creation, viewing, analysis, taxonomy creation, custom document creation and various other automation features. XBRL can be stored in RDBMS as well as XML databases like Marklogic.

So what is the problem? Why do we need Semantic Technology in this context? While XBRL allows for more accurate consumption and interpretation of financial information, there is still a need to connect to the authentic source of the document and to recombine the XBRL content with other data sources. The fundamental issue here is that XBRL document working with other dat source doesn't understand anything about the semantics of data. There is just no meaning associated with the nesting of tags. The limitation of XBRL becomes more obvious when you have to use/analyze/query XBRL reports along with other sources of data which is not XBRL compliant.

If you read this article in Wall Street Jornal on Toxic assets then it will make you think more clearly about importance of "semantics" in reporting in the world of derivatives.. The key points are:

Ever since humans started trading, lending and investing beyond the confines of the family and the tribe, we have depended on legally authenticated written statements to get the facts about things of value
Derivatives are the root of the credit crunch. Why? Unlike all other property paper, derivatives are not required by law to be recorded, continually tracked and tied to the assets they represent. Nobody knows precisely how many there are, where they are, and who is finally accountable for them.
Every financial deal must be firmly tethered to the real performance of the asset from which it originated.
All documents and the assets and transactions they represent or are derived from must be recorded in publicly accessible registries
Governments can encourage assets to be leveraged, transformed, combined, recombined and repackaged into any number of tranches, provided the process intends to improve the value of the original asset
Financial institutions will have to serve society and fully report what they own and what they owe -- just like the rest of us -- so that we get the facts necessary to find our way out of the current maze
Governments can no longer tolerate the use of opaque and confusing language in drafting financial instruments. Clarity and precision are indispensable for the creation of credit and capital through paper.

XBRL, by itself can't fulfill all of these requirements as we need to corelate/link/resolve various reports to the source data - a very important thing in the world of derivatives reporting. You need Semantic Technology for that! We need to represent XBRL in RDF or OWL representation. I will recommend my readers to read another rebuilding public trust - a nice article on the same topic. The author is also talking about services which can allow the financial data in XBRL to be combined with data from other industry and government sectors — basically, transforming the way we explore information.

There are various techniques to convert an XBRL document to RDF. I will not go into those details in this blog. One example - GoodMorningResearch.com machine automates XBRL tagging of Excel data in RDF format with one-click Save As XBRL functionality.

I believe that long-term (probably very long-term) vision of XBRL reports should be to publish it as RDF triples and make it a part of Linkedata cloud. This will help in achieving all linkages, transparency and verification as far as financial reporting is concened. I would like sceptics to know that by April 2009, more than 600 XBRL reports, approx. 1,3 million RDF triplets, were already part of Linked data cloud. But at the same time, you need lot more governance, regulations and process behind this effort to get real value. Also, there has to be some kind of incentives for financial organizations to do this.

Yet Another Knowledgebase Knowledgebase?

2009-10-14T20:04:00.012-04:00

Well, those who know yet another hacker jargon or have used yacc, will probably guess where I am going with this! Microsoft recently announced that Bing will have its own reference page - http://www.bing.com/reference. Microsoft, with the help of Powerset team behind Bing, is basically positioning Bing reference as nothing short of key source on the web for all reference needs. I am still struggling to understand the purpose/value behind it as if you try to use it then you can see that most of the articles are from wikipedia or freebase. So, basically, it seems that it is not really a knowledgebase. In that case, it can be more of a branding strategy.

The world of web knowledgebases has become more interesting lately. The vision was really set by cyc - started as an artificial intelligence project back in 1984 - which attempted to assemble a comprehensive knowledge base with the goal of enabling AI applications to perform human-like reasoning. But it ended up being criticized by one of the most controversial endeavors of artifical intelligence history. Wikipedia successfully completed the vision by keeping it simple and used the power of collaboration. Freebase, the effort from metaweb technologies, which is combination of semanticlly enriched wikipedia articles and many other sources like musicbrainz etc.. is already sixty percent more than English wikipedia articles. Despite its size, freebase is still criticized that there are not enough strong use cases to leverage the rich semantic knoweldgebase. Also, I suspect that despite its large size, number of visitors to freebase will be significanlty lower than wikipedia.

Things got more interesting after the release of dbpedia, which is a comunity effort to convert wikipedia into semantic web representation. Now, many folks are saying that dbpedia and freebase should combine their efforts. So after all of this, how can Google, "the database of intentions" as described by John Battele in his famous book - "The Search: Business and Culture in the Age of Google", be far behind! Their knowledge base effort, called Knol - a unit of knoweldge, though still in Beta, is still a mystery to many. It is often compared with wikipedia and some suspect that Knol results will rank higher in search results. I haven't see that though. Many people like Knol as as you are not credited with anything when you contribute something to wikipedia. Whereas, Knol is an open knowledge bank, where you can maintain an account and write what you want. You can edit yourself or allow somebody else to edit the pages there. What's more, you can earn money there by the google adsense program. Money talks!

I haven't even started to talk about WolframAlpha which adds new dimension of knowledgbase computing to the world of knowledgebases. I will save it for another blog.

Only time will tell which knowledgebase(s) will survive.

The Anatomy of an Enterpreneur!

2009-10-14T12:08:00.000-04:00

Enterpreneurs are among the most celebrated people in our culture. I came across this interesting study by KAUFMANN - The foundation of enterpreneurship. A lot is known about the celebrity enterpreneurs like Bill Gates, Steve Jobs, Larry Ellison etc. but very little is known about the rest. It is a very interesting study across various industries. It can be a great source of motivation for future enterpreneurs who are reading this. Basically, its never too late for anyone. Some of the interesting findings are:

Company founders tend to be middle-aged and well-educated, and did better in high school than in college
67 percent ranked their academic performance among the top 30 percent of theri undergraduate class, but a smaler percentage (37.5 percent) ranked their performance among the top 10 percent
The majority of respondents (75.4%) had worked as employees as other companies for more than six years
Significant percentages of respondents started their first companies after working eleven to fifteen years (23.3 percent), sixteen to twenty years (14.3 percent), or greater than twenty years (10.3 percent) for someone else.

You can get the full report here.

Semantic Search, Lexis-Nexis and the Issue of Trust!

2009-10-13T22:20:00.002-04:00

It really caught me by surprise when I heard the news that Lexis-Nexis, a leader in legal publishing, chose Pure Discovery, for its patent research search. Again, I want to remind the readers of this blog that semantic search in this context has to do with 'semantics' only - it has nothing to do with semantic web technology as described by W3 standards. Actually, the business need here is nothing more than what you will typically expect an 'enterprise search' use case to be - Basically, getting the right piece of content or document from a huge corpus. In this case, it is approximately 10 million. I am pretty sure that decision makers and decision influencers in this context were well aware of the enterprise search engines like Fast, Autonomy, Endeca and many others in the crowded category of search engines. All three search engines mentioned above have unique algorithms and can handle the scalability aspect easily. So why did they do this effort collaboratively with a relatively unknown company? The main reason cited here is the issue of transparency! Most search engines expect the end user to blindly trust the relevant results - simply trust the algorithm. But in case of PureDiscovery, each query is enhanced by the machine intelligence and "shown to the user" for their complete understanding and engagement. The user can also control the weightage of the query words in a visual manner called querycloud. It seems that search vendors' saying that "its just magic" didn't work this time

Applying Semantic Technology to cure cancer!

2009-10-13T11:16:00.001-04:00

All of us know that in the world of molecular biology, significant research efforts are moving from wet labs to the dry labs. And by dry labs, I mean bioinformatics. It is a relatively a new discipline but a lot of work has been done in last few years. In a nutshell, bioinformatics is about building an information system using biological data. Semantic Technology is playing a big role in transforming this biological data into connectable information which is giving scientists completely new insight about biological mysteries.

Recently, scientists at The University of Texas, M. D. Anderson Cancer Center got funding to derive meaningful information from an ocean of data about the aberrant genetics that drive human cancers. The work will be primariy focused on parsing the multiple genetic pathways that fuel more than 20 types of cancer. Medical Scientists are going to use semantic technology to solve this puzzle.

For full news:
http://www.eurekalert.org/pub_releases/2009-10/uotm-mda100709.php

How Semantic Technology is revolutionizing the massive e-Science project!

2009-10-12T21:35:00.004-04:00

The researchers at Rensselaer Polytechnic Institute are planning to leverage the semantic technology to solve knowledge acquisition problems for scientists in the scientific domain. Scientists can ask questions outside their domain and get the relevant answers by collaborating with data from various sources. The effort is very similar to the wikipedia except that the data will not reside at one place and all data will be semantically encoded. Each website will semantically represent the scientific data i.e various ontologies will be built for each area.

There are two things which I like about this initiative:
- Raw data is available for everyone, scientists, researchers or a school teacher to access
- It won't be confined to peer review only which is customary in scientific publishing. Even an elementary school teaacher can review the work of a scientist and comment on it.

For full news, please go to: http://news.rpi.edu/update.do

When Cloud Computing meets Semantic Technology!

2009-10-07T11:21:00.001-04:00

There is no doubt that Cloud Computing is one of the most talked about topic in the technology world. While the industry is still working on defining it in a better way, we are begining to see examples where cash strapped smaller companies are actually using it to their advantage. Franz, Inc, a company offering semantic database product, called Allegrograph, is the one who leveraged the concept in very cost effective way. They did series of QA tests for their product for $200 by staging its service on Amazon S3 instead of spending 100k on capital expenditure. As a database company, Franz is under constant pressure to improve its technology and maintain a competitive edge over alternatives like open source or RDBMS vendors.

AllegroGraph is suitable for the purpose of storing very large quantities of Semantic data and supporting complex queries. Franz’s unique “Activity Recognition” package combines Semantic data with Geospatial data, Temporal Logic, Social Networking Analytics to allow users to easily combine this information and perform queries that won't work with tradional RDBMS.

For example : “Find all events that happened in last two months within 10 miles of Newyork that were attended by you and your social network.”

Overall, I perceive it as a great news for future enterpreneurs as the cost of doing business is going down dramatically at least for the software startups!

For full news, please go to:

http://www.ddj.com/web-development/220300736

Where is the scope for innovation in Semantic Technology?

2009-10-02T13:29:00.000-04:00

The research in the area of semantic technology has been going on for more than a decade. W3C has done a great job in finalizing various standards. So what is the next step? It is of course innovation. We will continue to see new innovative software companies cropping up who will build verticalized semantic solutions for various domains leveraging existing data. There is already so much of data out there that the time has come to develop some new algorithms in context of semantic technology. Who says that software is dead now!

In this context, the company which comes to my mind is Metatomix. The Metatomix platform is combination of rules/policy engine, semantic technologies and enteprise-class data integration. They use semantic technologies for informaition modeling and facilitating data integration; use rules engine to drive behaviour; and use integration platform to work with data, connect to data sources, recevie events etc.. The company has developed a very good solution for law enforcement systems. They have developed actionable insights for justice systems personnel - basically finding the right information and then bringing to the right people at the right time can be life saving in the world of law enforcement. Please go to metatomix website to see video which tells you about a real case. It will take only few minutes. The company also has secured a new round of funding lately.

SQL versus Xquery versus SPARQL!

2009-09-30T22:48:00.000-04:00

There is no argument that sql is the most widely used query language and has been in use for decades. It is a long established standard and has been primarily used for tabular data. Millions of developers already know it and tt always tends to get complex when more tables are used. I will avoid talking more about it as I am assuming that people who are reading this blog will know about the limitations and pain points of sql.

Lets focus more on the new query languages like xquery and sparql:

Xquery is great for finding data in tree representations. You will find almost all relational database vendors mixing xquery with sql though it tends to get very complex. Xquery is even becoming part of sql standard now. Xquery works great with native xml databases like marklogic and xhive (now Documentum). Xquery is most popular with content publishers. It is a standalone programming language for information intensive applications. To get deeper into this, please use this brilliant prez.. Xquery is still a relatively new standard so the developer community is nowhere in the league of sql.

Sparql is more useful in a pattern matching paradigm when you have to traverse many relationships. It makes federated queries possible, and federated queries can be thought of as queries made simultaneously to multiple, disparate databases that could be located anywhere on a network. This is a key concept in making disparate and separate data stores seem like a single respository. Those who want to dive deeper into sparql should use this tutorial.

Why Cambridge Semantics can make a difference during these times?

2009-09-30T21:42:00.003-04:00

There is no doubt that these are hard times for smaller software companies to get into the radar of a CIO/CFO of any enterprise - small, medium or big. All of us know that in any case only 20% of budget is allocated for new projects or products. It will be much harder for a company who is smaller and is offering a relatively new technology which only very small percentage of technology folks have even heard of it. I really like the practical approach and focus of Cambridge Semantics, a Boston based semantic technology startup.

They are not asking the end users to write big Ontologies or think about any other paradigm shift. They use semantic technology to solve some very common problems like consolidating various spreadsheets. The Anzo suite of products enables you to liberate your data, allowing it to be exchanged freely between proprietary applications, relational databases or even Microsoft Excel spreadsheets. They simply leave the data where it is, read and write data between multiple data sources including spreadsheets and let the user create the view they want. The application can be up and running in few days.

Recently, UCB, a leading biopharma company, selected Cambridge Semantics to combine and analyze assay data from dozens of partners around the world. You can get the full news here.

Interesting research from European Semantics reseachers!

2009-09-28T11:04:00.001-04:00

The research, called "Triple space", is related with making machine talk more in an asynchronous way using web services. The analogy is very similar to cloud computing in which computational resources are distributed and provided as a service over the internet, the Triple Space deals with data – offering a simple, scalable way for machines to share information asynchronously.
To create the Triple Space, the TripCom researchers worked on making web services and the data they use understandable by computers, using semantic web technologies to communicate machine-readable knowledge rather than raw data.

Just as humans can access the same webpage with different web browsers and different operating systems, computers are able to publish and read information in the Triple Space without format, process or technical constraints. One of hte use cases cited is e-Health record systems. For full story, please go to :

http://www.sciencedaily.com/releases/2009/09/090923105631.htm

Is Bing taking the market share from google?

2009-09-28T10:51:00.002-04:00

The story posted in cnnmoney.com reads like this : "Microsoft stages a big comeback." The story talks about new offerings from Microsoft like windows 7, cloud computing and Bing - their new semantic search engine. Microsoft, who is viewed as a laggard behing google in the search/advertising business seems to be getting more market share from its rival. Its share of online business has grown to 9% (some sources even say it is 10%)after it released Bing.com . The 9% takes msn also into account. If they manage to make it to 15%-20% range then it will transalte into billions of dollars.

For full story, you can go to:
http://money.cnn.com/2009/09/28/technology/Microsoft_stock.moneymag/index.htm?postversion=2009092804

Google adds semantic web support for video search!

2009-09-25T12:20:00.000-04:00

Google, who has been behind Yahoo in semantic web efforts, is trying to catch up fast. Google announced support for enhanced markup for video search. This will allow webmasters to include important information, such as titles and descriptions, in machine-readable HTML along with the JavaScript or Flash videos themselves. It will use structured data open standards such as microformats and RDFa to give users more detailed previews of the information contained on the web page. Yahoo! searchMonkey has done something very similar for the content.

Tim Berners Lee video about his journey and vision!

2009-09-25T11:15:00.001-04:00

This interesting video was published this year.

http://www.ted.com/talks/tim_berners_lee_on_the_next_web.html

He talks about his journey of last years from regular web to the semantic web.

How Semantic Technology helps Data Strategy? The discipline of Business Semantics!

2009-09-24T20:35:00.000-04:00

Data Strategy has again become a "hot topic" in the enterprise. The two things which are of utmost importance in this area or require painstaking work are:

1) Data Standardizaton - Agreeing what to call things, which things are equivalent

2) Data Governance - Who "owns" the data, who is the steward, who gets to change things, who is responsible for quality

If you really think about it then more time is spent on analyzing the quality of data and doing reconciliation then anything else. Bottomline is that there has to be a common language between business and IT. Business semantics or developing a dictionary based solution is the key here. You can call this business dictionary as a knowledge base or a an ontology based solution.

In my view, an “Enterprise Ontology” in the Semantic world is equivalent to an Enterprise Data Model in the Relational world. Developing an ontology is similar to creating a data model. However, one important difference is that ontologies allow you to seamlessly move from logical design to physical implementation. Also if you have established a business dictionary using the Web Ontology Language (OWL), you have encoded it in machine-processable as well as human-readable form.

Relational databases (and the applications that use them) behave as if every row in a table is unique. So when a customer database contains "Microsoft” and “Microsoft Inc.,” the system behaves as if they are two different customers. In the world of ontologies, we should be able to simply declare the equivalence of the two instances and be done with it.

Business Semantic is a fresh market and there are very few case studies out there. This market will only grow in the long run as using semantic technology approach for ensuring data lineage and data quality is the best possible approach from flexibility and maintability perspective. There is a start up in Belgium called Collibra which is actively working on it using purely semantic approach.

Web 2.0 versus Web 3.0! Which is real and has more long-term value?

2009-09-24T11:47:00.005-04:00

Recently, someone forwarded me this interesting video - http://www.youtube.com/watch?v=oalBUgzKaLw&hl . This video argues whether social media or web 2.0 to some is fad or not. In my opinion, this video is powerful! Social media is a great concept and it works - it is exploited very cleverly by early adapters. But there is not much you can do from enterpreneurship or innovation perspective. I see that there are three categories: 1) Facebook, Linkedin, myspace, youtube, wikipedia etc. etc.. - Basically user-generated- content! The game is pretty much limited as there is opportunity for very few players here. All these companies are real and are going to stay for long. The only way to make money here is through advertising. 2) Software companies who are making wikis, blogs and other interesting ways to collaborate etc. - Sharepoint will be a good example. This is very horizontal in nature. The focus is productivity. There is not much scope for innovation here also and it is very saturated market. In the end, it is all about configuration, governance and best practices. 3) Wikinomics examples - Top companies are inviting people/customers/clients to give design/product ideas etc.. e.g Nissan or P&G inviting their loyal customers for new ideas. In the end social media is not about software but the power to attract a certain segment who has common interests and are interested in contributing content. Number 1 category is the only one where you might see some limited innovation in a verticalized domain. This is the only category where very few will ever make any money using advertisement. Social media is really about people to people. Tim Berners Lee, the father of web, goes on to say that "if Web 2.0 for you is blogs and wikis, then that is people to people. But that was what the web was supposed to be all along." There is just no new technology here! The viedo, mentioned above, doesn't talk about the business value of TV and radio aagainst social media. When TV and radio were invented, there were two kinds of people who benefitted tremendously: a) Those who designed/manufactured and sold them. b)The people who created content/channels/programs - this is endless cycle as you always need to create a new content and a vast industry mushroomed around it. Social Media is all about user-generated content but people who create content gain not much monetarily. Whereas, In case of TV and radio, people who created content benefitted tremendously. So Where does this leave Semantic Web or Web 3.0 called by many? The key concepts have been explained in earlier blogs and I will continue to do in future blogs. But Web 3.0 is really a data web or executable web. It is about reassembly of data and reorganization of data pieces. It can help in solving problems from drug discovery to national security. Even a semantic wiki has alot to offer than a regular wiki. We are going to see unprecedented level of innovation in this space.

How Open Calais initiative is helping Semantic Technology!

2009-09-23T13:49:00.011-04:00

Open Calais initiative, started by Thomson-Reuters, is one of the most interesting things which has happened in favour of semantic web vision. Reuters acquired this technology as part of their ClearForest, one of the leading vendors in the text analytics space, acquisition. It is stated that the service could quickly become the largest repository of metadata (in the form of named entites and facts) on the Web if it stored the resulting metadata from each request. Open Calais is the "metadata extraction service" ; it is a Web service that allows you to automatically annotate content and extract information like facts and named entities (people, places, and organizations, and much more) from unstructured text. Calais uses linguistic parsing (also known as entity extraction) in a service enables way to producr RDF triples and Semantic Web data models.
Open Calais opens the door to the possibility of lowering the barrier enough for everyday users to publish semantic content. It finally does what critics say to be the greatest obstacle to the Semantic Web: Taking the metadata burden from the end-user by providing an automatic meta-tagging tool. Open Calais initiative will also be one of the biggest enabler of the Linked data initiative.
Recently, CNET has joined OpenCalais initiative as one of the first commercial media companies to publish core data assets for public, programmatic use on the open semantic Web. CNET will leverage OpenCalais' connection to the rapidly expanding 'Linked Data cloud' to allow its original content -- such as tech product reviews on laptops, TVs, smart phones, and digital cameras; news articles and blog posts from its CNET News editorial staff; and parts of its core technology product catalog - to be available for public use.

Example of Semantic Technology in action - Yahoo Search Monkey

2009-09-22T12:06:00.015-04:00

Its is a misconception when people question the viability of semantic technologies by saying that "it isn't possible to convert all the data to RDF format?" In reality , it is not true at all. Again, it is important to keep in mind that we are not talking about semantic web, we are talking about semantic technologies. It can be best explained by talking about how Yahoo Search Monkey is using RDFa (RDF in Attributes) to enhance search results more useful and visually appealing. It is just one of the simple examples but it can hopefully answer the sceptics. It can be a boon to small businesses who can drive more traffic to their web sites. SearchMonkey looks for special data inside websites, based on a standard called RDFa. Your website should include this data so it is available to Yahoo as they crawl and index your site. This way your business information is available to any developers who build SearchMonkey apps, and you will show up with enhanced results as this gets adopted over time. The SearchMonkey platform has three main components: - "Site owners share structured data with Yahoo!, using semantic markup (microformats, RDF), standardized XML feeds, APIs (OpenSearch or other web services), and page extraction. -Third party developers build SearchMonkey applications. -Consumers customize their search experience." RDFa is a way to encode data within HTMLand XHTML pages which helps people and machines to embed structured data within HTML and XHTML pages. The underlying representation of RDFa is RDF because it is flexible enough to let publishers build an devolve their own vocabularies. You can see Search Monkey in action by clicking on this link http://www.yelp.com/search?find_desc=nobu&ns=1&rpp=10&find_loc=San+Francisco%2C+CA#find_loc=new%20york
You can see the enhanced quality of the result "nobu new york restaurant". You will probably realize that you have many using Semantic technologies without even realizing it.

What do we mean by "Semantics" in Semantic Search?

2009-07-14T11:46:00.031-04:00

I have often been asked this question in last few months by clients and other people who are interested in finding more about the activities in the world of search. Is it same as natural language search? Or it is just applicable for semantic data expressed in RDF and OWL? So companies who are developing semantic search are basically trying to throw google out? How relevant is it for an enterprise. These are the standard questions which I often get asked. At high level, semantic search engines is about interpreting meaning of the "query" in a smart way. Most of todays search engines today are primarily based on "key words" focused though the trend has been changing since early 2008. If you ask a question: q: What is the time difference between US and Tanzania ( a country in Africa)? average search engine : Will give you all documents having key words 'time difference', 'US' and 'Tanzania'. This is the case with millions of web sites who are using "plain vanilla" search. good search engine (having some semantic capabilities) : Will probably understand the intention of the query and give you links to the websites which talks about time difference between countries. There are still various inconsistencies in the answers if you test any of the popular search engines out there. You just need to ask the same question in a different way to validate the 'semanticity" of the engine. A Real Semantic Search Engine : Will give you a time difference by getting the right content or it will tell you that it doesn't know the answer. This is very difficult to attain on a consistent basis by by any technology out there. So most of the discussion will focus on good semantic search engines. Even though, the popular search engines like google, yahoo, bing (Microsoft), ask.com etc. claim that they have semantic abilities, they will still be in the category of a good search engine. They are hybrid of "key word" and "semantic" based search. None of these engines are purely semantic in nature. Google is very clear about its messaging - they are not going to replace key word search with semantic search capabilities. Most of the search engines today are really based on how the query is phrased. They do a poor job in understanding the meaning of the query if is phrased in different way. Semantic search engines should do disambiguation very cleverly - If someone is talking about "furniture", the semantic engine should even cosnider documents with and "tables" and "chairs" in it. Basically, it should understand the context. A good semantic search engine takes the burden from the user to answer a query even if it is asked in very different way. It is also very important to understand that these engines are not getting their content from semantic web. They are still working with the same set of millions of document in the regular web. They haven't touched semantic web for your query needs. The "semantic" aspect in these search engines are only related with their ability to interpret the query. Swoogle is the only semantic search engine which only fetches the content from the semantic web - basically data written in RDF format. The need to improve the "semanticity" of search engines will grow exponentially. It is very difficult to measure it though. There are primarily two categories : Pure play semantic search engines like Hakia,Sensebot, Congnition Search, Exalead, Powerset and many others. And the other category is Google,Yahoo, Microsoft and Ask.com of the world who are integrating semantic search algorithms in their core search technology. The line is blurring between these two categories. There is not just one approach to semantic search. Most semantic search engines mix and match them in various ways to yield a unique search experience for their users. There are at least four approaches to semantic search. Different semantic search engines may use one or more of these approaches. The point of semantic search is to use meaning to improve the user's search experience. For example, one approach is to use contextual analysis to help to disambiguate queries. Another approach focuses on reasoning. Given a set of facts that are represented in the system, additional facts can be inferred from them. A number of semantic search engines emphasize natural language understanding. These engines process the content they index and the queries people submit to try to identify the intent of the information. They use the syntax of the sentence and rules to identify people, places, organizations, and so forth. Powerset makes extensive use of natural language understanding. The fourth approach uses an ontology to represent knowledge about a domain and expand queries. On this approach, when a user enters a query for a word like "sofa sets," the system adds terms from its ontology (e.g., "furniture" because a sofaset is a kind of furniture) to make the search more focused as well as more broad. This approach is used by a large number of semantic search systems. Google and Yahoo will continue to have an edge because of existing vistors to their site and their continued investments in these technologies. Despite claims by various vendors, semantic search engines have a long way to go. There is a big gap between the existing information out there versus the tools which can actually get that information.

How do you explain Semantic Technology to someone who has never heard of it?

2009-07-13T13:15:00.001-04:00

“Any intellegent fool can make things bigger and more complex. It takes a touch of genius - and a lot of courage to move in the opposite direction.” Albert Einstein

This blog is not about me trying to be a genius in a subtle way. But is about a question which has been bothering me for some time. How do you explain a concept like semantic technology to someone higher up in your organization who will never have the time to go to a conference or a read a book on OWL? The idea is to simplify it in simple words but at the same time convey the value of it. Explaining it as Web 3.0, read-write-execute web or rebirth of AI won't really help.

IMHO, semantic technology can be explained as a way to describe data models and representation which allows to be linked with other data models as if they are part of the same big gigantic database.
Semantic technology solves data integration problem in a big way. It simplifies the way you can connect and exchange data with many systems. The value is that you can maintain it much easily. Also, with the introduction of new semantic tools in the marketplace, we will continue to see that it will be maintained by the business users with mimimal code change. THe W3C, the standards committe behind the semantic web, has done phenomenal job with coming up with languages like RDF and OWL which makes all of it possible.
Business have already made substantial investments in databases, data warehouses, BI, ILM, ERP, CMS and enterprise search. Forming a complete picture of enterprise data is very hard to achieve. Tracking data on timely basis is one of the biggest business and technical challenge! At the end of the day, we need relaible information about business performance! Semantic web technologies, if applied correctly, can solve this problem without scraping your existing investments.

This primer http://www.semantic-conference.com/primer.html might be useful to more inquistive kinds.

Semantic Conference 2009

2009-07-12T21:46:00.001-04:00

I was fortunate to be one of the attendees at the Semantic Conference 2009 held in Fairmont hotel, San Jose, Ca. I have to admit that I was very impressed to witness a huge crowd, probably 1300+, during these recessionary times. I was told by one of the organizers that the number of attendees has been growing considerably since the inception of this conference back in 2005. It was also very encouraging to see more than 20% of the attendees from the business side that just validates the fact that semantic technology is a no longer confined to R&D labs and intellectual discussions. There is a strong interest to apply this technology to derive maximum business value from it. Some notable sessions were:

Keynote from Reuters about state of Semantic technology and where is the money
Semantic Search discussion between C-level executives from Microsoft, Google and Yahoo
At least two sessions from the VC community expressing strong interest in the business side of semantic technologies

I had no doubt, even before going to this conference, that semantic technology can solve some problems which is very difficult to solve using the traditional or well-accepted technologies in the enterprise. I was was very disappointed to see almost lack of representation from even a single well known name from the Wall Street. It is well-known fact that financial industry has been an early adapter of most of the new technology but it seemed this wasn't the case in this context. Maybe, the turmoil in the financial industry is taking its toll on everything. On a brighter side, I saw a large number from the health care and pharma companies who had presented many use cases and application of this technology to solve some real business problems. I was also surprised to see a large convoy from Europe who presented many government-sponsored projects in the semantic technology. It seems that Europe has finally decided to take a very pro-active and leadership role in advocating semantic technology!
To sum it all, I left the conference with a smile on my face and with added enthusiasm for the semantic technology. There is no doubt left in my mind that it is just a question of time when this technology will be a mainstream technology and business benefits will be very real.
More to follow in the next blog ..