Showing posts with label Semantic technology. Show all posts
Showing posts with label Semantic technology. Show all posts

Thursday, September 29, 2011

Data Without Borders: Data can be a burden if it is not set free!

Wikipedia describes philanthropy etymologically means "the love of  humanity"— love in the sense of caring for, nourishing, developing, or enhancing. Historically, philanthropy has always been associated with giving generous donations of money. It will continue to be associated with giving generous donations but some professionals can make bigger impact by donating their skill set than money. Yes, I am talking about the skill set of data science! Data science is relatively a newly coined term and probably originated from data geeks working on hard data problems in the companies like Linkedin, Facebook and other technology companies who needed these experts to make sense and insights from the vast amount of data being produced everyday. Data Without Borders, a newly founded organization, seeks to match non-profits in need of data analysis with freelance and pro bono data scientists who can work to help them with data collection, analysis, visualization, or decision support. The concept is brilliant and makes sense!

There are various initiatives out there where technology is being leveraged creatively to help the non-profit organizations. The Bill and Melinda Gates Foundation has recently funded a new digital-media hub call ViewChange.org. The hub uses semantic technology to create a platform that combines the video sharing power of YouTube with the open information of Wikipedia and the mission of your favorite advocacy organization. I had written about it in more detail in one of my posts titled - Philanthropy goes Semantic.  Ushahidi, initially started as a simple web site to map reports of violence in Kenya, is another non-profit tech company that specializes in developing free and open source software for information collection, visualizing and interactive mapping. To my knowledge, Hans Rosling, a medical doctor and a statistician with decades of work studying outbreaks in Africa, is probably the first data science philanthropist. He co-founded Gapminder foundation which developed the Trendalyzer software, acquired by Google, that converts international statistics into moving, interactive graphics. His TED presentation about his best stats you have ever seen is worth watching.

The genesis of the idea of "Data without Borders" is to match the NGOs, who are sitting on lots of data with nobody to look at because of  resource and budget constraints, with data scientists who have the energy, time and passion to make sense of this data. Timing of this initiative couldn't be better because data scientists can now have a common and noble cause to rally behind! It is the beginning of a powerful vision but it will surely have its own challenges.  Having some experience with an NGO myself, I can say that sustaining the enthusiasm and commitment of data scientist for a long-term can be challenging. We are all aware that data scientists are going to be one of the most sought after, busiest and highly paid professionals in the next decade! So I will go for a good data scientist with more commitment over a rock star data scientist in this context. Also, a weekend of data hackathon in this context will probably won't be enough because data Science is an iterative process and will require an ongoing engagement. It is still not clear to me that why there are not initiatives like open government data in case of NGOs to build powerful data mashups. I am aware of new standards like IATI but its more about aid spending by governments. In this context, I believe that too much data can be a burden if it is not set free and used effectively. Ideally, in case of NGOs, open data shouldn't have political or privacy barriers. In the end, the co-founders of "Data without Borders" will need all possible support, structure and maybe funding, to be successful in their mission. Winston Churchill, rightly said, "We make a living by what we get, but we make a life by what we give."

Wednesday, June 8, 2011

Debunking Myths about IBM's Watson!


There are many instances of televised technological feats of human race which have managed to leave a lasting impression on us - In my opinion, probably, Apollo 11 landing on Moon on July 20th 1969 will always retain the number one spot - in terms of impact value at a given time. I don’t know who is the close second but IBM Watson’s televised Jeopardy challenge when it bested Brad Rutter, the biggest all-time money winner on Jeopardy!, and Ken Jennings, the record holder for the longest championship streak, will always have a place in history. Organizing these kind of challenges is not new to IBM - who can forget when Deep Blue, a chess playing computer developed by IBM, beat world champion Gary Kasparov in a controversial match on 11th May 1997.




A lot of great things have been written about Watson, named after the IBM's first President but also influenced by the name of Sherlock Holmes' assistant Dr. Watson, since the televised challenge. Though, at the same time, you will also find many tweets ridiculing, hopefully in a good humour, that it needed lessons in geography after the (in)famous "Toronto" answer. But if you understand even little bit of information retrieval, natural language processing, machine learning, knoweldge representation etc.. then you would have realized what an amazing accomplishment this is. Recently, I had the opportunity to meet Aditya Kalyanpur, one of the team members of the DeepQA project which built Watson. Rome wasn't built in a day! Similarly it took more than four years and roughly twenty five brilliant technologists to build Watson. There are many unknown facts which we will come to know in due course of time. for e.g. Did you know that from September 2010 through December 2010 Watson played 55 games against Tournament of Champion Jeopardy! players and won 71% of the games. These players represent some of the best Jeopardy! players in the world.

Aditya managed to debunk few myths about Watson and also highlighted the approach taken by the team to develop the software. I would like to share them with you:

Three prominent myths:

• Watson answers a question by changing the query to structured query; then it queries a structured knowledge base. This is not true at all. This approach is taken by traditional QA systems and is very brittle with poor domain coverage.

• Watson identifies the question type and generates candidates from precompiled list of instances. This is false. Watson relies on a radically novel open domain type “coercion” technique.

• It uses either structured or unstructured data analytics. This is not true either. Watson integrates information from both unstructured and structured data analytics.


Some of the notable things about the DeepQA project:

• It does deep analysis of a question by breaking it down into relevant components like the focus, key entities and relationships, classifying it into broad classes, requirements of special handling etc..In one such strategy the system identifies independent facts within a given question, poses these as new questions to the underlying QA system, and generates answer candidates supported by the facts. Candidates that have support from multiple independent facts reinforce each other to boost system confidence in them

• Hypotheses, which in the QA case, are potential answers to the question, are generated by the Search component, which retrieves content relevant to a given question from the large volume of local knowledge resources Watson can access. The sources of information for Watson include encyclopedias, dictionaries, thesauri, newswire articles, and literary works. Watson also used databases, taxonomies, and ontologies, for example, DBPedia, YAGO, WordNet.

• Candidate Generation component identifies potential answers to the question from the retrieved content. A variety of answer scoring algorithms are then applied using DeepQA’s pervasive probabilistic framework. Other than linguistic processing, taxonomic, geospatial, temporal, popularity and source reliability are some of the evidence dimensions used by the scorers to constrain or support the right answer.

• UIMA-AS was used for orchestrating the overall processing and an in-memory implementation of Sesame was used for storing RDF data. Initially, it used to take 1-2 hours to answer a question but using a massively parallel architecture and exploiting more than 2800 P7 cores, QA time was reduced to a few seconds.

• Machine learning and Monte Carlo methods used by the strategy components to estimate and optimize the win probabilities for the various players in a particular game state



Sunday, January 23, 2011

Hakia's SenseNews: Can it really tell you when to buy and sell stocks?

There is a always a big difference between an exciting technology and an exciting business. And understanding those differences is the diffference between success and failure. Hakia always had an exciting technology if you ever cared to go to their web site and tried to evaluate their semantic search results. Unfortunately, they never had a succesful brand name like Google, Yahoo etc. which is probably true for most of the startups in the semantic search space. As a result, they could never attract too many visitors to their website. What got my attention is their recent announcement about Sensenews which is a service to make buy and sell recommendations for any stock. Sensenews reads news sources (over 30,000 news sources), blogs (over 1 million), Twitter and performs an advanced computation to make buy/sell recommendations to you. 



Stock movement is always correlated with news - good or bad. An example in the fall of 2008 shows how unexpected news can impact prices dramatically. At 1:37 AM, September 7, 2008, Google's newsbots picked up 2002 story about United Airlines filing for bankruptcy. The story had a cascading affect and it was picked up by a person, who obviously failed to notice the date on the story, and put it on Bloomberg. The stock droped 76 percent in six minutes with a huge spike in volume.

The issue with news is that it is not mathematical in nature. It is very hard to make machines understand news. Various semantic technology, text extraction, natural langauage and statistical techniques are used to maker sense out of vast amount of news for a particular stock, currency or commodity. Hakia is not the first one who is trying this as:
  • Monitor 110, funded with 20 million dollars, was one of the first high profile failure in the capital markets and Hedge fund space. They had signed up almost hundred trial clients. They closed the shop in July 2008.
  • Relegance (a company out of Israel), one of the last acquistions (between 50 million to 100 million US dollars) of AOL, was also in this space. AOL is often accused of not leveraging Relegence’s technology effectively. But then AOL is accused of doing lot of things not in a right way in last few years. Relegance's technology is being used for AOL Money & Finance and WalletPop.
  • BNP Paribas developed their own indicator that reads the news looking for words that imply strength or weakness, and decides how to trade depending upon the number of times various words are used.
  • German exchange Deutsche Borse acquired Need to Know News, a Washington D.C.-based provider of machine-readable news for automated trading engines.
  • Reuter's NewsScope and Sentiment Analysis product is designed for use in algorithmic trading, where computer programs make all of the trading decisions including the entry and exit of any trades. More than half of the trading is now automated, much more for US stocks.
  • Stocktwits - The service takes financial related data from tweets and structures it by stock, user, reputation, etc. 
This is a very hard and known problem which is becoming more painful because of the information overload. The whole idea is to:
  • Exploit market inefficiencies by showing the unrealized value of a stock
  • Manage event risk by reacting to breaking developments
Probably, not even 2% of the firms in the capital markets are doing this. There are hardly known successful cases - there is always a secrecy aspect also. But I don't know any vendor who has truly succeeded in making a successful business out of it. Semantic technologies are sometimes viewed as a technology which is looking for a business problem to solve but this case is different. At some level, the business problem is defined (yes, the requirements of every trading desk or traders in different firms will vary), but the technological limitations are exposed by its inability to semantically filter the right content, duplicate removals, understanding the right sentiment, categorizing it correctly, getting rid of marginal news, irrelevant stories and many other things. I also think that it is very hard to understand the social media manipulations as it is an opportunity for someone to make lots of money by fooling so called intelligent programs.

What is interesting in the case of Hakia is that they have gone a step ahead by building a recommendation model on top of semantic filtering of the content. This is really bold and interesting! Their delivery model is subscription/consumer based which is different from most of the ones who have tried this before or trying now as they were/are more enterprise focused with the exception of stocktwits. I do think that they should have offered thirty day free trial period for this service from marketing perspective. It is not clear what level of fundamental and technical analysis is done. I also don't understand the profile of the trader they are targetting. It also doesn't seem to be realtime in nature and won't work correctly for stock splits. But if they can deliver based on what they are claiming then it is commendable. Only time will tell whether they will be successful in this new business where no company has truly succeeded yet. But one thing is for sure that they have understood that future of semantic search - that it is very much related to being some kind of content clearing house i.e.  clean, curate, collate and categorize content.


Thursday, November 11, 2010

Predictive Analytics for social data: Is there a role for Semantics?

Recently, Gartner, the leading technology analyst company, came out with its predictions for key technologies for 2011. The list comprises of cloud computing, mobile applications, social collaboration, next generation analytics, social analytics and many others - shouldn’t surprise you if work in information technology. The only thing which was not clear to me that why next generation analytics and social analytics are in two different categories when social collaboration is already emphasized as a part of a roadmap for large enterprises. I am sure they must be having their own valid reasons to do so. But the point is that overall it is getting a bit confusing about the various terms which are being used in context of analytics. What I mean here is what is the difference between just analytics versus predictive analytics versus forecasting versus predictive modeling versus optimization versus data mining versus advanced analytics. To many people, it sounds same! Also, it will depend upon who you ask this question. If you are a vendor or a consultant then try explaining it to a decision maker in an enterprise - basically, try not to get into that conversation. Many of these disciplines are more than a decade old as something like predictive modeling has been used in credit scoring for years. Also, academically, there is not much difference between classic techniques used in data mining and in statistics. Though, data mining has evolved to deal better with real life messy data. Unfortunately, unlike analytics, statistics could never become a hot topic but maybe it is about to change. In short, analytics or predictive analytics is the umbrella term or the new term - maybe the buzz word. It seems there is new surge of interest in predictive analytics because it is about the future outcomes in context of business intelligence. There is a difference between insights and gaining foresight!

You can always question that companies were always worried about the future outcomes so what has changed now if many of the methods were available before also. Probably, more data is available now, and there has been advancement and simplification of  tools/techniques - you can hire a good business analyst to do the job instead of someone with a doctorate in statistics. Predictive analytics enables you to develop mathematical models to help you better understand the variables driving success. Predictive analytics relies on formulas that compare past successes and failures, and then uses those formulas to predict future outcomes. Also, if you consider the fact that IBM has spent almost eleven billion dollar in the last five years acquiring software companies, like SPSS and Unica, for its analytics consulting organization then it starts making more sense.

On top of that, social analytics is a new kid on the block and there is new buzz that it is going to play a significant role in predictive analytics. I do believe that it will become true gradually but it is  not going to happen as quickly as we are being made to believe. What are the challenges in it? From the process perspective, predictive analytics is about understanding the prediction variables to the business problem, selecting the relevant statistical technique, validating the model with the test data and finally applying/adjusting the model iteratively with the production data. Do you think that it should be very different in context of social data? First of all, just because a company is doing brand monitoring (there are just too many companies), it doesn't make it a social analytics company. What I mean here is that if they are just following converations about an entity and don’t have much semantic intelligence in their software. If you look at most of the common examples of so called social media helping in predictions, they are about topics about election predictions (as recently claimed by Facebook political team) or about how a new product like iPad is perceived - the outcome is most of the time boolean i.e success or failure. In my opinion, these are interesting examples but much more is expected to do predictive analytics from social data. Maybe, if you consider social analytics and all associated prediction with it as a seperate or standalone discipline then it is good enough - but then you don't have an integrated view from an enterprise perspective. Maybe, in some cases, you don't need to integrate social data with enterprise data and still can get some value. But, you still need to build a repeatable predictive model using the social media data. And before you build the predictive model, you need to do true semantic analysis of the social data. Build some kind of normalized social data model to work with enterprise data for predictive analytics. IBM has come out with a new offerring where it claims to have enhanced SPSS software with social analytics and it can do predictive analytics for your business needs. It also offers semantic network analysis of the social data. I am not aware how well it works or if it can work across different business domains without too much business analysis or customization.

It is not an easy task to build a repeatable predictive analytics with a everchanging large volume of social data. The quality of meaningful data is also very important in this context. I see the ambiguity of social data as one of the biggest hurdle. You will have to do lots of preprocessing and deal with many new attributes being added in the social context. Every company is unique and we will see predictive analytics manifesting itself differently for each of them. Though, I do believe that we will see many companies building very high number of predictive models which will take social data into consideration for their business needs - maybe a predictive model per product and total turn around time of a week or less from problem definition to scoring. That brings up another question - even if real-time social data is present, can you take real time actions? If not then what is the true value of real time data in context of predictive analytics? You really need a very different level of infrastruture to take advantage of it.
 
This whole integration of social media analytics with predictive analytics should be owned by business - not by IT.  Infact, in most of the cases, it should be owned by marketing departments because they value/understand social data more than any other department and they are also most qualified to define prediction goals from a social context, predictive behaviors, pragmatic tradeoffs and evaluating results.

It seems there are some good opportunities in this space. We still have a long way to go. A lot needs to be understood before making any claims in the industry. There are definetely some good examples in the research community like SOMA, a forecasting model developed by a researcher at the University of Maryland, Terror Organization Portal. It analyses a wide range of information about politics, business and society in Lebanon to predict, with surprising accuracy, rocket attacks by the country’s Hizbullah militia on Israel. By the middle of 2010 SOMA was sucking up data from more than 200 sources, many of them newspaper websites. There is another example of two researchers at HP Labs who have established that they can use tweets to predict how well a movie will do - the results turned out to be fairly accurate. What we don't understand that can these examples be generalized for the indusry adaption? It will be good to know more examples from the industry.

Tuesday, March 23, 2010

XBRL Implementation Status: Doing a Reality Check

It was only few months back when I wrote the post - Semantic Technology, Financial Reporting and Toxic Assets. The post was more about how XBRL and the Semantic technology can play a big role in addressing issues related with toxic assets in the world of financial derivatives. It is becoming more clear now that XBRL is almost a movement which is going to have deep impact on how information between businesses, regulators and investors across the Internet will be communicated in the next decade. XBRL is not only considered the most revolutionary change in financial reporting since the first general ledger but also one of the most successful semantic web format. So what is the status of XBRL implementations? You can check this  interview with Eric E. Cohen, co-founder XBRL, about the recent updates. You can also find some good information at Charles Hoffman's blog or  http://xbrlplanet.org/. I also liked the section on XBRL in David Siegel's Pull where he talks about the history of XBRL.



Some of the points which I found interesting are:

  • XBRL is almost ten years old but the real adaption has started in the last few years only
  • Regulators are one of the first group to adopt it. Regulators who are adopting XBRL are : Capital market regulators, tax offices, banking regulators, national statistic offices, corporate registrars
  • Corporate America has already been complying with a mandate from the Securities and Exchange Commission for nearly a year to “tag” financial data in XBRL
  • There have been issues in many projects but no significanlty failed XBRL project has been reported
  • June 15th, 500 public companies did XBRL filings. Another 50 have started doing this even though they didn't have to file. 2009 taxonomies came late so 2008 taxonomies were used
  • June 15th 2010, another 1500 will start filing in XBRL
  • June 15th , 2011, another 10,000 will do
  • One pending bill in Congress would direct all federal agencies to adopt XBRL for all requests for government bailout funds and all required reports on how those funds are used
  • Edgar Online gets $12 million from Bain Capital  for XBRL efforts
  • Using XBRL, FDIC reduced the time to report information from 45 days to 2 days
  • European Parliament is the largest government body who has expressed interest in XBRL
  • The Securities and Exchange Commission launched its XBRL information portal, which can be found at http://xbrl.sec.gov/
  • The investment in person hours that it took to create either the IFRS or the US GAAP taxonomies dwarfed the total hours needed to create XBRL itself
  • Over 90% of Spanish banks now report in XBRL
  • Holland and Newzealand are already accepting tax returns and other government required documents in XBRL
  • Nevada is one of the first US state who is trying to use XBRL in many of its operations
  • XBRL taxonomies may not be interoperable. For e.g, the US GAAP and the IFRS taxonomies are all used for financial reporting but are significantly different
  • MIX (Microfinance Information Exchange) which collects information from more than thousand microfinance institutions is using XBRL. It is the first non-profit to use XBRL
  • Data aggregators and distributors have not embraced XBRL in any significant way
  • Taxonomy extensions is one of the hardest issue to solve as it reduces standardization and make interoperability very difficult
  • You can find some common errors in XBRL implementations in this excellent article
  • It is observed that few technologists have accounting domain knowledge and few accounting experts have technology domain knowledge in XBRL projects
  • XBRL International, Inc. (XII), has released “XBRL: Towards a Diverse Ecosystem," a discussion document seeking public comment on the future business requirements and technical roadmap for the XBRL business information standard. The document may be downloaded here.
I am sure that there will be many more highlights but it gives good idea that things are looking good for XBRL despite some initial challenges. It is still in early stages of adoption on Wall Street. But the progress is being made and it looks like that the financial reporting and analysis landscape is going to be a very different one in the coming years!

Tuesday, March 2, 2010

Semantic Search momentum continues: Netbase gets 9 million dollar funding

It was only six months back when Netbase, a semantic search company focused on Health and Consumer segments, went through lot of negative publicity after Techcrunch published this story titled "Netbase thinks you can get rid of Jews with Alcohol and Salt." NetBase Solutions’ HealthBase, a semantic search engine that aggregates medical content from millions of authoritative health sites including WebMD, Wikipedia, and PubMed was the center of this story when many users found some glaring glitches. It wasn't over for Netbase despite heavy criticism as it is evident from this news which say that it has raised $9 million in venture capital funding, completing its Series C round with Altos Ventures and Thomvest Ventures Inc --obviously, the VCs are seeing something which the media couldn't. In my opinion, these technologies are still a long way from being perfect and we will continue to see 20% to 40% errors - in many cases, it is dependent on the domain/content and the technology needs to be tuned for that.



The marketing message of Netbase is not very different from what you hear from many companies in the semantic search, natural language or text analytics space but they do have global brands like PandG and Elsevier as customers. The company also claims that by April about sixty percent of nurses around the world will be looking up information in its resources with the help of NetBase’s HealthBase Insight Discovery tool.

Overall, it is good news for semantic search companies who are trying to raise money as VCs believe that they have a good future and demand will only grow. I am sure they are aware of imperfections but can still see the value it can deliver to solve certain problems in a particular domain.
Reblog this post [with Zemanta]

Friday, February 26, 2010

Bill and Melinda Gates Foundation: Philanthropy goes Semantic

Bill and Melinda Gates Foundation is undoubtedly one of the most transparent and largest privately owned philanthropic organization in the world. The foundation's work ranges from providing vaccines to prevent childhood diseases to cutting edge research to improve agricultural yields and prevent malaria, among other things. Though, philanthropy is philanthropy and there is no big and small in that world as anybody who is doing it genuinely is a unique entity - you only need to travel outside US, Western Europe and few other developed nations to understand how desperately it is needed in rest of the world. Still, I believe that this particular foundation is very visionary in all its pursuits and focus areas if you follow them closely. Today, I was pleasantly surprised when I was contacted by Fenton communications who handles communications for them to write about this interesting press release on my blog. This is the bare minimum I can do as I understand the importance of what they are trying to accomplish - I have also been fortunate to work with few philanthropic organizations and I understand how valuable this concept can be for them.

The Bill and Melinda Gates Foundation is funding a new digital-media hub call ViewChange.org. The hub will use semantic technology to create a platform that combines the video sharing power of YouTube with the open information of Wikipedia and the mission of your favorite advocacy organization.

ViewChange.org is being created by a social change organization, making it one of the first time a non-profit is on the leading edge when it comes to technological innovation. They’re partnering with Zemanta, FreeBase and OpenCalais - the three well known companies in the semantics world.

Actor Danny Glover announced the launch of the project today via email and in a video.





ViewChange.org is using the power of semantic technology to make videos, articles, blogs, and actions readily available to people working in global development. While watching high-impact video stories on the site, viewers can choose to dig deeper by exploring up-to-date details on which organizations are involved, links to related content, and lists of relevant actions they can take.

For example, imagine you are watching a short documentary about clean water issues in India. As the video plays, adjacent windows will dynamically generate links to actions and media directly related to each scene. These could include organizations involved in clean water and sanitation, action campaigns related to water issues, relevant videos from YouTube, articles from research organizations, and the latest updates from news services and blogs.

This is a great cause and it will be immense help to anybody who is doing anything noble for any society. Please spread the word in any way you can.

Thursday, February 25, 2010

Where does my money go? A begining for an open government in US, UK and rest of the world

Once James Madison, the political philospher and the fourth president of America, said "If men were angels then no government would be necessary." Though, there are no clear historical records available about the first official government, democracy or parliament, despite various claims by few old civilizations, but I guess that men figured out much before James Madison that they can never be angels and they will always need a goverment to live happily and prosper. I am sure people just didn't want any government but also hoped and craved for a smarter, open and transparent government. But it seems nobody could define it clearly in last so many generations what openness and transparency really means for a government. The good news is that all of it is changing! Surprisingly, for the first time "data" is taking the lead in defining an open government - maybe because it is measurable and never lies.

The two big initiatives, data.gov and data.gov.uk (still in Beta), were launched by US and UK government respectively in May 2009 and January 2010. Infact, Prime minister Gordon Brown of UK asked Berners-Lee to look at access to government data in June, after Barack Obama's administration launched an open source data site. Well, UK already ranks number three in the OECD (Organisation for Economic Cooperation and Development) study behind Austria and Portugal in the sophistication of its e-services so making it data available was the next logical step. At high level, the goals of both the initiatives are same as they want to make the government data available online to general public for improved access; creative use of that data outside the walls of the government; public participation, collaboration and feedback; identify unexpected and insightful data relationships - insights that would normally take several decades and hundreds or thousands of brilliant socialist scientists, statisticians, psychologists, focus groups and public policy experts to simply suspect. Both these initiatives are great because of the intention behind them but lets take a closer look at the state of the initiatives.


The data.gov started with forty seven data sets but already has thousands of datasets from eighty-one US agencies. It links directly to data files in various formats including CSV, XML, Excel, and KML. A lot seems to be lacking though:
  • It makes little effort to highlight or promote any projects that uses the data from the site
  • The focus is more on a repository
  • What you do with the data is not very clear
  • The website needs lot of work in terms of clarity and user experience
  • It is still not developer friendly and needs to develop an ecosystem
  • There's no basic demographic data like population from the Census Bureau
  • Browse and search functionality seems to be missing
The Sunlight labs, a DC based non-profit organization, is working on some projects to take advantage of this data but it seems you need to have larger developer community doing the same. I have also seen some good applications from the team of  James Hendler of Rensselaer Polytechnic Institute, USA. His team is converting the data sets into RDF and  taking advantage of semantic technolgies to build few applications. For e.g:
  • One of the application is about the amount of money received by the government for corporate and personal income taxes projecte through 2014 - click on the link to access it. 
  •  If you want to know how knowledgeable is your state - click on the link to access it.

I am still not sure why Linked data (Semantic Technology) approach was not taken from the begining. Overall, data.gov in US has still a long way to go before its goals are met. Ideally, it will be great to see more impressive applications which uses data from different sources and gives you an insight about a specific problem. Nevertheless, it is moving forward - it is also understandable that managing and simplifying the process of publishing humungous data from so many agencies is a herculean task.

Now, when I look at data.gov.uk then I have to say that I am just simply impressed considering the progress they have made in six-seven months. Kudos to the team, along with Sir Tim Berners Lee, who has been working on it. They just used the semantic technology, basically linked data, approach from the begining. It also has a modern design with a very developer friendly approach. Combining data and creating mashups from different sources in this context is not an easy task but semantic technologies have made it possible. Overall, data.gov.uk's approach is simple and clear - they have used open standards, open source and open data. The website has quality and elegance written all over it even though it is in beta. They also need to figure out many things like modelling various datasets behind the scenes, encourage more participation and many other things you can think of in a project of this complexity. But the results are showing! The top ten application as rated in this telegraph article are impressive. You can see the screen shot of  one of the application called "Where does my money go." Or click here to access the prototype.







 If you are really keen to go deeper in the approach of UK govenment in this implementation then I will encourage you to read this document - Putting the frontline first:Smarter Government. UK might have followed US, Australia, New Zealand in implementing open and smarter government but it seems at least now that their template will be followed by rest of the world. It will be interesting to see if/when countries like India, China and Russia will follow this trend!

"This is very much the beginning. Hopefully, this is the tip of the iceberg. There is a whole lot more to do." But what a beginning! These were the words of Sir Tim Berners Lee when he launched the beta site for data.gov.uk.

Monday, February 22, 2010

After Best Buy, now Tesco adopts Semantic Technology!

It wasn't longtime back when I wrote how Best Buy is using Semantic technology to define a new trend. Now, it is Tesco , largest British retailer, who is following similar steps. Simply put, Tesco is to UK the same way Walmart is to US. Though, both of them get into each other's territory as it was almost an year back when Tesco launched Fresh and Easy,a chain of 10,000 square foot convenience stores in US to compete with Walmart.



Tesco has also been known as a company with a unique ability to manage vast reams of data and translate it into sales. It also uses information gathered from Dunnhumby, a British data mining firm of which it has majority control, to manage every aspect of its business, from creating new shop formats to arranging store layouts to developing private-label products and targeted sales promotions.

Tesco has always known to be pioneer in e-commerce and it is one of the most visited online supermarket sites in UK. They have started experimenting with RDFa in their website which seems like a logical next step. I won't be surprised if they adapt Goodrelations ontology also in near future.

Thursday, February 18, 2010

Cognition Technologies to power Microsoft's Bing now!

According to this news, Bing will also be powered by Cognition Technologies as Microsoft has licensed its technology for its search application. It is an interesting news because Congnition Technologies was always compared with Powerset who Microsoft acquired for $100 million plus in June of 2008. It is not replacing Powerset technology but adding more power to its capabilities. What is the relevance behind this non-exclusive licensing?

Though, Bing has made lot of improvements in the last year, to become more relevant in the search space and grab more share of the search market, it still needed to add more semantic capabilities in its arsenal. Bing is perceived to be good in following things:
  • Clean interface and a good user experience
  • Excellent information aggregation capability
  • Extracting concepts and summarization
  • Very useful for advanced search on Wikipedia and Freebase


Precision and Recall (the two criteria to measure the accuracy/relevancy of search results) is always a debatable topic if you ask any search company. Eveybody claims different results! As far as capability to undersatnd semantics/meaning of query is concerned, that is also debatable as you get mixed results if you test different queries with different search engines. Different techniques are used to really understand the meaning of a query by all semantic engines. But in case of Google, it gets its context from the humungous amount of data it indexes - they are processing so much data that they have lot of context around things like acronyms etc.. Suddenly, the search engine seems smart, like it achieved that semantic understanding, but it hasn't really. There have been instances where I have seen google nailing it quite accurately! So what is left to measure the effectiveness of search engine? Probably, this is the reason, the site traffic ends up becoming the only crietria for effectiveness or success. But, it takes years to build traffic and every improvement in the search engine counts.

I believe that Congition's real value propostion in this licensing is its advanced semantic map (think of it as a combination of dictionary, thesaurus, ontology etc.) which has millions of semantic connections that are comprised of semantic contexts, meaning representations, taxonomy and word meaning distinctions. Bing should benefit from this in making its query parsing more powerful and in improving relevance!

Tuesday, February 9, 2010

New Patent from IBM for Semantic Web!

IBM has always been a leader in patent filing. Do you know that just in 2008, it filed more than 4000 patents which is three times more than its nearest rival HP. Recently, IBM filed a patent to improve traditional tag clouds by using semantic technology. Basically, the idea is that since tags are single words and users can't have description and context with a tag, it is very limiting and value of tag diminishes as the tag space grows. For e.g. picture tagged as "dog" will not show up when the user searches for content associated with tag "puppy." You can think of hundreds of similar examples. So, with the help of ontologies and associations, you can have more meaningful, descriptive and understandable tags. For example, once this tag cloud is represented in an ontology form then a "German Shepherd" can be classified as a type of dog with attributes like eye color, fur color etc. and relationships like "owned by". You can also specify that Puppy is a yound dog in this context.




The method, explained in the attached filing,  comprises of receiving a tag cloud which includes tags that hyperlink to web content. It will seperate the tag into different linguistics categories, assigning a weight to each tag, and grouping the tags into clusters, whereas tags in a cluster are associated with a context. The server will have components like linguistic analyzer, semantic domain analyzer, taxonomy builder, attribute analyzer, relationship analyzer and ontology generator. Like any onology generator, the process will be iterative in nature. As a result of this, it will eventually lead to more accurate searches for the content you are looking for.

All of it makes sense to me but I wonder if there are risks associated with companies in future, who might try to accomplish similar goals using different flavors of technology, without infringing on this patent. I will let patent lawyers figure this out in future.

Saturday, February 6, 2010

Your new Virtual Assistant on iphone: Can it deliver?

It always surprises me how little is known about Siri, the virtual assistant, outside the Semantic Technology circle which is still very small community. Intuitively, everyone understands what a virtual assistant can do or deliver but most of the people still think that these technologies are not really ready for serious use. For those of you who don't know, Siri was born out of SRI's CALO Project, the largest Artificial Intelligence project in U.S. history. (CALO stands for Cognitive Assistant that Learns and Organizes). Made possible by a $150 million DARPA (Defense Advanced Research Projects Agency) investment, the CALO Project included 25 research organizations and institutions and spanned 5 years. Siri is bringing the benefits of this technology to the public in the first mainstream consumer application of a virtual personal assistant. The Siri application is just released for iphones and you can download it for free.


Siri - The Personal Assistant in your Phone from Tom Gruber on Vimeo.

It uses Semantic Technology for intelligent mash-ups that automatically make connections, takes action and communicates information based on dimensions such as personal data, theme or task awareness, time and location awareness much the same way a real live virtual assistant working the Internet could. It is almost like a very personalized semantic search focused on concierge-oriented tasks but can also take voice as input.

I downloaded the application on my iphone and it did manage to surprise me pleasantly. This technology has come a long way! I did face few issues as sometimes it struggled to understand what I am trying to say. Actually, it uses Nuance technology for voice recognition.I don't know how it will work outside United States - basically, lets say if you are in India, China or Russia then can this virtual assistant deliver the same help as it can do in US?

All of us need virtual assistant at some point in our professional and personal life - more when we think that time spent on some tasks is not worth our time. Ofcourse, we won't mind as long as this virtual assistant is working free for us and giving us some extra input. Will we trust it enough to do high-value transactions just based on its advice? Maybe buying a movie ticket is not a big risk and the prices are almost standard. But beyond that? It really comes down to confidence we need to develop gradually in our virtual assistant so that we can start delegating more and more tasks to Siri or other similar virtual assistants in future. Nothing different from level of complexity of  tasks which we assign to out real life assistant - the trust needs to be earned everyday.

Release of Siri on iphone is still a very important milestone in a new era of AI and semantic technology-based applications.Can Siri build a formidable user base and become synonymous with the consumer Internet? It still has a long way to go and company will have to learn many things from its user base. Can it make enough money by just charging its affiliate network and giving the application to consumers for free? Time will tell this. I still think that the company can eventually make more money by making it more focused for business-oriented professionals. It just needs to think more about applying its technology to solve unique use cases. In the business scenario, it will still have to compete with other flavors of virtual assistants likeTimesvr, which is pronounced Time Saver. It is virtual assistant of different kind - an offshore-based online service which provides a task based (as opposed to assistant based) solution, where each task goes into a queue and is handled by whatever assistant (a real person) is available and qualified. It works surprisingly nicely and economically for few. Well, you can argue, it is not really fair to compare a top-notch AI/Semantic Tech. startup with an offshore-based service whose business model is basically labor arbitrage. But this is the reality of this flat world!You have to compete at every level and in all the fronts to make business sense!

Wednesday, February 3, 2010

Measuring Semantics!

A lot has been written and discussed about the power and benefits of semantic technology. But still when it comes down to quantifying the benefits of semantic technology, you will find very few case studies where it is measured or some kind of ROI analysis has been done. It is always good to know about this case study by Telefonica which is one of the largest fixed-line and mobile telecommunications companies in the world: third largest in terms of number of customers only behind China Mobile and Vodafone, and in the top five in market value. They used semantic technology for their tariff calculation which is a very complex acitivity for an operator of their size. They could achieve 80% reduction in working hours and 75% reduction in errors for activities related with tariff calculation. If Semantic Technology has to become mainstream in the enterprises then we will continue to need more case studies like this.

Monday, January 25, 2010

Semantic Search: Finding Stuff and Creating more Businesses in this Flat World!

If you want more good jobs then spawn more Steve Jobs" says Thomas Friedman, the author of  "The World is Flat "  in this new article in NY times. Not everyone is a genius like Steve Jobs, can put 10,000 hours or is a part of 1955 club which were the key characterstics of these successful enterpreneurs as pointed by Malcom Galdwell in his interesting book "Outliers." Not every company becomes Apple also. Well, what US really needs is more of small businesses than ever which are one of the driving force behind its economy. There are some real facts about the small businesses in US:
  • Employ just over half of the country’s private sector workforce
  • Hire 40 percent of high tech workers, such as scientists, engineers and computer workers
  • Include 52 percent home-based businesses and two percent franchises
  • Represent 97.3 percent of all the exporters of goods
  • Represent 99.7 percent of all employer firms
  • Generate a majority of the innovations that come from United States companies





We need more enterpeneurs than ever who can identify opportunities worldwide and develop it into profitable ventures. Where do you start? How do you get the information? How do I know about the gaps in the markets for particular products and services? What should be the focus area? Has it been done before? Who can partner with me? And there are so many questions you can think of. I undertand that you don't start every business just by searching on the web as there are other important things like personal contacts, capital, your network, your own experience, trade associations etc. etc.. But the search on the web is increasingly becoming the major starting point for many of these activities. It is more relevant than ever to do the research because everything you want to do can be outsourced; can be imported; maybe already exists somewhere; or demand is going to go away and you are blissfully unaware. Not that it is easy to figure this out but atleast we should have more resources than just the big three search engines - Google, Yahoo and Bing.

Try to find or research that information on these engines and you will know it is so hard. All three of them have done good jobs in the last many years but they can't continue to be all things to all people in all the contexts. Too much of emphasis has been on ranking also. I personally like google but it definetely falls short as far as exploratory and interactivity aspect is concerned. Bing, which calls itself decision engine, has shown some very good improvements in last one year. Also, somehow the big three have ended up promoting a marketing view of search on the web and they thrive on the tension created between SEO consultants/advertisers and them. It seems that media and analyst community are also too concerned with glorifying every percentage gain by Bing over Yahoo as you can see in this news and may others. Maybe, "its just not about search, its about business" - probably, Michael Corleane (from movie Godfather) would have said if he worked for Google in this era.

In general, I have seen a very narrow view of the search on the web from a school of thought which believes that whatever could be done in search will be just confined to these three as far as web is concerned - they think that new entrants will make some noise initially and then go away quietly. I completey disagree. In my opinion, it is no different from the view in eighteenth century when there was a school of thought which believed that whatever human beings can think of or can invent has already been done - there is no scope of anything new. Sounds ridiculous if you evaluate the progress mankind has made since then!

A lot has been written about the benefits about the Semantic Search and how it is better than the key-word based search. I have also written in one of my previous article about "semantics" in semantic search. Recently, Seth Grimes also compiled a very good article about types of semantic search. So I am not going to talk about what semantic search is but more about the opportunities for the new breed of semantic search engines.

Occassionally, you do see articles like "semantic search engines which will change the world which lists new breed of semantic search engines - some call them google killers. I always wonder why these semantic search have not been able to make measurable impact yet. Some of these engines have very good technology also though the list doesn't include many others which are out there. The internal details of most of these engines are still proprietary and they combine a natural-language processing with various flavours of semantics. One more semantic engine which I like is TipTop which mines Twitter database and does sentiment analysis also. But there are more than six Twitter search-based products in the market as you can see in this review - product from Tiptop is not even mentioned here while the other products may not be doing semantic search. Now, even Bing has a product which searches Twitter. So what can be the next step for these new semantic search engines?

In my opinion, the big issue is the lack of focus for some of these "semantic search" startups and their obsession to boil the ocean. Many of them waste lot of time comparing themselves with google. Some of them are also trying to do very similar things. Market will continue to get crowded with semantic search engines in next few years but there is a risk that many companies with excellent technologies will get lost in the crowd. Very soon, every search engine will start calling itself a semantic search engine, the same way every SAAS offering is a Cloud offering nowadays. The other issue is lack of understanding from business and users about what "semanticity" in search engines really means. Ideally, Semantic search engines should have some aspects of natural language, contextual (focus on disambiguating queries), ontologies and reasoning. The hard part is always developing the understanding how much of work is required to customize the technology to incorporate all these aspects so that it is relevant for a particular domain.

There is a great opportunity for these new breed of semantic search engines to rethink about their strategy. They should also not try to be all things to all people. They really need to carve out a space for themselves in specific segments. If they go after enterprises, they will face stiff competion from the big three in the enterprise - Sharepoint/Fast, Autonomy and Endeca who have customers in hundreds and have evolved over the years. Among them, Autonomy has done a great job in e-discovery space by following a vertical strategy. Even companies like Marklogic with its powerful XML server can solve many search related problems for unstructured content. In my opinion, semantic search startups can always continue to tweek their algorithms and enhance semantics but simultaneously the focus should be on verticalization, branding, strategy, positioning etc.. Application-centric or vertical strategy will be better for them as opposed to platform-centric strategy.  They can also think about merging what is there on the web with the enterprise data/content to give extended BI inxights which is still a new area to develop powerful applications. Though I can count few small companies in this area also and even big ones like Business Objects after acquistion of Inxight. Still , there is ample oportunity to think creatively and develop useful analytics-based applications for enterprise.

Recently, Financial times launched Newssift (still in Beta) which is a business news semantic search engine which indexes thousands of news sources worldwide. They have used Endeca technology for faceted search and sentiment analysis is provided by Lexalytics. It can be a useful  tool but more can be done in this area. If any of these new breed of semantic search engines can correlate data from historical sources and the one which is acquired from multiple sources to: identify patterns and indicate important events then it can be a killer application on Wall Street. Unstructured data is already being leveraged in electronic trading strategies but the adaption is not so fast. Generating alpha from the stream of unstructured data is not an easy task but a great opportunity.

Another set of innovative companies I want to mention in this context are Bintro ,Trialx and Echo Nest. Bintro matches you to what you are looking for like employment, partnerships, investment and joint ventures. TrialX is a free service that matches participants to relevant clinical trials based on their personal health information. TrialX uses a comprehensive database of 25,000+ clinical trials approved by the Food and Drug Administration (FDA) in the United States. Echo Nest helps you find your audience with targetted music production. It claims that it can understand every music writer on the web (bloggers, review sites etc.) and helps you find the writer most likely to review your music. Again, very useful way to leverage semantic technology to solve problems in a particular domain!

In the end, I believe that there is a scope for hundreds of similar applications for semantic search engines and they can happily coexist with Google, Yahoo and Bing. We will also see very interesting changes once the data web evolves and semantic markup starts becoming more prevalent.
 


Wednesday, December 23, 2009

Mobile Internet report: some interesting insights

In this age of information overload, it is very easy to miss this comprehensive report by Mary Meeker of Morgan Stanley. It is a solid and very detailed work! Basically, the theme of the report is about the growth of mobile which will be much bigger than any of us can imagine.  This new technology cycle is compared with what Windows 3 did for the PC in 1990 and Netscape browser did for desktop internet in 1995. And how the mobile internet has potential to create/destroy more wealth than prior computing cycles!




The report is well supported by solid research and numbers to back the analysis and forecasts. It took me some time to go through it but it was worth it as it gave me some interesting insights. In general, if it comes to mobile, we don't need to read a report to predict the future - we just have to look around and see what/how everyone is using these smart devices. I will skip the obvious like the spark created by iphone, 100k+ apps etc. but there are still few interesting facts such as:
  • 5 trends converging - 3G + social networking + video+ VOIP + impressive mobile devices
  • 57 m iphone +163% growth, 125k developers worldwide, 2B+ downloads
  • Many consumers are finding that their online usage rises dramatically when they have 24*7 mobile access to cloud based stuff
  • Growth/monetization roadmap for mobile is provided by Japan
  • Physical products are gaining share in mobile ecommerce - 20% in Japan but less than 1% in US
  • China leads world in virtual good monetization
  • Google (Android) has the best chance to to serve as a more open counter-balance to apple. Opera leading transformation of mobile browsers
  • Professional content repository is still open - amazon, iphone, netflix, hulu?
  • Open mobile web potentially more attractive to developers - Apple may believe its platform management is prudent way to ensure high quality content but it also runs the risk of stifling development and innovation
  • Clouds will be providing real infrastructure for mobile applications
  • Location aware ads will be better targetted
  • People are more willing to pay for content on mobile than desktop
  • AT&T - 50* mobile traffic growth in last three years despite only 40% subscriber growth over the same period
  • Shift in type of applications from games, lifestyle, utilities, enterptainment etc. to business oriented though among the top 100 applications on iphone, business oriented applications are just five
  • Daily usage for productivity based applications is probably less than 5-6 %
I wish there was little more information about the user profile but I still consider it the best work by Mary Meeker after her famous Internet report in 1994. Since new business models are often created during technology changes so there is an opportunity for eveyone inlcuding Semantic Technology to take advantage of this booming market because:
  •  Browser still remains the weakest link in these devices. There is no integrated/personalized experience.
  • The principles of end user interaction have not been established
  • Semantic search will become more relevant as users will have less tolerance for too many results.
  • Semantic and location specific advertising will have a role to play
  • Cloud computing will be one of the biggest enabler of writing semantic applications for mobile devices as computing power will no longer be an issue for even smaller shops
  • Semantic web powered commerce can also be triggered
  • Agent technology should find better business cases
Obviously, any analyst report will not be specific about the type of future applications/services which will reside in these mobile devices. That job is for innovators to figure out!
Reblog this post [with Zemanta]

Thursday, December 3, 2009

Semantic Technology and iPhone!

Recently, I came across this intersting article about "Inside the Apple Economy"  - http://bit.ly/2FnBPt . Please read about some of the unknown names who have made a killing in last two years just by exploting this new ecosystem. I believe that Semantic web, linked data and Semantic technologies in general for the enteprise will also be creating an ecosystem which will be exploited by many people. But I only realized this Thanksgiving weekend during a NY city visit along with few friends of mine that there can be a corelation between iphone and Semantic Technology.

We had not made any dinner reservations as we were not sure about timings and were confident of finding an interesting cusine in the NY city, practically a paradise for your palate, which would work for everyone in the group. I was also counting on my Zagat membership which I have always found to be a very trusted source for restaurant reviews. We soon realized that everytime we agreed on a particular cuisine and restaurant after looking up ratings on Zagat on my iphone, the restuarant was always few miles away and it was very difficult to find two cabs on a crowded weekend. So we decided to use "Around Me" - an interesting application on iphone which can list critical services around you — banks, coffee shops, bars, gas stations, hospitals, movie theaters, restaurants and so on. Using geolocation, the app orders each service by its proximity to you — how many yards away — and, like other apps aimed at the traveler, maps out a route from here to there, if requested.





We kept looking up restaurants in "around me" but we couldn't find ratings and reviews for those restaurants. So I ended up toggling between these two applications, during our long walk, for a while till everybody agreed on a Turkish cusine.

I was wondering if these two services shared there data using Semantic Technology, basically Linked data, then life would have been so simple. All I had to do was to ask - "Give me all restaurants with minimum 4 star rating within 400 yards of my position." Hopefully, some food for thought for enterpreneurs who want to exploit the iphone economy. As somebody rightly said, "The food that enters the mind must be watched as closely as the food that enters the body."

Tuesday, November 24, 2009

Finally, a National Semantic Technology Roadmap!

Probably, you might have assumed it that I am talking about the US but actually it is envisioned by the Malaysian government on their National Information technology Council (NITC) web site. More details would have helped but I will still give lot of credit to Malaysian government to start thinking in this direction and having the courage to recognize Semantic Technology on their official technology web site under the Ministry of Science, Technology and Innovation. Among other great things, the Malaysian governement also has the credit of creating one of the tallest towers in the world called Petrona Twin Towers. You have to be there to believe it and these towers have also become the epitome of astonishing growth in the last two decades in Malaysia.


While in US, sometimes, we still have to run sessions in conferences like - "Is Semantic Technology for Real? "

Friday, November 20, 2009

The Death of Taxonomies! Is there a role for Semantic Technologies?

Congressional Tweet Word Cloud (research in pr...Image by justgrimes via Flickr
There is a very nice article by CMSwatch about how the practice of building taxonomies is dying - this discussion is more oriented towards content/document-centric world. It basically highlights few trends:
  • Since social computing is being widely adapted by companies so there is no fixed way of categorizing things i.e days of single, heirarchical taxonomies are far behind. People will tag content in any way they want.
  • You should be able to get to information the way you want, which may be different from your colleague's approach.
  • Text mining and auto-tagging software is gradually improving, and extracted terms can be applied as metadata. Metadata needs to be very fluid - cloud like. See the fig. above
  • Metadata architects should really understand the domain
I believe there are some solid arguments as I have seen this trend myself. I have only few things to add:
  • What will be the future of existing large taxonomies?
  • Many companies have already invested in more than one content management system like Sharepoint, Documentum or Opentext. There is a good opportunity for semantic web technologies, basically OWL and RDF, which can unify the taxonomies of different content management systems and provide a single data model to retrieve the content through.
  • We should always be very clear about what the word "Metadata" means in any context. It means different things to different people and has been overused. A metadata in a document-centric world is entirely different from say a metadata in a word processing program.
  • Semantic Web is really an umbrella language for metadata. More thought needs to be given to how this new trend of adding tags/metadata on the fly can be leveraged to add more value to the semantic web or linked data cloud.
Reblog this post [with Zemanta]

Thursday, November 12, 2009

Semantics, Metal Reserves and how long will they last!

The picture below tells you about the state of planet earth's metal reserves!



Basically, this diagram says that there are fifty eight years left when all the metal resources will be exhausted  if the world continues to consume at today's rate. There is nothing to worry about because I am sure that geologists on this planet will figure something out much before that.  I am saying this because many of us worried about the state of oil reserves also in last two decades after many pessimistic predictions - more than half of the time people were "Crying Wolf." In any case, the point I want to talk more about is the role of semantic technology in helping the geologists around the world.

I came across an interesting paper in this context. The key points explaining why semantics is becoming important in geology are:
  • Geology, perhaps more than any other science, has advanced with the aid of pictures. The pictures, which are geological maps shows the distribution of different rock types. The current efforts to improve them have spawned a global e-science initiative to review the semantics of geology.

  • One example of issue of semantics in context of geology is - The US department of agriculture and Forest service and the British Columbia Ministry of Lands, Parks and Environment will describe and classify the same geological phenomenan in a completely different way

  • When working at relatively large scales, geologists are always dependent on data and information gathered and reported by other geologists regarding smaller scale features. Semantics is very critical here also

  • One of the consistent problems in geology is the omni-present "map boundary fault" - an apparent geological continuity along the border of adjoining maps which is nothing more than difference in nomenclature or semantics

  • Research is being done to prototype vocabularies by developing web services which can be used to store, compare, and rank on similarity, decsriptions of models (concepts) and instances (physical entitites and events) of mineral deposits, landslides and landslide hazards.
To go through the full research papaer please go here.
Reblog this post [with Zemanta]