Social Networks & Big Data - Testing business hypothesis On the Cloud

Business innovation reflects in the form of new products and services, re-engineered / revamped business processes, new delivery mechanisms and sometimes a very different engagement model. One common factor that defines the mood of teams who role out these innovations, is the fear of unknown. Most innovations ( as opposed to changes) typically dabble in the paradigm of rolling out the untested and the unproved with absolutely no clue around how the end consumers might react. Enterprises typically follow a prototype and pilot model before a Big Bang roll-out to counter most these issues. Some forward looking organizations have adventured tools such as market predictions to gauge the reaction of people to new concepts ( could be a new product or a new HR policy) with the employees (employees are end consumers too).

To gauge reactions to a concept post ideation, one needs participants who are willing to put their mind to it and react. We need a platform that can capture these reactions and tools to analyse these reactions. Social networking sites provide this platform and the needed digital footfall , big data technology helps analyse the reactions. Business hypothesis can be tested on social networking sites to take a call on how well the assumptions measure up.

At the CIO round table on the evening of the first day of TechEd 2013 in Bangalore, one of the CIOs brought this point up and we were wondering, where do the Microsoft's products stand on Big Data. To begin with Apache Hadoop is made available as a service called HDInsight on Azure Cloud. What this means is that VMs on Azure cloud support full functionality of HDFS and MPP execution of MapReduce. HDInsight service also has ODBC connectors to enable analysts and BI professional leverage the simplicity of usage of Excel and the power of analytics + reporting now available in MS Sql. As far as I know, this is possible for structured data. To use unstructured data, PigLatin scripts will need to be run on the VMs. One can get visualisation of results going in excel ( from what I gather - I have not tried this or seen this as a demo yet). If this works out fine, one could potentially use Excel, to analyse the end results with the power of dissecting information that excel so very well provides. I am definitively sure excel empowering users to fire  HIVE queries  from it's interface.

There were slides shown pointing to Pig, Hive, Flume, Pegasus, Sqoop and Mahout  implementations provided for on the HDInsight services on Windows Azure. This however needs to be tested for ease of usage and the actual power this implementation puts in the hands of the analyst and the business users. Microsoft has enriched HDInsights service with the ability to control accesses and privileges using Active Directory. For enterprises that use active directory, this wheel needs no reinvention for Big Data setup, if done on Azure.  HDInsight does not dilute any of Apache Hadoop capabilities ( hence running as a Azure service you get the same high availability and reliability) with the HDInsight clusters managed thru System Center 2012.

HDInsights service on Azure cloud, with the simplicity of integrating enterprise data warehouses and data marts with unstructured data in HDFS and with the simplicity of using excel and other in-house BI tools to query and visualise results; enables enterprises to empower their business users to start testing their hypothesis and make better informed choices and increase the probability of success on their innovations.