Social Networks & Big Data - Testing business hypothesis On the Cloud

Business innovation reflects in the form of new products and services, re-engineered / revamped business processes, new delivery mechanisms and sometimes a very different engagement model. One common factor that defines the mood of teams who role out these innovations, is the fear of unknown. Most innovations ( as opposed to changes) typically dabble in the paradigm of rolling out the untested and the unproved with absolutely no clue around how the end consumers might react. Enterprises typically follow a prototype and pilot model before a Big Bang roll-out to counter most these issues. Some forward looking organizations have adventured tools such as market predictions to gauge the reaction of people to new concepts ( could be a new product or a new HR policy) with the employees (employees are end consumers too).

To gauge reactions to a concept post ideation, one needs participants who are willing to put their mind to it and react. We need a platform that can capture these reactions and tools to analyse these reactions. Social networking sites provide this platform and the needed digital footfall , big data technology helps analyse the reactions. Business hypothesis can be tested on social networking sites to take a call on how well the assumptions measure up.

At the CIO round table on the evening of the first day of TechEd 2013 in Bangalore, one of the CIOs brought this point up and we were wondering, where do the Microsoft's products stand on Big Data. To begin with Apache Hadoop is made available as a service called HDInsight on Azure Cloud. What this means is that VMs on Azure cloud support full functionality of HDFS and MPP execution of MapReduce. HDInsight service also has ODBC connectors to enable analysts and BI professional leverage the simplicity of usage of Excel and the power of analytics + reporting now available in MS Sql. As far as I know, this is possible for structured data. To use unstructured data, PigLatin scripts will need to be run on the VMs. One can get visualisation of results going in excel ( from what I gather - I have not tried this or seen this as a demo yet). If this works out fine, one could potentially use Excel, to analyse the end results with the power of dissecting information that excel so very well provides. I am definitively sure excel empowering users to fire  HIVE queries  from it's interface.

There were slides shown pointing to Pig, Hive, Flume, Pegasus, Sqoop and Mahout  implementations provided for on the HDInsight services on Windows Azure. This however needs to be tested for ease of usage and the actual power this implementation puts in the hands of the analyst and the business users. Microsoft has enriched HDInsights service with the ability to control accesses and privileges using Active Directory. For enterprises that use active directory, this wheel needs no reinvention for Big Data setup, if done on Azure.  HDInsight does not dilute any of Apache Hadoop capabilities ( hence running as a Azure service you get the same high availability and reliability) with the HDInsight clusters managed thru System Center 2012.

HDInsights service on Azure cloud, with the simplicity of integrating enterprise data warehouses and data marts with unstructured data in HDFS and with the simplicity of using excel and other in-house BI tools to query and visualise results; enables enterprises to empower their business users to start testing their hypothesis and make better informed choices and increase the probability of success on their innovations.

A Social Network for Enterprise Applications

Put together a few day-dreamers, high skilled innocent techies who can be taken on a fantasy ride, some chilled beer and the beautiful spring evening of Pune city; what you end up with is a crazy concept and a design document written on tissue papers and thick-paper coasters, for the wackiest  and unimplementable idea one could ever come up with. If only one could get some venture capitalist or seed funders on the same table; the next one year could be a well funded sabbatical for most of us around that table. Two fantastic experiences I still savour in from last evening. The build-up of the wacky concept and the associated technical design document (if you can call it that) and the green tea with fresh ginger to end the night!

The big idea - a social network for enterprise applications. A space where different applications on heterogeneous platforms, from different enterprises could reach out to each other on a need basis or otherwise, to befriend one another, share and exchange both data and intelligence for mutual benefit and in a secure manner without violating privacy laws. The applications decide their privacy settings and create trust groups with trust levels of various degrees. I guess by now you guys would have gauged the height and depth of lunacy that prevailed around that table. I state again, there was just beer on the table, nothing else!

Straight thru collaboration between enterprises has always been looked at as a fictional idea. Enterprise IT Policies, data governance mandates and zero tolerance towards alien incoming traffic into enterprise network have prevented straight through collaboration. Management will power always gave way to the logistic nightmare of establishing trust relationships between enterprise networks, working thru the mindsets of enterprise IT teams, enterprise risk teams and IT auditors. Cloud and particularly Azure services seem to have unknowingly offer a solution in this space.

Some of the elements needed for straight thru collaboration between enterprises for consumer's benefit :

1. Authenticating consumers , employees and partners for the scope of the collaboration in focus.
2. Persisting session data across multiple systems belonging to different enterprises and  enabling these systems to access this data. 
3. Persisting Master Data of the consumer to be accessed by collaborating enterprises,  their respective processing partners and the consumer himself, subjected to data access rules of these organizations.
4. Data transformation outside of enterprise networks as needed by enterprise apps, partner apps and consumer's personal apps enabling these systems to understand the data against the backdrop of diversity in codification of master data and other data items. 
5. A very effective 2 phase commit of transactions across collaborative systems, one that is non-intrusive.

There are a lot more demands that STC ( straight through collaboration) will place, so the list if not truly a complete representation of all the challenges that would need to be met.

Azure services such as :

• Identity (AD on the cloud)
• Caching (App Caching and content caching on the cloud)
• Business Analytics ( BI and Hadoop Services)
• Messaging services ( Queues and Service bus on the cloud)
• Data Services ( Sql Db, Tables and Blobs on the cloud)

can be meaningfully stitched together into a solution around the scope of collaborative services enterprises want to offer to their consumer. As enterprises start authenticating with the Aadhaar framework, ability to collaborating on a non-intrusive mode will get a further boost. I hope enterprises see merit in authenticating not just consumers but all actors needed to get collaboration going, being authenticated by credible third party services.

Ability to transform data, run transactions to their logical end with asynchrony as a key element of the collaboration fabric, with 2 phase commit as the final goal across collaborating systems, would have been impossible without Azure messaging services ( queues and service bus on the cloud).

Azure business analytics services that unlocks enterprise intelligence to empower it further with big data outputs, will add the winning elements of context, timing and relevance to the collaboration services being delivered to the cloud. Getting these three elements right for a single large enterprise is itself a complex activity, getting it right across multiple collaborating partners is a near impossible task without the ability to transform and interpret data on the cloud. This is the reason why we need multiple services as bullet-ed above to be stitched together into a solution. 

The other issue that crops up is that enterprises would not like to remain wedded to it's partners to get STC going for delivering value to the end consumer. The collaboration arrangements could run out of relevance with passage of time , sometimes under 3 hours ( example: credit card companies wanting to collaborate for discounts with in-mall merchants for 2 hours). We need a framework where enterprises could negotiate online the terms and conditions of collaboration, do an online hand shake and get the collaboration going. This is very similar to the scenario where we allow apps to connect to our facebook profile, accept friends invitations on facebook and keep tuning our privacy settings and keep categorizing our friends to get a meaningful discussion / collaboration going. A similar framework will need to be evolved for enterprises to hook-on and hook-off with each other for various transactions that need collaboration between their systems,  their partners systems and consumer's personal apps and devices. A social network for enterprise applications!

What started as a fun fantasy discussion, quickly spiraled into some serious paper napkin drawings of architecture design and some specifications bullet-ed down behind a few thick paper coasters   We were not completely insane, so there were some assumptions that were made. We assumed systems would be intelligent enough to decide whom to befriend and decide on the degree of trust they would like to establish with the system seeking to connect. We decided to tackle the implement-ability of such a scenario on a later date, and focus for now on other issues to get this concept moving forward! 

I need to mention here why I picked Azure. The way Microsoft is approaching the cloud computing paradigm, empowering it with platform services that enterprises need, putting in frameworks for extending these to mobile clients and keeping an open mind on heterogeneity of personal devices and strong preferences of the enterprises to have the flexibility of the cloud with the comfort of implementing their corporate policies. Most of Azure services could literally be an extension of on-premise digital assets as enterprises evolve their view around the right model of governance , security and  management for leveraging the power of the cloud.


written for :

2013 : The year of Zero Tolerance

Its very apparent. It stares you in the face, you cannot miss it. If there was one thing that made an impact in 2012, that was technology - both the device and the software. The mobile devices with social networking apps on them were the kings and queens of 2012. I went into a research mode to see if there was an equivalent ubiquitous thing in the past that made a huge impact. Not surprisingly, what came up was a weapon. The AK-47 assault rifle.This was one rifle that is so light and efficient that sadly it can be used by children with minimal learning. This is the most abundantly mass produced rifle and has changed the way wars are fought!

The same can be said of mobile device+social networking apps in 2012. This weapon has given huge strength to the common man across cultures and geographies to raise voice over what they feel is undesirable, unethical and wrong. Mature democracies to dictatorship regimes have belted down under the force of these voices fueled by the mobile device+social networking app weapon. Occupy wall street to Arab Spring and now in our own backyard in India. Governments are seeing the absolute non-tolerant citizen who might not be voting but is definetly out there to tear down structures that don't find relevance in today's times. Not very sure if the mobile devices are feeding on the need to be socially connected on the move or the apps are feeding on the device proliferation. Computerization of technology is happening at an amazing unprecedented phase, that most CIOs could only wish for their projects in the the enterprise. 

One needs to take cognizance of the fact that the attitudes are changing. Absolute intolerance to mediocricity, blatant abuse of power and flouting of authority to crub voice of dissent will only grow in strength. I foresee 2013 to be a year where people in public service will come under scrutiny and will suffer lack of credibility and respect. The government and hence the police will find themselves completely lacking in credibility. Socilally connected mobile individuals armed with the power of convergence of video  audio and data over the internet will be putting extreme pressure on every social structure - government, police, tradition, religion and family to align to their way of life and find relevance in a positive manner as they live their lives. You will see people in position and power react rather clumsily at first as you have seen in the last two weeks and hopefully, they will adapt and align to serve rather than dictate.

Sooner or later the citizen is going to bring the same attitude to work on monday morning and the affect of this will be felt on the office floors all thru till friday. Enterprises who dont engage on a connected platform with their employees, will find employees engaging outside of the cortporate structure on corporate issues, thru their 3G connected cellphones armed with facebook, twitter and google+. Many corporates today are still heirarchical structures of power when it comes to most things that significantly impact organizational fabric. This is where employees might draw parallels with what ius happening in our society outside the corporate world. Mediocricity and inefficiencies might come under fire within corporates.

Connected employees are much more potent than unions. Most organizations who engage employees effectively, will find a greater appreciation coming their way. The better life inside the security of the corporate building as compared to the life on the street, will only go improve and increase employee engagement scores and drive higher employee loyalty.  meaningful engagement within the connected corporate that not just contributes to the employee but acknowledges employees contribution by keeping them in the decision making process, will go a long way. This is not only the right thing to do, this also keeps the spill over of the anger and frustration from entering the office environment. One such organization I know that dos this effectively is Indigo.     

CIOs have a huge role to play here. They can be the people who can drive the ground rules of connected enterprise, not as an order taker but as a direction setter. Can a CIO stand up and hold their own when some of the ground rules come under pressure from HR, Finance, Legal  and Brand ? many a CIOs will be told that its not their business to set the ground rules of engagement on the platform. How the CIOs react in those situations and conduct themselves, will differentiate the direction setters from the order takers. Direction setting is about painting a strategy and executing on the same with the intent of building long lasting structures on solid foundations. Each CIO knows what the best foundation for their organization is. Jan 2nd 2013 , Wednesday - most people will come into office with a mixed feeling of having mourned the brave-heart, wondering what beholds as future for our girls and women back home, and definetly some excitement around the fact that the old year has gone by and maybe the new year will bring in some hope and cheer. Are the enterprises ready to take on the challenges that will start walking their way starting this Wednesday?