Social Networks & Big Data - Testing business hypothesis On the Cloud

Business innovation reflects in the form of new products and services, re-engineered / revamped business processes, new delivery mechanisms and sometimes a very different engagement model. One common factor that defines the mood of teams who role out these innovations, is the fear of unknown. Most innovations ( as opposed to changes) typically dabble in the paradigm of rolling out the untested and the unproved with absolutely no clue around how the end consumers might react. Enterprises typically follow a prototype and pilot model before a Big Bang roll-out to counter most these issues. Some forward looking organizations have adventured tools such as market predictions to gauge the reaction of people to new concepts ( could be a new product or a new HR policy) with the employees (employees are end consumers too).

To gauge reactions to a concept post ideation, one needs participants who are willing to put their mind to it and react. We need a platform that can capture these reactions and tools to analyse these reactions. Social networking sites provide this platform and the needed digital footfall , big data technology helps analyse the reactions. Business hypothesis can be tested on social networking sites to take a call on how well the assumptions measure up.

At the CIO round table on the evening of the first day of TechEd 2013 in Bangalore, one of the CIOs brought this point up and we were wondering, where do the Microsoft's products stand on Big Data. To begin with Apache Hadoop is made available as a service called HDInsight on Azure Cloud. What this means is that VMs on Azure cloud support full functionality of HDFS and MPP execution of MapReduce. HDInsight service also has ODBC connectors to enable analysts and BI professional leverage the simplicity of usage of Excel and the power of analytics + reporting now available in MS Sql. As far as I know, this is possible for structured data. To use unstructured data, PigLatin scripts will need to be run on the VMs. One can get visualisation of results going in excel ( from what I gather - I have not tried this or seen this as a demo yet). If this works out fine, one could potentially use Excel, to analyse the end results with the power of dissecting information that excel so very well provides. I am definitively sure excel empowering users to fire  HIVE queries  from it's interface.

There were slides shown pointing to Pig, Hive, Flume, Pegasus, Sqoop and Mahout  implementations provided for on the HDInsight services on Windows Azure. This however needs to be tested for ease of usage and the actual power this implementation puts in the hands of the analyst and the business users. Microsoft has enriched HDInsights service with the ability to control accesses and privileges using Active Directory. For enterprises that use active directory, this wheel needs no reinvention for Big Data setup, if done on Azure.  HDInsight does not dilute any of Apache Hadoop capabilities ( hence running as a Azure service you get the same high availability and reliability) with the HDInsight clusters managed thru System Center 2012.

HDInsights service on Azure cloud, with the simplicity of integrating enterprise data warehouses and data marts with unstructured data in HDFS and with the simplicity of using excel and other in-house BI tools to query and visualise results; enables enterprises to empower their business users to start testing their hypothesis and make better informed choices and increase the probability of success on their innovations.

Mobile Architectures

Enterprise applications follow certain architecture guidelines to deliver the business needs on functionality, TCO and ROI. To achieve this, trade offs made often tend to sacrifice asynchrony and flexibility in favour of assembly-lining of processes to deliver maximum bang for the buck spent and IT security. Large enterprise applications work on the philosophy of FTR ( getting things first time right). Efficiency, effectiveness and cost-control are derived from FTR theme. Any incremental and/or dramatic changes to the systems are mellowed down against the need to keep things familiar, avoid employees going thru a learning curve. Hence most changes are never dramatic, on a floor shop where most processes are simplified and assembly-lined. The pressure to avoid drastic changes is high in order to keep re-learning cycles low (to realize efficiencies from changes faster and at lower cost). In such environments, you would find IT systems architected in a fabric of synchrony when it comes to processes and events. In such environments you will find human resources avoiding the complexity of multi-tasking on mutually exclusive process threads.

Architectures for mobile apps need to deliver on different goals. These goals being asynchrony, multi-tasking and usability. Events arranged in a tight sequence will not work in the real world as the real world is a very different place when compared to the controlled environment of back-office and enterprise shop floors. This is true for sales processes as well. If you really look at the retail sales processes and the events as they unfold on the ground, they are seldom in tandem with the way the processes in the systems have sequenced events, implemented controls and designed the processes. This is one of the many reasons for change requests budgets and exceptions being sought for on mandated process. This is one of the reasons why systems have the facility to request for exceptions and an approval authority workflow for their approval.

When I look at some of the mobile apps from the enterprise that are either customer facing or sales force facing, I have found these apps tightly stitched from an enterprise operations viewpoint, backend to the front. While in reality the mobile app needs to be de-linked from the enterprise backend to avoid the trap of synchrony and the trap of irrelevance between sequence of event play-out in the real world (there are no fixed sequences here) and the fixed sequence of processes manifested thru screens and logic on enterprise applications. How often have people got into BPR exercise initiated as and when these gaps are experienced at a medium to high frequency.

Cloud plays a big role in delivering asynchrony and de-linking from backend to keep the app behaviour relevant to the way events play out in the real world. Windows Azure Mobile services provides some fantastic platforms on the cloud to make this happen.

As an approach the mobile developers should focus on usability and architect the app to handle each fundamental unit event in entirety without stitching the code to create dependency on another event's initiation and/ or closure. Enterprise IT need to provide for multiple entry points for data from mobile apps into  processes on the backend systems, without imposing the synchrony that is inherently built into the backend systems. For those who believe that this is an expensive change to make on the backend, it could be and again it may not, if one leverages the cloud queues and service bus to hold the data till all the pieces fall in place for a backend process to consume the data. This will require constant polling from enterprise network to the cloud / a mechanism for the cloud to let the enterprise network know that data is ready to be consumed.

The mobile app, writes to the cloud. Depending on the need, this could be a table (no-sql table) , cache, queue or a service bus. Build the processing logic on the cloud and not on the mobile app itself. On the cloud you would have two roles. A web role, typically the web app executes in a web role and a worker role which typically are cloud resources such as cache, queues and tables which the web app uses. The mobile app is stitched for usability to cater to various events as they play out in the real world. The processing happens on the cloud. The backend systems consume the data to process and give an output back to the cloud. The mobile app takes this output from the cloud to display on the mobile screen and wait for consumers next request, rather than initiating the next sequence for the consumer.

In the traditional development approach, a developer would sit down to create table structures and database schemas , normalise the database design, design screens, write processing logic to read and write data, logic for integration points with the enterprise database and so on. In the new normal, the developer will just focus on UI design, use Azure no-sql tables to store data. The Azure tables, create the structure depending on incoming data. The processing logic is held on the cloud as a web app in a web role. Session data persistence is managed by azure cache. If one needs to capture images or videos ( surveyor capturing data for car insurance claim processing), this can be held on the cloud CDN. Interaction with enterprise systems (for synchronous processing) can happen thru the service bus on a publish/subscribe model.

For example while shopping in a real world brick and mortar store, as I see what I like, I could take my mobile out, invoke the store's mobile app, read the bar code off the merchandise on display, attach credit card from my m-wallet and leave a delivery address and walk out. While different stores will have their own respective mobile apps, the bank could just publish a web app on the cloud, cloud storage (no-sql table) to which different apps could write credit card data, create three different themes on the service bus :

- Shopping Item theme

- Payment Approval Theme

- Item delivery theme

..and publish/subscribe to these themes depending on the context of the transaction. The store app would publish item details and price to the first theme, the bank could subscribe to it for it's records. The bank would publish credit card authorization decision to the service bus , to which apps could subscribe to. As the item is delivered to the customer, the enterprise apps of the stores could publish these details for the bank's systems to subscribe (in case there is a dispute later). The enterprise systems could have web services for interaction with the service bus on the cloud and the web app on the cloud.

Its clear from my above explanation that the app developer can create magic while being completely ignorant of the underlying enterprise systems that come into play. The developer need not worry about database design and structure and can focus only on UI and logic. The enterprise IT teams re-use the existing processing and business logic on the system by directing the data from the cloud into the right systems thru a SOA layer, they build and manage. Welcome to the new normal.

written for:

A Social Network for Enterprise Applications

Put together a few day-dreamers, high skilled innocent techies who can be taken on a fantasy ride, some chilled beer and the beautiful spring evening of Pune city; what you end up with is a crazy concept and a design document written on tissue papers and thick-paper coasters, for the wackiest  and unimplementable idea one could ever come up with. If only one could get some venture capitalist or seed funders on the same table; the next one year could be a well funded sabbatical for most of us around that table. Two fantastic experiences I still savour in from last evening. The build-up of the wacky concept and the associated technical design document (if you can call it that) and the green tea with fresh ginger to end the night!

The big idea - a social network for enterprise applications. A space where different applications on heterogeneous platforms, from different enterprises could reach out to each other on a need basis or otherwise, to befriend one another, share and exchange both data and intelligence for mutual benefit and in a secure manner without violating privacy laws. The applications decide their privacy settings and create trust groups with trust levels of various degrees. I guess by now you guys would have gauged the height and depth of lunacy that prevailed around that table. I state again, there was just beer on the table, nothing else!

Straight thru collaboration between enterprises has always been looked at as a fictional idea. Enterprise IT Policies, data governance mandates and zero tolerance towards alien incoming traffic into enterprise network have prevented straight through collaboration. Management will power always gave way to the logistic nightmare of establishing trust relationships between enterprise networks, working thru the mindsets of enterprise IT teams, enterprise risk teams and IT auditors. Cloud and particularly Azure services seem to have unknowingly offer a solution in this space.

Some of the elements needed for straight thru collaboration between enterprises for consumer's benefit :

1. Authenticating consumers , employees and partners for the scope of the collaboration in focus.
2. Persisting session data across multiple systems belonging to different enterprises and  enabling these systems to access this data. 
3. Persisting Master Data of the consumer to be accessed by collaborating enterprises,  their respective processing partners and the consumer himself, subjected to data access rules of these organizations.
4. Data transformation outside of enterprise networks as needed by enterprise apps, partner apps and consumer's personal apps enabling these systems to understand the data against the backdrop of diversity in codification of master data and other data items. 
5. A very effective 2 phase commit of transactions across collaborative systems, one that is non-intrusive.

There are a lot more demands that STC ( straight through collaboration) will place, so the list if not truly a complete representation of all the challenges that would need to be met.

Azure services such as :

• Identity (AD on the cloud)
• Caching (App Caching and content caching on the cloud)
• Business Analytics ( BI and Hadoop Services)
• Messaging services ( Queues and Service bus on the cloud)
• Data Services ( Sql Db, Tables and Blobs on the cloud)

can be meaningfully stitched together into a solution around the scope of collaborative services enterprises want to offer to their consumer. As enterprises start authenticating with the Aadhaar framework, ability to collaborating on a non-intrusive mode will get a further boost. I hope enterprises see merit in authenticating not just consumers but all actors needed to get collaboration going, being authenticated by credible third party services.

Ability to transform data, run transactions to their logical end with asynchrony as a key element of the collaboration fabric, with 2 phase commit as the final goal across collaborating systems, would have been impossible without Azure messaging services ( queues and service bus on the cloud).

Azure business analytics services that unlocks enterprise intelligence to empower it further with big data outputs, will add the winning elements of context, timing and relevance to the collaboration services being delivered to the cloud. Getting these three elements right for a single large enterprise is itself a complex activity, getting it right across multiple collaborating partners is a near impossible task without the ability to transform and interpret data on the cloud. This is the reason why we need multiple services as bullet-ed above to be stitched together into a solution. 

The other issue that crops up is that enterprises would not like to remain wedded to it's partners to get STC going for delivering value to the end consumer. The collaboration arrangements could run out of relevance with passage of time , sometimes under 3 hours ( example: credit card companies wanting to collaborate for discounts with in-mall merchants for 2 hours). We need a framework where enterprises could negotiate online the terms and conditions of collaboration, do an online hand shake and get the collaboration going. This is very similar to the scenario where we allow apps to connect to our facebook profile, accept friends invitations on facebook and keep tuning our privacy settings and keep categorizing our friends to get a meaningful discussion / collaboration going. A similar framework will need to be evolved for enterprises to hook-on and hook-off with each other for various transactions that need collaboration between their systems,  their partners systems and consumer's personal apps and devices. A social network for enterprise applications!

What started as a fun fantasy discussion, quickly spiraled into some serious paper napkin drawings of architecture design and some specifications bullet-ed down behind a few thick paper coasters   We were not completely insane, so there were some assumptions that were made. We assumed systems would be intelligent enough to decide whom to befriend and decide on the degree of trust they would like to establish with the system seeking to connect. We decided to tackle the implement-ability of such a scenario on a later date, and focus for now on other issues to get this concept moving forward! 

I need to mention here why I picked Azure. The way Microsoft is approaching the cloud computing paradigm, empowering it with platform services that enterprises need, putting in frameworks for extending these to mobile clients and keeping an open mind on heterogeneity of personal devices and strong preferences of the enterprises to have the flexibility of the cloud with the comfort of implementing their corporate policies. Most of Azure services could literally be an extension of on-premise digital assets as enterprises evolve their view around the right model of governance , security and  management for leveraging the power of the cloud.


written for :