Enterprise has, for a long time struggled with how to share data with individual business units in a manner that was safe for the data, yet flexible and allowed data sharing without breaking business models.
I’ve always liked simple things, so watched in bemusement at the recent waves of data warehousing and ESB implementations go by. As a wise Professor of mine once told me ‘Almost any problem in IT can be solved by a layer of abstraction’ – though some abstractions are better than others
By my reckoning, there are serious issues in both approaches for business.
For data warehousing:
1. How to get data there, and keep it current?
Businesses seem to by and large just implement siphons of data from existing business systems, but leave out actually providing much value aside from perhaps reporting to the CEO how many widgets he sold yesterday. For highly streamlined logistics or supply chain businesses, this can be enough, but its hardly putting things to good use.
2. BI on data warehousing is traditionally very expensive, and expensive to maintain.
Most businesses don’t have good data analysis skills to ask the right questions, or understand the answer from BI tools.
3. Security is hard on big data sets, and they are ripe for abuse.
4. Changes anywhere in the data models have the tendency to break the transforms people have built up, and this can be hard to detect and correct.
As for ESBs, they have their own challenges:
1. Many, but not all are hard to configure, hard to maintain, and a single point of failure for the business.
2. Typically security models for the ESBs don’t exist, or aren’t implemented.
3. ESBs don’t necessarily solve any business problems, as they still need to be integrated at each touch point.
4. The business needs to decide how much of their business rules will be implemented in the ESB. This, while seeming simple, is hard.
So what is the solution? Like a lot of things, it’s all about web startups doing their own thing showing traditional IT how it’s going to be done. They use APIs.
The big move in the web now, is around publishing, and encouraging developers to integrate with, your APIs. This is made a lot easier nowadays with the moves towards proper web application separation, and REST interfaces becoming the norm. If you need speed, and don’t need complex data sets, you can use JSON with many API’s too. The use of REST and JSON as de-facto standards takes away the need to discuss or negotiate the method by which you convert and transmit data between business or functional units. Web Apps nowadays live and die by the integration they have with mobile, other web apps, and how you can get data in and out of them. Extensibility via plugins is also huge for many applications, and APIs are used for this too.
As businesses, the lesson we can take away from this is that structural separation via APIs is a workable model, and can provide benefits in allowing information interchange while still abstracting away the direct interface with data.
Of course there are challenges. Building API’s is difficult and normally need in-house development, but the benefit for new players right now is that with so many public APIs out there, there are plenty of models to work from. And legions of developers able to complain in areas like stack overflow when an API turns out to be terrible.
Also APIs force individual data owners to enforce business rules through the specification of the API, and will need to implement their own security. But in the end, this is what happens in practice in many organisations anyway, whether they like it or not. Using a federated authentication is also a challenge, but again, the web has a range of OAuth and open source implementations that are ripe for the picking, and I would suggest bound to run into a whole lot less trouble than many of the big software vendors systems – but more on that in another post.
While APIs are probably the best solution for many businesses, there are a couple of cases where APIs probably aren’t best.
1. You need wholesale, fast access to data. Data Warehouses are obviously built for this, and excel at it. ‘Big Data’ type problems are best solved away from the overheads of APIs.
2. You need a range of automation and business rules implemented between systems themselves. This is prime for ESB – but evaluate APIs first, as its too easy to move to ESBs based on problems that can be easily solved with APIs, even if you are dealing with 10’s of systems.