What is Enterprise Integration and Where is Needed?
Today, organizations rely heavily on technologies – which are often of different nature - and applications, running in different business units and likely based on different technologies. So, integrating these applications into a unified set of business process has emerged as a priority. Enterprise Integration provides a mechanism to build a composite application and share processes and data.
As several studies indicate (here and here), Enterprise integration started as point-to-point integration and evolved to cloud based enterprise service bus architectures. In a point-to-point integration, integrating between two applications requires code development and creating information you could exchange between them. This may prove cost-saving and effective on small networks, but it results in exponential growth of cost and required effort for enterprises. Forrester Research estimates that up to 35% of development time is spent to create application interfaces.
In an Enterprise Integration platform. Enterprises integrate between applications through an Enterprise service bus (a middleware platform) which orchestrates the information exchanged between the integration endpoints, reducing the number of interconnections. Additionally, Enterprise Integration platforms usually provide features such as
- Scalability, high availability and sizing
- Use of industry standards as interface technologies (no custom development-universal connectivity)
- Transactional processing, since the information should be consistent within multiple applications
- Process driven approach embedding logic into integration flows
- Code abstraction, lowering development cost
- Monitoring and managing components, making all transaction executions accessible and clear.
- Deployment management components
A typical integration project has the following phases:
- Project planning and conceptual solution definition (business review, project scoping and planning)
- Detailed analysis and design of the solution (architecture and design, development environment setup)
- Development and testing of the solution (Development, testing, rollout planning, production environment setup)
- Go live to production (Rollout preparation, go live)
Most of the challenges that may arise come from over-focusing on the development (phase c), compared to project planning analysis and design (a and b). While this is common on all IT projects, it is essential for enterprise integration.
Incomplete or Not Well-Defined Analysis
Business people should know exactly what they are looking in an integration project, and what the deliverables will be. The delivered solution may include processes for several areas, departments, so the project team should have an overall knowledge of business processes.
This also apply for IT people, since applications and systems involved are numerous and may run in different platforms and heterogeneous environments. In terms of technical analysis and design and project management, this makes the composition of individual systems and application into a composite one, difficult to manage. Additionally high consulting rates prevents often, receiving external support.
Not open platforms
There are systems that provide limited interface or non-standard capabilities. Other systems require a license to upgrade and extra costs to customize. In such cases, you might need a custom development (from integration platform side) solution, resulting in extra effort, time and cost.
If a system is unsupported, closed and does not have an interface, you should dispose it.
Clear vendor management is critical for an Enterprise Service Bus Integration project when:
- Vendors cannot do modifications to application for several reasons, or require a new license
- The result of a requested modification may not fulfil time specifications
- Arguments and conflicts regarding areas of responsibility may arise
- Application vendor may not support custom modifications in future upgrades
In these cases, you must strictly define your requirements, responsibilities milestones, and deliverables within your vendor agreement
Performance, Sizing and High Availability
You should know your performance objectives upfront. Some of things you should consider include:
- What is the most important factor, number of supported concurrent users, response time or system throughput?
- Is a stress test required?
- How someone can measure bottlenecks or poor performance?
As general guide to face performance issues:
- In design components that talk frequently should be placed in the same network
- If possible, workload should be distributed
- Bottlenecks should be located during tests
- Focus on processes that take the 80% processing
- Perform step by step changes. Measure impact before apply
- Keep historical logs
Scalability is the ability of a system to handle increased workload. Basic question you should ask in the first step is: vertical or horizontal scalability?
The first is the quick, less expensive solution (e.g. increase memory in a server). The second is the long term: adding a new server in a clustered environment, implementing an external load balancer, etc. Usually, a combination of both is applicable.
Every business that runs an enterprise service bus integration platform wants their integration platform to be available as much as possible. Since this is strongly relates to cost and performance, you should figure out how much downtime your business can tolerate prior to design and implementation.
Transactional integrity: while it is very common for multiple integration services to execute atomically, we require either an “all complete” or “all no complete”. (Most RDBMs and ERPs provide such functionality)
Most integration platforms provide transactional processing capabilities with some additionally guaranteeing the processing of documents.
Setting Up Test Environments. Incomplete Tests
Setting up in integration project test environments requires duplicating (simulating) your production ones, including all systems and applications involved. Additionally, it requires that you establish all network connections between test components.
This may prove difficult in some cases:
- Integration between real time applications or applications that require hardware resources not available (e.g. control systems)
- Integration between cloud applications or integration between applications that may require extra licensing for implementing a test environment
Usually, you perform tests in batches on specific dates -according to the project plan-, with the intermediate time dedicated to corrections and improvements. This requires management effort for not having any incomplete tests.
Since data can come from different applications, you need data access control in your integration platform when that Integration platform has an end user component.
If some of your applications include entity information, data mapping comes 1st priority when you integrate these applications.
Usually, you resolve this by applying the “canonical” model, a defined superset of entity’s fields and attributes that the integration endpoints should follow.
Monitoring Tools are not Implemented
You often find integration management software missing from your final solution, usually for cost reasons.
Management and monitoring tools provide information to end user, regarding integration executions, execution statuses, timestamps, which are often valuable in troubleshooting
In conclusion, in an enterprise application and process integration project based on enterprise service bus architecture, your biggest challenge is to focus on analysis and design -both operational and technical. This is because these projects often deal with different applications, technologies and processes that make these projects complex.