Open source enterprise data integration tools, such as, WSO2 DSS and WSO2 DAS , play a key role in enterprise data integration.
Plus, open source models can reduce your costs considerably, while eliminating any lock-in issues. According to industry statistics, you can save about 40% to 48% in software licenses by just using open source tools alone as referenced in: ENTERPRISE BUSINESS INTEGRATION FOR A SEAMLESS TOMORROW
The WSO2 Carbon light-weight, open source, service oriented platform hosts a rich set of basic middleware components including security, clustering, logging, statistics, management and others.
The component manager lets you extend the Carbon base platform to address your unique requirements by installing additional components with point-and-click simplicity.
Figure - 2: Overview of WSO2 Data Services Server architecture
Developed on top of WSO2 Carbon, the WSO2 Data Services Server turns your data into WS-* and REST web services and resources with the identity, statistics, governance, security, clustering and management features of WSO2 Carbon.
Figure -3, WSO2 DSS
The Server also uses SOA development to provide an easy-to-use platform for creating and hosting data services, enabling the easy integration of the data into your business processes, mashups, gadgets, BI applications and any service in general.
So, what are the best practices for enterprise data integration using WSO2 DSS Server?
- Data Governance usage :
Data governance specifies decision rights and responsibility framework to have set of correct behaviors. It consists of evaluation, creation, maintenance, storage, usage, archival and removal of the data. To ensure the effective use of it, it emphasizes standards, processes, roles, metrics in every milestones.
- Data Governance targets:
It approves data strategies, policies and procedures. It sponsors, observes and manages the data and its services. It resolves data issues and promote data assets.
It references for more details in
Enterprise information management best practices in data governance
- Take holistic approach and build development iteratively
- Obtain sponsorship exclusively. Flow should be from top to bottom
- Build business case; Estimate quantifiable benefits accordingly
- Collect, establish and report metrics to find the progress on each milestone.
- Maintaining data quality
It is making sure of profiling, cleansing and matching to standards. It includes identification of stakeholders and establishing roles and rights. It also monitors and reports the status of the quality of data.
- Capacity planning
It is about taking care of maximum capacity size.
- Naming convention
Standard naming conventions to be followed.
- Ability to understand the pattern and variations.
- Ability to have master data management
You build good master data management to address the dimensions and reporting.
- Ownership of data
It is needed to be clear whether the data is maintained on-premises or in cloud.
- Single truth of the data
The same meaning columns are integrated even if they are different in names.
- Full view of all transactions
Helps in many things such as row level security, audit logging features.
- Analysis of relationship in multiple angles.
Efficient Entity Relational models to be designed.
- Data Integration components
Identification of all possible master and meta data which helps in Integration and in reporting.
- Data Security
Needed in single sign on and end to end identity management.
- Data compliance:
Data standards should be maintained for compliance and regulation requirements.
- Feature on scalability
Horizontal scaling is better though vertical scaling is possible. Vertical scaling is expensive.
- Standard notification framework.
Clear communication plan and stake holder’s management helps in achieving this.
- Data synchronization.
Data consistency is maintained.
- Monitoring the failure and using fault tolerant system
Having exceptional handling in right places helps to identify the errors and alerts.
- Parallel computing
Wherever possible, parallel processing is enabled.
- Web based easy visualization interfaces
- Identification relevant data stores as input sources
- Batch processing for heavy load processing
Clear definition for historical load processing and real time processing should be available.
- Provision for Real time data analysis
- Performance within SLA
They are referred in
• Top 10 practices of data integration
Figure- 4, Analytics
The best practices in enterprise data integration using WSO2 Analytics?
- Plan on key metrics.
Need to understand the metrics which makes meaningful to the business. Some questions such as which is your final goal for the data collected. How will this data can help your business ? It also means value of data, management cost, number of decisions made, objectives reached, data representation and coverage, data stake holders and management process maturity.
- Extremely High Performant Processing Engine
- Data aggregation
Flexible granular level aggregated reports.
- Support in Big data analytics and reporting.
- Powerful and Extensible Query Language for Temporal Event Stream Processing
- Partitioning support to achieve parallel processing
- User-friendly Execution Management
- Support for Rich Event Model
- Extremely High Performant Event Capturing and Delivery Framework
- Supports Multiple Alert Notifying Mechanisms
- Easily Integrates with any Enterprise System for Event Capture
- IoT (Internet of Things) Integration
Sensor data; unstructured data and edge analytics support are available.
- Debugging Support
- Massively Scalable
- Highly Available Deployment
- Support for Long Duration Queries
- Integrated, real-time, and batch analytics
- Interactive analytics
- Built-in Support for WSO2 Products
- Possibility to create custom dashboards
- Detects conditions and generate real time alerts and notifications (SMS, push notifications, email, physical sensor alarms etc.)
- Exposes event tables as an API via WSO2 API Manager and WSO2 Data Services Server.
- Self-service analytics
- Comprehensive management and monitoring Web console with enterprise-level security
- Built-in collection and monitoring of standard access and performance statistics
- Flexible logging support with integration to enterprise logging system
- In built support for rich representation of data via a dashboard
- Simulating capabilities via the Event Simulator.
Figure -5 shows high level diagram of data flow.
WSO2 Enterprise integrator version 6 comes as a single package with the following enterprise Integration platform modules and profiles included (Wso2 Library)
- ESB Service Integration (WSO2 ESB)
- Message broker (WSO2 MB)
- Business process (WSO2 BPS)
- Micro-services and Analytics.
- Real-time data processing (WSO2 DSS)
- WSO2 Governance registry
- WSO2 business activity monitor (BAM)
- API manager and identity server
Having one tool addressing this topic is really makes simpler. If the tool like WSO2 Integrator is open source and capable of addressing the requirements, then it provides simpler and more useful ways.
WSO2 creates a standardized, fully interoperable, single point of contact for all your enterprise Integration platform needs. This allows you to quickly monitor and handle any risks and points of failure in your network. Compared to other tools, it makes developing, implementing, and maintaining your network easier for you, your IT staff, and your budget.
Top 10 practices of data integration
ENTERPRISE BUSINESS INTEGRATION FOR A SEAMLESS TOMORROW
Enterprise information management best practices in data governance