Cetrix Blog

Customer Enterprise Data Integration Best Practices

Paul Nilsen - Apr 14, 2018 4:23:00 PM

Open source enterprise data integration tools, such as, WSO2 DSS and WSO2 DAS , play a key role in enterprise data integration.

Plus, open source models can reduce your costs considerably, while eliminating any lock-in issues. According to industry statistics, you can save about 40% to 48% in software licenses by just using open source tools alone as referenced in: ENTERPRISE BUSINESS INTEGRATION FOR A SEAMLESS TOMORROW

WSO2 Carbon Frame Work Figure-1, Framework

The WSO2 Carbon light-weight, open source, service oriented platform hosts a rich set of basic middleware components including security, clustering, logging, statistics, management and others.

The component manager lets you extend the Carbon base platform to address your unique requirements by installing additional components with point-and-click simplicity.

WSO2 Data Services Server

Figure - 2: Overview of WSO2 Data Services Server architecture

Developed on top of WSO2 Carbon, the WSO2 Data Services Server turns your data into WS-* and REST web services and resources with the identity, statistics, governance, security, clustering and management features of WSO2 Carbon.

WSO2 DSS Figure -3,  WSO2 DSS

The Server also  uses SOA development to provide an easy-to-use platform for creating and hosting data services, enabling the easy integration of the data into your business processes, mashups, gadgets, BI applications and any service in general.

So, what are the best practices for enterprise data integration using WSO2 DSS Server?

  • Data Governance usage :
    Data governance specifies decision rights and responsibility framework to have set of correct behaviors. It consists of evaluation, creation, maintenance, storage, usage, archival and removal of the data. To ensure the effective use of it, it emphasizes standards, processes, roles, metrics in every milestones.
  • Data Governance targets:
    It approves data strategies, policies and procedures. It sponsors, observes and manages the data and its services. It resolves data issues and promote data assets.
    It references for more details in
    Enterprise information management best practices in data governance
  • Take holistic approach and build development iteratively
  • Obtain sponsorship exclusively. Flow should be from top to bottom
  • Build business case; Estimate quantifiable benefits accordingly
  • Collect, establish and report metrics to find the progress on each milestone.
  • Maintaining data quality

    It is making sure of profiling, cleansing and matching to standards. It includes identification of stakeholders and establishing roles and rights. It also monitors and reports the status of the quality of data.
  • Capacity planning
    It is about taking care of maximum capacity size.
  • Naming convention
    Standard naming conventions to be followed.
  • Ability to understand the pattern and variations.
  • Ability to have master data management
    You build good master data management to address the dimensions and reporting.
  • Ownership of data
    It is needed to be clear whether the data is maintained on-premises or in cloud.
  • Single truth of the data
    The same meaning columns are integrated even if they are different in names.
  • Full view of all transactions
    Helps in many things such as row level security, audit logging features.
  • Analysis of relationship in multiple angles.
    Efficient Entity Relational models to be designed.
  • Data Integration components
    Identification of all possible master and meta data which helps in Integration and in reporting.
  • Data Security
    Needed in single sign on and end to end identity management.
  • Data compliance:
    Data standards should be maintained for compliance and regulation requirements.
  • Feature on scalability
    Horizontal scaling is better though vertical scaling is possible. Vertical scaling is expensive.
  • Standard notification framework.
    Clear communication plan and stake holder’s management helps in achieving this.
  • Data synchronization.
    Data consistency is maintained.
  • Monitoring the failure and using fault tolerant system
    Having exceptional handling in right places helps to identify the errors and alerts.
  • Parallel computing
    Wherever possible, parallel processing is enabled.
  • Web based easy visualization interfaces
  • Identification relevant data stores as input sources
  • Batch processing for heavy load processing
    Clear definition for historical load processing and real time processing should be available.
  • Provision for Real time data analysis
  • Performance within SLA
    They are referred in
             • Top 10 practices of data integration
             • Experts-reveal-the-top-5-data-integration-best-practices
             • three-more-data-integration-best-practices-that-you-may-not-know-about

Analyze Report Collect Figure- 4, Analytics

The best practices in enterprise data integration using WSO2 Analytics?

  • Plan on key metrics.

Need to understand the metrics which makes meaningful to the business. Some questions such as which is your final goal for the data collected. How will this data can help your business ?  It also means value of data, management cost, number of decisions made, objectives reached, data representation and coverage, data stake holders and management process maturity.

  • Extremely High Performant Processing Engine
  • Data aggregation

Flexible granular level aggregated reports.

  • Support in Big data analytics and reporting.
  • Powerful and Extensible Query Language for Temporal Event Stream Processing
  • Partitioning support to achieve parallel processing
  • User-friendly Execution Management
  • Support for Rich Event Model
  • Extremely High Performant Event Capturing and Delivery Framework
  • Supports Multiple Alert Notifying Mechanisms
  • Easily Integrates with any Enterprise System for Event Capture
  • IoT (Internet of Things) Integration

Sensor data; unstructured data and edge analytics support are available.

  • Debugging Support
  • Massively Scalable
  • Highly Available Deployment
  • Support for Long Duration Queries
  • Integrated, real-time, and batch analytics
  • Interactive analytics
  • Built-in Support for WSO2 Products
  • Possibility to create custom dashboards
  • Detects conditions and generate real time alerts and notifications (SMS, push notifications, email, physical sensor alarms etc.)
  • Exposes event tables as an API via WSO2 API Manager and WSO2 Data Services Server.
  • Self-service analytics
  • Comprehensive management and monitoring Web console with enterprise-level security
  • Built-in collection and monitoring of standard access and performance statistics
  • Flexible logging support with integration to enterprise logging system
  • In built support for rich representation of data via a dashboard
  • Simulating capabilities via the Event Simulator.

Configure WSO2Figure -5 shows high level diagram of data flow.

WSO2 Enterprise integrator version 6 comes as a single package with the following enterprise Integration platform modules and profiles included (Wso2 Library)

  • ESB Service Integration (WSO2 ESB)
  • Message broker (WSO2 MB)
  • Business process (WSO2 BPS)
  • Micro-services and Analytics.
  • Real-time data processing (WSO2 DSS)
  • WSO2 Governance registry
  • WSO2 business activity monitor (BAM)
  • API manager and identity server

             Having one tool addressing this topic is really makes simpler. If the tool like WSO2 Integrator is open source and capable of addressing the requirements, then it provides simpler and more useful ways.

WSO2 creates a standardized, fully interoperable, single point of contact for all your enterprise Integration platform needs. This allows you to quickly monitor and handle any risks and points of failure in your network. Compared to other tools, it makes developing, implementing, and maintaining your network easier for you, your IT staff, and your budget.


Wso2 Library
Top 10 practices of data integration
Enterprise information management best practices in data governance

Learn How to get more out of the existing IT infrastructure without quick-fix band-aids

Download the eBook Now.


Band-Aid or Wellness

Band-Aid or Wellness

Topics: Application Integration

Paul Nilsen

Paul Nilsen

Paul Nilsen is the director of solution development at Cetrix Cloud Services. He also works with the Solution Development Department at Cetrix Technologies, where he headed the department for three years. Paul has over 20 years of experience in IT, working on software engineering projects, hardware-software product integration, design of special purpose systems using artificial intelligence, and implementing CRM and ERP software systems.

Previous Post

EDI Legacy Integration vs. Enterprise Bus Platform

Next Post

SaaS integration best practices


Subscribe Here!





Related Blogs