The Role of XML in Cloud Data Integration

Presenter:

David RR Webber,

Oracle Corporation

October 15th, 2010

Introduction

Cloud services introduce new challenges for informationsharing. While certain aspects are familiar and yet stillunresolved, new techniques can be utilized. A canonicalapproach underpinning Cloud data exchanges is vital to ensureconsistent understanding and enhanced interoperability.

Developers face many challenges and complexities in usingtoday’s industry standards for information exchanges. How canwe simplify this and rapidly develop consistent and conforminginformation exchanges in the Cloud?

Open source tools will be discussed and government andindustry example exchanges presented.

Agenda

Challenges, New and Old

Why a canonical approach?

Adaptive, agile and context aware infrastructure

Avoiding the n(n-1)2 dilemma

Ensuring Simplicity at the Foundation

Never underestimate the ability of engineers to addcomplexity

Open source and open standard solutions

Examples from emergency management domain with NIEM*

Summary and Q&A

* National Information Exchange Model (NIEM) approach

Challenges, New and Old

 Why a Canonical Approach?

 Adaptive, agile and context aware infrastructure

 Avoiding the n(n-1)2 dilemma

Why a Canonical Approach?

Traditional XML information exchange has been schemadriven; many issues for Cloud integration:

W3C XML Schema is inflexible, static, brittle, localized, expensive

Canonical dictionaries exploit Cloud approach withdistributed availability, flexible collaboration and dynamicupdating and referencing

Amazon Web Services AWS catalog an example of dynamic approach

Canonical dictionaries provide the components thatunderpin the exchanges while leaving the preciseexchange formatting open for implementers; the “what”not the “how”; content can change hourly on AWS!

Neutral XML-based syntax is future proof

Canonical XML dictionary

A collection of distinct components that representdiscreet business information for an application domain

Includes singleton components and combinations ofrelated components together as sub-assemblies

Information is represented in a simple neutralconceptual data format that captures the criticalconcepts about the data e.g. name, description, contenttype, contextual usage pattern, hierarchy

Wikipedia definition:

http://en.wikipedia.org/wiki/Canonical#Computer_science

Baking in Interoperability

Using consistent component definitions dramaticallyimproves interoperability and reuse

Having formal design methods makes development faster,easier, predictable and repeatable

Aligning local practice to industry domain dictionary canreduce complexity and reinforce best practices

Dictionary definitions can be automatically evaluated forcommon mistakes and this reduces the opportunity forerrors during design phase

Generating software artifacts from neutral dictionarydefinitions ensures reliable information exchange resultsacross user communities and their particular systems,platforms and tools

Neutral Content Model Representation

Neutral representations allow business stakeholders toparticipate in dictionary development without technologybarriers

Concise neutral formats can be viewed as simplespreadsheets as they have no special syntaxdependencies

Based on open public standard specifications, semanticconcepts and leading knowledge domain techniques

Neutral representation prevents lock-in by vendor,syntax, tooling or platforms

Maximizes flexibility and future proofing of dictionarydefinitions

Linguistic and Semantic Alignment

Formal community domain naming and design rulesprovide consistency of definitions

Consistency of definitions minimizes duplication andoverlapping of dictionary components

Dictionaries allow collaboration on component developmentto improve the overall results

Formal component content detail drives alignment

Design best practices ensure logical self-containedcomponents that can be selected contextually

Avoids explosion of complexity and excessive overdefinition (e.g. “kitchen-sink” schema)

What is a Canonical Approach?

There are several flavors of canonical approaches; somemore complex than others – e.g. UBL vis OAGi vis CCTS

Avoid dependence on W3C XML Schema mechanisms

Core Components Technical Specification (CCTS) simplecomponents with basic hierarchy

Parent components with child entities, and/or components

Associated attributes that denote context and related factors

In CCTS parlance these are ABIE, BBIE and ASBIE

Parent = Aggregate Business Information Entity

Child = Basic Business Information Entity

Attribute = Association Business Information Entity

Conceptual Information Model

Child

(BBIE)

Item

Child

(BBIE)

Item

Parent(ABIE)

Item

Parent(ABIE)

Item

Follows

Naming and Design Rule (NDR)principles and guidelines

CanonicalComponentsDictionary

XML

CanonicalComponentsDictionary

XML

ebXML CCTS terms (ABIE, BBIE, ASBIE)

Parent = Aggregate Business Information Entity

Child = Basic Business Information Entity

Attribute = Association Business Information Entity

Parent(ABIE)

Item

Parent(ABIE)

Item

Parent(ABIE)

Item

Parent(ABIE)

Item

Parent(ABIE)

Item

Parent(ABIE)

Item

. . . . .

Child

(BBIE)

Item

Child

(BBIE)

Item

Child

(BBIE)

Item

Child

(BBIE)

Item

Child

(BBIE)

Item

Child

(BBIE)

Item

Attribute

(ASBIE)

Attribute

(ASBIE)

Attribute

(ASBIE)

Attribute

(ASBIE)

* CCTS – Core Components Technical Specification

Attribute

(ASBIE)

Attribute

(ASBIE)

Attribute

(ASBIE)

Attribute

(ASBIE)

Each compound component

Each atomic component

Optional attributes of component

Example – Person Name

Person Name (ABIE)

Language Code (ASBIE)

Verified Details? (ASBIE)

Has Alias? (ASBIE)

First Name (BBIE)

Middle Name (BBIE)

Last Name (BBIE)

Previous Name? (ASBIE)

Language Code may exist independently of Person Name

Verified Details and Previous Name are flags that denote additionalinformation about the entity they are associated with

There are three component items aspects:

structure relationships; content rules; definitions

Naming and DesignRules (NDR) alsoimportant in ensuringshorter non-specificcontext names

e.g. comparePersonName toIncidentPersonName

Methods for creating Canonical Dictionary

Harvest from collection of domain exchange schema

Export from SQL database to schema; harvest; rename

Export from modelling tool to schema; harvest; rename

Create manually in XML or spreadsheet

Sample Dictionary Building Processes

EDM

Ele

Def

DDL

ExportComponents inXSD syntax

Collection of objectsfrom model

Option 1 – From Enterprise Data Model

Import XSD and refactor for use with OASIS CAM

Option 2 – Derive from existingexchange XSD schema

Import each XSD and merge into CAM dictionary

ExchangeXSDschema

CAMtemplate

OASISCAMtemplate

ModelComponentsXSD schema

CAMtemplate

OASISCAMtemplate

NDR

Evaluation,

Refactor,

Renaming

Tool

NDR

Evaluation,

Refactor,

Renaming

Tool

Apply Naming andDesign Rule (NDR)checks and edits

NDR

Evaluation,

Refactor,

Renaming

Tool

NDR

Evaluation,

Refactor,

Renaming

Tool

Generate

Standard

ComponentsDictionary

XML

Generate

Standard

ComponentsDictionary

XML

Merge &GenerateDictionary

XML

Merge &GenerateDictionary

XML

Dictionary

of exchangecomponents

XML

Import

Dictionary

of exchangecomponents

Automated

Manual

LEGEND

AnalystReview

ebXML CCTScompatible(ABIE, BBIE,ASBIE)

ebXML CCTS compatible(ABIE, BBIE, ASBIE)

Ensuring Simplicity at theFoundation

Never underestimate the ability ofengineers to add complexity

Adaptive, agile and context aware infrastructure

XML validation framework that is configurable dynamicallythrough the use of XML templates and rules.

“In today's complex information exchanges with XML and associatedlarge XSD schema, coupled with an array of trading partners, itbecomes a significant challenge to support and maintain accuratehandling of all incoming transactions”.

“With a more adaptive and fault tolerant process, the application isable to handle a wider variation in content and, hence, more easilysupport a broad set of interaction partners with reduced support andmaintenance costs”.

http://www.ibm.com/developerworks/library/x-camval/index.html

Avoiding the n(n-1)2 dilemma

New XML validation framework

Automotive parts repair with STARBOD example

Utilizing validation framework with singleton validationtemplates that are context rule driven

Diagram of life cycle (Model, Author and test, Deploy, Monitor) for developing validation rules

Source: http://www.ibm.com/developerworks/library/x-camval/index.html#figure2

Agile Solution Components

Def

Ele

Domain

applications

Industry

dictionary

formatted as XML

Interchanges

XML

exchange

realistic test

examples

XML

Schema

Unit Test

Harness

Test

Blueprinttoolkit

Automated

Manual

LEGEND

Definitions Repository (XML)

Exchange Structure Schema

Domain

dictionary

formatted as XML

Templates

Build

CAMV

engine

Content Hints

ExchangeDesigner Tool

User Interface

ReviewStructure

Assembly

Pick

Components

Wantlist

WSDL actions

(optional)

Business

Context

Rules

Agile Validation

Engine

Canonical

Dictionaries

Leveraging Cloud Deployment strengths

Collaboration tools for sharing canonical componentdictionaries

Repositories of templates and code lists

Fault tolerant deployment architectures with redundancy

Machine accessible APIs to allow real time updates andpropagation of changes

Standards based implementations that provide open access

Open source resources for shared implementation support

Open Source and OpenStandard solutions

Examples from EmergencyManagement domain with NIEM,OASIS EDXL, LEXS

Example Emergency Management Scenario

Emergency Response Services Workflow using OASIS EDXL exchanges

Haiti demonstrated

need for agile

exchanges torapidly cope withunfamiliar scenarioand environmentchanges

Cloud-basedsharing of openadaptive commoninfrastructurecomponents

Top Down Solution Approach

Def

Ele

Industry

dictionaries

formatted as XML

Exchange

generatortools (CAM)

Automated

Manual

LEGEND

Components Definition (XML)

Local domain

dictionary

formatted as XML

Build

Exchange

BlueprintDesignerUser Interface

Expand

Structure

Exchange Structure

Pick

Components

Structure OutlineBlueprint

Target

applications

EDM

Ele

Def

DDL

Exchange

Package

Exchange

Package

Exchange

Components

Exchange

Components

EnterpriseData Model

Import and refactorfor use with CAM

Dictionary

Repository

Assembling Components from dictionaries

Determine your business information exchange componentsat conceptual level

Search and locate candidate components from appropriatedomain dictionary collections

Catalogue the parts to be used

Dictionary components can be referenced individually or as collectionsby an assembly blueprint that puts them all together to create acomplete information exchange

Components can be selected from multiple dictionaries

Note any new extension pieces as needed

Select components from multiple physical dictionary files

Blueprints themselves also have high re-use value

Can be sub-assemblies and patterns not just exchange models

Example Assembly Blueprint Outlines

LEXS messaging blueprint

Reusable messaging envelope constructs

OASIS EDXL HAVE message

Business functional components

Message handling,

delivery and control

Payload goes here

Top level sets ofbusinessinformationcomponents

Individualcomponent

these examples available from CAM editor install package

~ CAMeditor\eclipse\workspace\CAMEditor\dictionary\blueprints\

LEXS – Law Enforcement eXchange System – http://www.lexs.gov

Exchange Development Process Tools

Expander

Tool

Expander

Tool

Industry dictionary

Domain dictionary

Component Definitions

Webtool

Excel

Tools

Blueprint

Designer

Blueprint

Designer

Insert

Dictionary

Parent

Components

Insert

Dictionary

Parent

Components

CompletedExchange

Template

CompletedExchange

Template

Summary and Q & A

Review

Resource links

Summary

Canonical XML component dictionaries

Neutral representation of components

Deployment to target environments and architectures

Collaborative development and open source

Uses open public standards and government guidelines (NIEM)

Available resources and tools

Illustrative use cases

Leverage strengths of cloud-based collaboration resources

Resources

Resource links

Supporting supplemental slides

Links and Resources

DOWNLOADS -

CAM Toolkit download

https://sourceforge.net/projects/camprocessor

SUPPORTING MATERIALS -

NIEM Naming and Design Rules (NDR) 1.3

http://www.niem.gov/pdf/NIEM-NDR-1-3.pdf

RESOURCES –

UN/CEFACT Core Components Technical Specification

http://www.unece.org/cefact/ebxml/CCTS_V2-01_Final.pdf

Tutorials - wiki.oasis-open.org/cam/CAM_Tutorials

Specifications

www.oasis-open.org/committees/cam

docs.oasis-open.org/cam

www.oasis-open.org/committees/emergency

NIEM site - www.niem.gov

LEXS site – www.lexs.gov

Available XML Dictionaries

LEXS 3.1.4 dictionary

OASIS EDXL dictionary

OASIS EML dictionary

NIEM 2.1 dictionaries

CBRN dictionary

Emergency dictionary

Family dictionary

Immigration dictionary

Infrastructure dictionary

Intelligence dictionary

Justice dictionary

Maritime dictionary

Screening dictionary

Trade dictionary

NIEM core dictionary

Immigration blueprint

Available from download site

direct link:

http://sourceforge.net/projects/camprocessor/files

XML

+ includes spreadsheets and sample blueprint

Packaged with CAM editor

see dictionary folder of install

+ spreadsheet

+ blueprint samples

XML

Note: Those marked in bold are model style dictionaries with recursive components.

Conceptual Information View

CAM toolkitprocessing

Apply tools in desktopCAM toolkit editor

CAM Template

DOMAIN DATA COMPONENTSDOMAIN DATA COMPONENTS

Structure

Rules

Definitions

Items

Item (ABIE, BBIE, ASBIE)

Properties

Name

Unique ID

Component Type

Cardinality

Content Type

Content Mask

Children

Group

Structure Context

Where from

Definition

Rules

Language, Label, Notes

* Required items in Blue

DICTIONARY COMPONENTSDICTIONARY COMPONENTS

XML View of Dictionary Content

Items

Name

Unique ID

Component Type

Cardinality

Content Type

Content Mask

* See slide notes forexplanation

Parent / Childlinkage

where

referenced

Excel Spreadsheet View

An itemper row

propertiesas columns

Type

(ABIE, BBIE)

children

Mapping to Dictionaries

You can compare a template of components to a dictionary

check within a domain for alignment to dictionary

check between domains for interoperability

merge new/existing components with dictionary

Matches on physical names

Reports matching items and details

Reports statistics and percentages of matching

Generates crosswalk xml file

Compatible with Microsoft Excel

Report can be used to do spell checking

Example cross-reference spreadsheet

Formatted viewin MicrosoftExcel of importof cross-referencereport details(fromgenerated XMLfile)

Matched

details;item andalignment,

definition