Data Exchange NeedsData Exchange Needs
Using ETL ToolsUsing ETL Tools
OverviewOverview
•Client
•Project Issue
•Methodology
•Schedule & Budget
•Results & Recommendations
•Challenges
•Acknowledgments
•Questions
Ministry Of TransportationMinistry Of Transportation
Project Issue & Primary GoalProject Issue & Primary Goal
•Current data conversion process
oEach time data is updated it needs to be redone manually
oNeed to streamline the process
oExtract Transform Load (ETL) Tools
•Safe Software Feature Manipulation Engine (FME)
•Esri ArcGIS Geoprocessing Tools (Data Interoperability Extension)
MethodologyMethodology
StandardMethodologicalApproach:
Identifyobjectives,issues & goals
Researchtopic and bestpractices
Identifyscheduling,budget, risks
Communicate& ConductMeetings
Gather informationfor evaluation,reporting anddecision making
Reportfindings
•Streamlined data conversion process
•User know-how
•ETL tools capabilities and limitations
•ETL tool decision making
•Final Report and recommendations
Goals & DeliverablesGoals & Deliverables
Extract Transform LoadExtract Transform Load
ETL - Traditional
ETL is used to migrate data from one database to another, to formdata marts and data warehouses and also to convert databases fromone format or type to another.
Spatial ETL – Safe Software Feature Manipulation Engine - FME
Provides the data processing functionality of traditional ETL, butwith a primary focus on the ability to manage spatial data (thegeometry, attributes, and coordinate systems of spatial datasets)
Oracle
SQL
BridgeDate xls
ArcSDE
ArcGISServer
![C:\Users\kdell1\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.IE5\8YF4275G\MC900434800[1].png](data/images/img29.png)
ArcGISDesktop
![C:\Users\kdell1\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.IE5\DJ4SRFG4\MC900432591[1].png](data/images/img34.png)
CurrentProcessCurrentProcess
MTO Proof of ConceptMTO Proof of Concept
MTO Proof of ConceptMTO Proof of Concept
MTO Proof of ConceptMTO Proof of Concept
MTO Proof of ConceptMTO Proof of Concept
MTO Proof of ConceptMTO Proof of Concept
FME OutputFME Output
FME ServerFME Server
•FME Desktop is theauthoring tool for FME Server
•Any spatial ETL workflowcreated in FME Desktop canbe published to FME Server
•FME Server is a scalableprocessing environmentthat allows multiple users viathe web
•Workflows can be sharedand automated in thisenvironment
“Safe Software sets the standard for spatial datainteroperability, so it was an obvious choice to use their FMEtechnology for Esri’s ArcGIS Data Interoperability Extension.”
– Jack Dangermond, Esri President
ArcGIS DesktopData Interoperability Spatial ETLArcGIS DesktopData Interoperability Spatial ETL
ArcGIS DesktopData Interoperability Spatial ETLArcGIS DesktopData Interoperability Spatial ETL
ArcGIS DesktopData Interoperability Spatial ETLArcGIS DesktopData Interoperability Spatial ETL
ArcGIS OutputArcGIS Output
Arc ServerArc Server
EvaluationEvaluation
•Architecture
•ETL Functionality
•Ease-of-Use
•Reusability
•Debugging
•Real-time
•Connectivity
•General ETL tool characteristics
ChallengesChallenges
•GIS Expertise, knowledge and training
•Assumed licenses acquired and ready to use
•Data availability and ability to work on data
•Client Representation
RecommendationsRecommendations
•Streamlined data conversion process
•User know-how
•ETL tools capabilities and limitations
•ETL tool decision making
AcknowledgementsAcknowledgements
•Client
oLaura Kingston
•Project Thesis Adviser
o Mike Wallace
•Technical Advising
oColin Bissell
oJonathan Lepage
oAllison Hughes
•FME Software Acquisition
oGerry McCoubrey
oCraig Vernon
QuestionsQuestions
