Powerful Google developer tools for immediate impact! (2023-24 C)
No pressure, but the Workspace needs fixing so the coworker can go have her baby...
1. CONNECT. TRANSFORM. AUTOMATE.
No pressure, but the Workbench
needs fixing so the co-worker can
go have her baby...
Bruce Harold
Product Manager, Esri
2. Where does Esri use FME technology?
Report or Story?
Show of hands please
3. Your Next 20 19 Minutes Journey
Ordinary World
Call to Adventure
Refusal of the Call
Meeting with the Mentor
Crossing the Threshold
Tests, Allies and Enemies
Approach the Innermost Cave
Ordeal
Reward
The Road Back
Resurrection
Return with the Elixir
Better World
4. Ethos, Logos, Pathos
Project an image of credibility
Then go prove it
Win hearts and minds
Get more work to do….
5. Patterns in FME Use Cases - Formats
Spreadsheets
DBase
OGC Formats
CAD
CSV (the optimists
spreadsheet)
Custom (weird)
structures
XML (what are
they thinking?)
JSON
RSS
Directories of Data
URLs
The baby is coming
7. The Dreaded Functionality Gap
Change is guaranteed
Change always meets resistance
It is hard to design for change
Luckily, our software is complete and perfect
Except for…
11. Community Maps
Crowd to Cloud
Authoritative Data for:
Streets, Topography, Imagery, Hydrography
Esri supply the data model
Contributor sends data regularly
Contributor data is Change Detected (FME!)
Adds, Deletes go to SDE Version, gets posted
Map tile cache refreshed
14. Community TIGER
Census Bureau project to harvest:
Streets with Address Ranges
Address Points (Multiple Dwelling Units)
Local Government Boundaries
39,000 Local Governments
Quality Control Steps – Data Reviewer
Interesting combination of DR workspace & DI
Change Detect, De-dupe, Compress, Load to SDE
23. Thank You!
Questions?
For more information:
Bruce Harold, bharold@esri.com
Esri Inc.
www.esri.com
Notas do Editor
FME technology use at Esri:Who wants a report and who wants a story?Bad luck for anyone wanting a report, here is my story…
Hollywood script buyers always look for these aspects of a story, so if you’re looking to retire early by selling a movie script make sure you include them.FME professionals will recognize these plot lines in their everyday work. You get a tap on the shoulder, grudgingly launch into a project, battle the ghouls of Bad Data, find a way to beat them in their hollowed out volcano lair, achieve the satisfaction of success, the banality of documenting your workflow, and emerge with a functioning workspace with which to make the (customer’s) world a better place.
The Greek ideals of persuasion (in our case to use Workbench)
More specifically, FME use cases tend to crop up in patterns. Here are some that are format-driven.An example of a ‘weird’ structure would be 4kB-wide fixed record length data from a vehicle data aggregator.The baby situation is just bad planning…
Here are some that are functionality-driven drivers for using FME technology.By ‘feature granularity’ I mean taking individual features down different tracks based on some test.‘ETL ease of use’ would be one theme I can pick out for why we elect to use FME technology ahead of (say) scripted workflows.
More specifically, FME use cases tend to crop up in patterns that aren’t functionality drivenThe baby situation is just bad planning…
Esri has 900 geoprocessing tools, plus change. You would think we covered it all, and we do, but sometimes at the cost of coding.FME technology is used by Esri when it makes sense, to plug gaps, and in an easily developed and documented environment – Workbench.
Esri’s core GP software does have field mapping and casting capabilities. See the Field Map control here, it crops up in several tools. It is tricky to script.Model Builder is of course the Esri environment analogous to Workbench.
The baby story is true, I got a call late one day from a co-worker who had to complete a job for an expectant mother being induced that evening.By that I mean the baby was coming that evening AND the project had to complete that evening. Who plans these things?The co-worker didn’t know Workbench well, but we made the deadline (and baby arrived).
The first project I’ll outline that depends on FME technology is Community Maps
If you are the curator for these types of useful data WE WANT IT.We will cook it into tiled map services and serve it to the World.
We do eat our own dog food, see here Models, script tools and of course an ETL tool handling the critical change detection function.
In fact it is all scripted after importing the models and tools…which IMHO is a better approach than the executable script approach native to FME.I think SAFE should re-expose fmeobjects as a Python module, but that is a discussion for another day.
Community TIGER is another major project where we use FME technology.TIGER is the US Census Bureau street centreline dataset they make publicly available.The general idea is to streamline aggregating street and address data for Census purposes.
Speaking of Census’ purposes, the URL isn’t really a workflow Esri does, Census build the geocoding service with our outputs. So that’s the end game.AFAIK this is the first announcement at any event of the existence of Census’ geocoder!
Note the ETL tools.
DI is handy for bulk 3D work, here I made some 3D parcel data for geocoding development work, for the real estate market.We will support 3D data in our new US Apartment locator style due out in Q3.
At the extreme end we have this example from the Chamber of Horrors – AIXM file processing.Aeronautical team process these very large XML files for military customers.
We find many businesses communicate in lowest common denominator formats – Excel, Dbase, CSV etc.You can script for these in Python but why go to the trouble?Workbench delivers easy development and documentation for business-friendly formats, in and out.
Nothing fancy here, just handling Dbase as we get it from the customer…
When I get bored I do research.I leveraged something I found in a Python list I’m on and was pleasantly surprised to see the approach worked in a PythonCaller.What is happening here is a pool of worker processes is doing something with geometry in parallel.SAFE will be able to tell you if this should work generally, and if it breaks their license agreement .
I just want to call out Python again here; Esrilites tend to be Python people, personally I use PythonCaller a lot, it gets you out of jail frequently.My joke hypothesis in this slide that with generic formats and dynamic schema, plus PythonCaller, you can do anything.