All Pipes is a collaborative initiative between York Region and its nine Area Municipalities to maintain a single common schema and water/wastewater utility dataset among partners. Spatial information about this infrastructure is collected and maintained by the municipalities, who then submit the information to the Region for inclusion in the common dataset.
Each municipality captures their spatial information and attributes in a different way, meaning that their data schemas are different and the level of completeness and quality between municipalities differs. In order to combine these into the common dataset, the All Pipes system must perform a number of quality review checks and translate the submitted data into the common format.
York Region had identified some limitations with the existing All Pipes platform and commissioned Spatial DNA to design and implement an enhanced solution.
Business Problem
To handle the difference between municipal schemas, the Region was using separate, custom workflows for each municipality. However, whenever a municipality’s schema changed, an All Pipes administrator had to manually update the relevant workflow which took time and was error-prone.
The existing All Pipes Portal was a custom web application that was disconnected from other Regional web services – municipal partners needed to log into one site to submit their data, then log into another to view the data. This hindered usability, and the Portal did not provide an intuitive way to see the status of uploads and data errors that required correction. Additionally, other stakeholders within the Region wanted more visibility into the All Pipes program, so there was a need to integrate All Pipes into corporate reporting and business intelligence systems.
The Region decided to enhance the All Pipes platform to reduce the amount of administrator intervention required, to integrate the solution into existing UI and reporting platforms, and to improve the communication and feedback to users and administrators.
Our Approach
Spatial DNA reviewed the existing All Pipes processes and implementation with key stakeholders and came to a consensus on how a revised approach to the All Pipes platform could address the business problems they were facing and provide a more easily supported solution.
There were a number of enterprise software components already in use at York Region that were well supported within the organization. Spatial DNA designed a revised All Pipes solution to leverage these existing platforms so as to expedite implementation and to minimize on-going maintenance effort. Latitude Geocortex was used for data submission and status reports, Safe Software’s FME was used for data validation and transformation, and the Esri platform was used for data storage and verifying the overall consistency of the All Pipes data. Information, statistics, and error reports for each All Pipes submission were recorded in a corporate database and made available to stakeholders using Crystal Reports published to the Region’s intranet.
The use of Geocortex as the user interface for municipal partners removed the need for a custom web application. Users were familiar with Geocortex since it was the platform on which they would view or access the common All Pipes data. This reduced the complexity of user training and sped up the adoption of the new system. As well, there were systems in place at York Region to support and maintain Geocortex and the revised All Pipes user interface.
All Pipes was using nearly 50 FME services to perform automated data transformation, several of which required on-going maintenance by Region staff. In the new approach, Spatial DNA reduced the number of services by 80% while at the same time implementing a comprehensive set of data quality checks and corrective actions within FME. All Pipes administrators were provided with a set of easy-to-maintain configuration files so that they no longer needed to edit and republish services whenever a schema changed or a new quality check was to be performed. This greatly minimized the effort and risk associated with changes to the system.
Results and Benefits
The enhanced All Pipes platform made a significant reduction in the administrative effort required by Region staff to support the All Pipes program and empowered municipal partners and Region stakeholders with better tools for using All Pipes. Spatial DNA’s agile project execution methodology ensured that the project sponsors could shape the project throughout the design and development phases, and thus the final implementation met their expectations. The solution designed and implemented by Spatial DNA was intended to be re-usable, and as such the Region plans to expand the program to include other collaborative datasets such as street lights and parks.
York Region and Spatial DNA jointly presented the solution at the FME 2016 World Tour in Toronto.
Watch the presentation