The Smart Data Blog

The End of ETL is the Beginning of the IoT

November 26, 2019 | By: Kathy O'Neil

RSS
Follow by Email
Facebook
Twitter
LinkedIn

Written by Sam Chance.

Ad-hoc ETL integrations have drained the enterprise of valuable resources for too long. It’s difficult to determine which is worse: the inordinate costs associated with these point integrations, or the exorbitant amounts of time dedicated to stitching them together.

Whenever sources or business requirements change, the largely manual coding process for these integrations must be reworked, resulting in more delays and repetitious work.

This culture of arbitrary data integrations effectively shifts the time that could be devoted to analyzing and operationalizing data resources to simply preparing them. The point ETL conundrum consistently produces a low ROI coupled with high non-productive time.

If organizations simply reallocated the same resources they currently bestow on ETL into holistic, standards-based integrations using W3C RDF and W3C OWL, they’ll realize a bevy of benefits. Firstly, they’ll go from a culture of continually stitching data together to one centered on a harmonized data fabric, reusable for endless use cases. They’ll effectively shift their organizational focus from processing data to profiting from data by spending more time using them.

Most importantly, they can devote these temporal, personnel, and technology expenditures to innovation opportunities in the IoT, which offers some of the most productive data-driven possibilities today.

The ETL Problem

Arbitrary ETL integrations create many difficulties, the most visible of which is cost. Specifically, there are high costs for:

  • Technology: Technological costs associated with ETL include any number of siloed repositories, multiple database instances, data marts, data lakes (including attempts to govern them, which may require additional purchases), and data preparation tools. Each Business Intelligence tool has its own way of creating ETL jobs, which creates more disparity.
  • Personnel: Many organizations have numerous system integrators, developers, and IT teams whose jobs largely consist of performing new, arbitrary integrations each day for ad-hoc requirements. Instead of paying to applying these resources to ETL data, organizations should shift this expertise to analysis.
  • Wasted time: Time spent configuring data for integrations decreases time spent deploying data. Point ETL integrations are too slow to avail organizations of expedient IoT applications.

Firms are much better shifting from these batch mode processes to on-demand data access.

The Resolution

The smart data approach of harmonized data fabrics simplifies integration efforts by standardizing data with uniform models, vocabularies, and unique identifiers. Organizations can normalize disparate data formats to graph using W3C RDF Harmonize data using W3C OWL. Reallocating the time, money, and personnel devoted to ETL enables organizations to continue leveraging existing IT investments instead of overhauling them. Personnel resources are repurposed to facilitate advanced analytics and build knowledge graphs of customers, products, or other domains. The focus evolves from IT manipulating data to business users acting on data in the IoT.

IoT Opportunities

Harmonizing data assets in a holistic data fabric predicated on business understanding of information empowers a plethora of ITLoT use cases. Many of these issue real-time value to drastically improve organization’s mission objectives. The Digital Twin tenet is swiftly revolutionizing manufacturing and the Industrial Internet. Operational digital twins (three-dimensional models predicated on streaming data and other sources) of entire manufacturing environments are critical for identifying bottlenecks, streamlining processing, and offering highly accurate predictions weeks in advance of actual needs.

Digital Threads present these advantages for the production process of equipment assets by eliminating the conventional silos of supply chains. Digital displays or digital price tags offer real-time dynamic pricing to retail customers by integrating sensor data from mobile devices with customer profiles to boost sales.

Broken, But Correctable

ETL is not fundamentally harmful, but ad-hoc, repetitive instances of it are. Semantic data fabrics decrease time and resources for integration, making IoT possibilities attainable. IoT deployments of data fabrics improve the capabilities of organizations to achieve business objectives. This method closes the gap between end users and technology to decrease reliance on IT, diminish decision cycle time, support self-service analytics, and create citizen data scientists.


Leave a Comment