As Open Table Formats Win, the Cost Case for Rebuilding Data Pipelines Gets a Closer Look

Source: International Business Times

For much of the past decade, companies moved data into their analytics systems with pipelines written as code: Spark or Python jobs that pulled records out of operational databases, reshaped them, and loaded the results into a warehouse. The approach works, but it is brittle. A single change upstream, such as a renamed or newly added column, can stop a job and send an engineer digging through custom code to find out why.