An often used concept in data warehouses is the concept of ‘stages’, where data moves from one stage to another. Stages are called often called ‘bronze’, ‘silver’ and ‘gold’. Data is ingested as ‘raw data’, which sort of means data as received via api or other endpoints as cvs or json or similar – then transformed to a nested key value data structure (aka ‘semi-structured’) such as a variant in snowflake, jsonb in postgres like databases.
From each stage, data is transformed, or improved/enriched to make it more suitable for analyses.
As software engineer and architect, when I just functions. Going from one stage means a transformation, aka applying a function. The function might be composes of several other functions, lots of sql queries in jinja templates (DBT) but it’s a function none-the-less.
So let’s treat it like that. Create unit tests for it. Don’t make fancy words as ‘stages’, just a intermediate step you want to persist. Create unit test for your transformation functions, apply type checking.
Never complain errors just happen and data-engineering is not like software engineering.