Transformation in your stages are just functions

An often used concept in data warehouses is the concept of ‘stages’, where data moves from one stage to another. Stages are called often called ‘bronze’, ‘silver’ and ‘gold’. Data is ingested as ‘raw data’, which sort of means data as received via api or other endpoints as cvs or json or similar – then transformed to a nested key value data structure (aka ‘semi-structured’) such as a variant in snowflake, jsonb in postgres like databases.From each stage, data is transformed, or improved/enriched to make it more suitable for analyses. As software engineer and architect, when I just functions. Going from one stage means a transformation, aka applying a function. The function might be composes of several other functions, lots of sql queries in jinja templates (DBT) but it’s a function none-the-less.So let’s treat it like that. Create unit tests for it. Don’t make fancy words as ‘stages’, just a…

Continue Reading Transformation in your stages are just functions

Custom laptop

Time for a new laptop, as my current laptop is over five years old. Currently looking for a new laptop. I enjoy some customization, so I have gathered an overview…

Snowflake primary keys woes

In my work I have used Snowflake quite a lot. The so called Data Cloud is very popular. Getting started is easy.One of the issues I have with Snowflake, it’s…

Using OpenAPI in Scala

Modern software development is for a large part integrating with other software. To integrate with other software you use the API software you want to integrate with. An API can…

Using the Woocommerce API

If you want to host your own webshop, Woocommerce is a popular option. You can install it as plugin to WordPress. Using plugins your shop can be extended with different…

Elixer and Phoenix Liveview

A car as metaphor, and featuring yet another introduction to functional programming, yet I liked reading This is the Future of Webdev. Crazy, Right? on Elixir, and Phoenix Liveview, a…

Inside the Machine

After a recommendation on Twitter, I start read the book Inside the machine by Jon Stokes, available as paperback and also somewhere as ebook. The book describes the design and…