Data engineering workflows with Apache Spark often involve complex setup procedures, environment configuration, and dependency management. Traditional approaches to creating local Spark development...
Encountering bad data is inevitable. Schema mismatches, missing values, and inconsistent formatting are causes of failures in data pipelines. The question is bad data frequency, not if it occurs. H...
Every data platform I’ve worked on has always been built in a single storage resource. I always wondered, why put all your eggs in a single storage basket? While it may seem simpler - less infrastr...
Every application with dependencies exists in two worlds: the strictly controlled stability of a developer’s environment, and the unpredictable dependency landscape of a user’s installation. This d...
Many teams already have code coverage set up in their CI/CD pipelines, often through tools like pytest or SonarQube. However, accessing these results usually involves waiting for pipeline runs to c...
Setting Code Quality standards is non-negotiable software development. Even more so for corporate environments where software products slowly mould into legacy, living an unintentionally and unheal...
As developers, we love to automate everything. We set up CI/CD pipelines to streamline the entire lifecycle of our projects, but when it comes to our local environments, we still find ourselves rep...
Code quality tools are essential for maintaining consistency and catching errors in a codebase. While Pre-commit hooks are a common solution, they only catch issues after the work is finished, lead...
Over time, any application environment can become unstable due to conflicting dependencies, configuration changes, and residual files. Container technologies are well-known for solving the aforemen...
Speed up test suite execution by using pytest-split to distribute tests across multiple CI/CD agents. This example uses Azure Pipelines, but the method applies to any CI/CD platform that supports p...