Configuring Apache Spark to oversubscribe the # of cpu cores

Apache Spark by default allocates 1 executor per CPU available on the system, so a 16 core VM will have 16 executors available. This is a sane default setting, each Thread will have a full cpu core available. Azure Databricks follows this convention.

ML in Production @ AZUG

A repeat of our Machine Learning in Production talk, this time at Azug.be.

ML in Production @ dataminds.be

@WesleyBackelant and I spoke at dataminds.be meetup on Machine Learning in Production using the Microsoft AI platform. Our talk was mostly focussed on the Team Data Science Process, enabled by Azure Machine Learning services and Azure Databricks with mlflow.

Using Jekyll and Nix to blog

For my blogging I use Jekyll and Nix, hosted on Github Pages. Now that we have a working Nix on Windows setup, we can start to blog.

Gemfile

It all starts with a simple, Gemfile where I specify the dependencies needed. It should look something like this:

 source 'https://rubygems.org'
 gem 'github-pages', group: :jekyll_plugins
 gem "minima"
 gem 'jekyll-seo-tag'
 gem 'jekyll-paginate'

Nix on Microsoft Windows 10

Using nix, the functional package manager on the Windows Subsystem for Linux is fairly easy, however a couple of additional steps need to be done. We will be using Ubuntu from the Microsoft Store, apply some workarounds, and install nix. Nix is also the basis of NixOS a purely functional linux distribution.