Configure Snowflake with External oAuth using Azure AD and device flow with MFA

Following the article at https://docs.snowflake.com/en/user-guide/oauth-azure.html and https://community.snowflake.com/s/article/How-To-Test-Azure-OAuth-Connection-To-Snowflake-End-To-End-Using-Python-User-Credentials-Flow, it is possible to implement a password flow for authenticating to snowflake using user's own credentials and assume roles they have been granted access to. however this hit an issue with MFA, unless you trusted the ip range of sagemaker this will not work.

To work around this issue, one can enable the public client feature ont he Azure AD client App and use msal to implement device flow instead. please refer to https://github.com/Azure-Samples/ms-identity-python-devicecodeflow for the sample code.

spark + jupyter notebook on ubuntu

 

Step 1: download spark from https://spark.apache.org/downloads.html

Step 2: unzip

           $  tar zxvf ../spark-3.x.x.tar.gz

Step 3: setup bash by adding the following t~/.bashrc

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH

Step 4: install jupyter notebook:  
        $ pip install jupyter

Step 5: Start Spark: 
                $ start-all.sh

Step 6: install findspark package: 
        $ pip install findspark

Step 7: launch jupyter notebook: 
        $jupyter notebook

Step 8: create a new notebook and add the following code for testing:

import findspark

findspark.init()

import pyspark

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.sql("select 'spark' as hello ")

df.show()

organize photo by date taken

 Here is a powershell script I am using, it is based on an exisitng scripts with some modification.

the original script snippet is shown below.

$src= 'D:\My Pictures'

$CharWhiteList = '[^: \w\/]'

$Shell = New-Object -ComObject shell.application

$i = 1

Get-ChildItem $src*.jpg -Recurse  | ForEach{

    $dir = $Shell.Namespace($_.DirectoryName)

    $taken = ($dir.GetDetailsOf($dir.ParseName($_.Name),12) -replace $CharWhiteList)

    #echo $taken

    if (!($taken -eq "") -and !($take -eq " "))

    {

        $datetaken = [DateTime]::ParseExact($taken,"d/MM/yyyy h:mm tt",$null)

        $path = $datetaken | Get-Date -f "yyyy-MM"

        $path = "$($src)\$($path)"

        If(!(test-path $path))

        {

             New-Item -ItemType Directory -Force -Path $path

        }

        $dest = "$($path)\$($_.Name)"

        if (!(Test-Path $dest -PathType leaf))

        {

            Move-Item -Path $_.FullName -Destination $path

        }

    }

}

What is a Lakehouse?

 

Databricks vs Synapse

https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html 

HADOOP eco systems

 

A useful DAX formula for calculating new work won from project variation

Below calculate newly won project work sourced from project variation, it checks a few 

things:

1. restrict the calculations to projects exists from previous month;
2. the formula can also calculate accumulated nww spanned multipel month


NWW Var :=
SUMX (
    VALUES ( DIM_Time[Calendar Month] ),
    SUMX (
        VALUES ( fact_result[project_code] ),
        IF (
            CALCULATE ( COUNTROWS ( fact_result ) ) > 0
                && CALCULATE (
                    COUNTROWS ( fact_result ),
                    PARALLELPERIOD ( DIM_Time[Date], -1, MONTH )
                ) > 0,
            CALCULATE ( SUM ( fact_result[EAC_Fees] ) )
                - CALCULATE (
                    SUM ( fact_result[EAC_Fees] ),
                    PARALLELPERIOD ( DIM_Time[Date], -1, MONTH )
                ),
            0
        )
    )
)

similary you can also derive the formula for calcualting new work won from new projects...

No module named PIL - when running tensorflow with databrick

Google didnt' tell me the answer directly... I have to install pillow onto the cluster and then the problem goes away.......