Disable Microsoft Defender for Cloud for Visual Studio Subscription (MSDN)

I use a visual studio pro subscription which comes with $150 azure cloud credit, for some reason Microsoft Defender for Cloud was turned on and it consumes more than half of that $150 credit. 

Unfortunately I have not come across a easy switch, some online post suggested there is a place to swith on/off resources but for some reason I can't find that configuration page either.

At last I lookup the az cli command manage to do all these in one line of code: 

az security pricing list |jq -r '.value[].name' | xargs -n 1 -I {} az security pricing create -n {} --tier "free"

Elevating LLM Deployment with FastAPI and React: A Step-By-Step Guide

 In a previous exploration, I delved into creating a Retrieval-Augmented-Generation (RAG) demo, utilising Google’s gemma model, Hugging Face, and Meta’s FAISS, all within a Python notebook. This demonstration showcased the potential to build a locally-run, RAG-powered application.

The conceptual flow of using RAG with LLMs. (Source)

This article aims to advance that groundwork by deploying the model and RAG functionality via FastAPI, with a subsequent consumption of the API through a straightforward ReactJS frontend. A notable enhancement in this iteration is the integration of the open-source Mistral 7b model and the Chroma vector database. The Mistral 7b model is acclaimed for its optimal balance between size and performance, surpassing the Llama 2 13B model across benchmarks and matching the prowess of Google’s gemma model. Continue here

Streamlining Real-Time CDC and Data Replication with Debezium and Kafka

 In today’s fast-paced digital landscape, efficient data management and replication are more critical than ever. This article walks you through setting up a streamlined, real-time Change Data Capture (CDC) and data replication pipeline using Debezium and Kafka. We’ll leverage Docker Compose for a simplified testing environment, avoiding the complexities of server provisioning.

For those considering cloud-based solutions, options like Confluent Cloud offer a Kafka service with a free trial. Alternatively, Azure Event Hubs or AWS’s Managed Kafka services provide robust platforms for handling large-scale data streams.

Full article can be read here: https://medium.com/@george.vane/streamlining-real-time-cdc-and-data-replication-with-debezium-and-kafka-b4d3bc56e2ab 

Database replication using Confluent (Kafka) and Debezium

I have been playing with confluent cloud and Debezium for a little while and found it is extremely useful in streaming data ingestion, the usual use case I came across includes the following two scenario:

1. Use Debezium CDC connector to generate change records to Kafka topics, dump the change records to either cloud storage or to delta lake, this is usually called the raw zone, you can then subsequently consume these change records in your favorate data platform, such as Databricks or Snowflake, both have a rebust streaming ingestion support.

2. Another way is often you just want to have a copy fo the production database for analytics usage, hence a like for like replication is what you need, you can you jdbc sink connector for that, the additional benefits is that you can replicate data to different target database platform, for example mysql to SQL Server, postgres to SQL Server, mysql to postgres etc.

Migrating IBM DB2 to Google Bigtable and achieving FIPS compliance encryption using Java custom encryption library

 This is about a project I undetook recently, the purpose was to migrate  large volume of on-prem db2 data to google bigtable using DataProc, a spark based solution on Google Cloud, a few things that are notable from the project:

1. I have to use SCALA to develop the solution due to the fact that the encryption libary was developed in Java and althoguh it has interoperativity to Python, it does have lot of limitation which stoped me from using Python... on the other hand, SCALA and JAVA just work together seamlessly.

2. for FIPS compliance, I have to use bouncecastle library, which introduces issues in managing dependencies, "dependency hell" as some named it, at then end I have to use mevan to manage dependencies and shade sbt due to the complexity.

3. I used hbase-spark connector for talking to bigtable, Since I am using spark 3, I have to complied the connector libaray manually, see https://github.com/apache/hbase-connectors/tree/master/spark 

(this project was done about a year ago)

Handling Large Messages With Apache Kafka

While working on handling large messages with Kafka, I came across a few useful reference articles, bookmarking here for anyone who needs them:

https://dzone.com/articles/processing-large-messages-with-apache-kafka

https://www.morling.dev/blog/single-message-transforms-swiss-army-knife-of-kafka-connect/

https://www.kai-waehner.de/blog/2020/08/07/apache-kafka-handling-large-messages-and-files-for-image-video-audio-processing/

https://docs.confluent.io/cloud/current/connectors/single-message-transforms.html#cc-single-message-transforms-limitations

Object Tracking Demo

 

In a proof of concept project I undertook a while ago, YOLO (You Only Look Once) object detection model was used in combination with the Deep SORT (Simple Online and Realtime Tracking) algorithm to track objects in real-time. The aim was to showcase how this technology can be applied to traffic monitoring, specifically measuring the time it takes for vehicles to pass through a road junction. The project demonstrated the effectiveness of the combination of these technologies in accurately detecting and tracking vehicles as they move through a monitored area. The results showed that this method could provide accurate data on traffic flow, which could be useful for traffic management and infrastructure planning purposes. The time taken for vehicles to pass through the road junction was also measured as part of the project.

The hidden cost of using a metadata driven ingestion framework

 I came across a few implementations of metadata driven ELT/ETL frameworks, designed and developed by some big consulting firms, perhaps great minds think alike, they all use hashed values  to detect changed records, to be able to avoid duplication and insert changes, this indeed saves a lot of effort and simplify the design.

However one of the major drawbacks of this approach is that often this makes optimization impossible, for example in delta lake, use merge into statement based on the hashed key makes the operation extremely expensive.  this will make partioning not useful or possible, often ended up high running cost.

It will probably make more sense to also define a partition key, zorder columns plus key columns used to identity uniqueness in the metadata, so the delta tables will be created using the optional partition key and zorder columns, then update the merge into statement to use join conditions based on the key columns so it will be able to do parition proning at least, usually the partition key should be part of the key columns  and in most cases is a date column , optionally defining the zorder columns to be the business key etc will also help.

Configure Snowflake with External oAuth using Azure AD and device flow with MFA

Following the article at https://docs.snowflake.com/en/user-guide/oauth-azure.html and https://community.snowflake.com/s/article/How-To-Test-Azure-OAuth-Connection-To-Snowflake-End-To-End-Using-Python-User-Credentials-Flow, it is possible to implement a password flow for authenticating to snowflake using user's own credentials and assume roles they have been granted access to. however this hit an issue with MFA, unless you trusted the ip range of sagemaker this will not work.

To work around this issue, one can enable the public client feature ont he Azure AD client App and use msal to implement device flow instead. please refer to https://github.com/Azure-Samples/ms-identity-python-devicecodeflow for the sample code.

spark + jupyter notebook on ubuntu

 

Step 1: download spark from https://spark.apache.org/downloads.html

Step 2: unzip

           $  tar zxvf ../spark-3.x.x.tar.gz

Step 3: setup bash by adding the following t~/.bashrc

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH

Step 4: install jupyter notebook:  
        $ pip install jupyter

Step 5: Start Spark: 
                $ start-all.sh

Step 6: install findspark package: 
        $ pip install findspark

Step 7: launch jupyter notebook: 
        $jupyter notebook

Step 8: create a new notebook and add the following code for testing:

import findspark

findspark.init()

import pyspark

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.sql("select 'spark' as hello ")

df.show()

organize photo by date taken

 Here is a powershell script I am using, it is based on an exisitng scripts with some modification.

the original script snippet is shown below.

$src= 'D:\My Pictures'

$CharWhiteList = '[^: \w\/]'

$Shell = New-Object -ComObject shell.application

$i = 1

Get-ChildItem $src*.jpg -Recurse  | ForEach{

    $dir = $Shell.Namespace($_.DirectoryName)

    $taken = ($dir.GetDetailsOf($dir.ParseName($_.Name),12) -replace $CharWhiteList)

    #echo $taken

    if (!($taken -eq "") -and !($take -eq " "))

    {

        $datetaken = [DateTime]::ParseExact($taken,"d/MM/yyyy h:mm tt",$null)

        $path = $datetaken | Get-Date -f "yyyy-MM"

        $path = "$($src)\$($path)"

        If(!(test-path $path))

        {

             New-Item -ItemType Directory -Force -Path $path

        }

        $dest = "$($path)\$($_.Name)"

        if (!(Test-Path $dest -PathType leaf))

        {

            Move-Item -Path $_.FullName -Destination $path

        }

    }

}

What is a Lakehouse?

 

Databricks vs Synapse

https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html 

HADOOP eco systems

 

A useful DAX formula for calculating new work won from project variation

Below calculate newly won project work sourced from project variation, it checks a few 

things:

1. restrict the calculations to projects exists from previous month;
2. the formula can also calculate accumulated nww spanned multipel month


NWW Var :=
SUMX (
    VALUES ( DIM_Time[Calendar Month] ),
    SUMX (
        VALUES ( fact_result[project_code] ),
        IF (
            CALCULATE ( COUNTROWS ( fact_result ) ) > 0
                && CALCULATE (
                    COUNTROWS ( fact_result ),
                    PARALLELPERIOD ( DIM_Time[Date], -1, MONTH )
                ) > 0,
            CALCULATE ( SUM ( fact_result[EAC_Fees] ) )
                - CALCULATE (
                    SUM ( fact_result[EAC_Fees] ),
                    PARALLELPERIOD ( DIM_Time[Date], -1, MONTH )
                ),
            0
        )
    )
)

similary you can also derive the formula for calcualting new work won from new projects...

No module named PIL - when running tensorflow with databrick

Google didnt' tell me the answer directly... I have to install pillow onto the cluster and then the problem goes away.......

Time for Password Expiration to Die

https://www.sans.org/security-awareness-training/blog/time-password-expiration-die

Clustered Stacked bar chart with line in secondary axis in Power BI

it looks like there is no native support for this kind of complex chart, initially I tried ggplot2 in R visual however I couldnt' get the ideal solution. finally it appears that this can be done much easier using python pandas + matplotlib.

sample code as below:

df = pd.DataFrame(dict(Subsidy=[3, 3, 3],

                       Bonus=[1, 1, 1],

                       Expense=[2, 2, 2],

                       Salary=[1, 2, 1],

                       Margin=[0.3,0.4,0.5]),

                  list('ABC'))

ax = df[['Subsidy', 'Bonus']].plot.bar(stacked=True, position=1,

                            width=.2, ylim=[0, 8], color=['orange', 'red'])

df['Margin'].plot(secondary_y=True, color='k', marker='o')

df[['Expense', 'Salary']].plot.bar(ax=ax,stacked=True, position=0,

                            width=.2, ylim=[0, 8], color=['blue', 'green'])

SQL upgrade from 2012 to 2016 hangs

during the final stage of the upgrade on the sqlenginedbstartconfigaction_upgrade_configrc_cpu64  screen, the installation process appears to stuck, no error message no progreess so far.

however by looking into ERROLOG, it shows the sql process is busy upgrading databases etc, leave the process running for about 30 minutes and it finishes successfully.

it is bit confusing though since there is absolutely nothing else indicatign the progress is not stuck... even the errorlog only get updates after a few minutes.

Containerize legacy asp.net applications (Epicor E4SE) - take 3

After I figured out the MSDTC issue, all seemed well until I got hit by the MSMQ problem, even though I have diverted the queue to be on a remote machine, E4SE refused to work until it is satisfied that MSMQ is installed on the container, it turned out that MSMQ support in Container is only possible after build 1803 which isn't supported under Windows Server 2016.

Since I havne't got a 2019 server box on hand, I opted to use my windows 10 machine and pulled the 1803 image and rebuilt everything, good news is after all these the MSMQ issue is also gone.
will do more testing later but hopefully there should be no more major issues.

Containerize legacy asp.net applications (Epicor E4SE) - take 2

In my previous post, I almost thought everything is working fine until I got hit by this infamous MSDTC problem, I came across a few posts online and although they are quite inspiring and some of them have got it working, they are not quite the same situation as I got.

here is a summary of what people have got working for MSDTC:

> both app and sql are containerised

> under AWS, using ELB and port mapping

> under Azure, using CNI 

what I want to achieve:

> Containerized asp.net web app

> SQL Server running on VM

>  the web app will need to enable windows authentication under the domain, sql server is running under domain also

For me, my window server 2016 container host is running under vmware, initially I want to use the transparent network to make things easier however enabling promiscuous mode on the vmware environment doesn't seem to be an option so  this is out, at the end I ended up with running container just in NAT network and expose custom port to host.

It turns out that the above mentioned AWS scenario is the closest to mine, what I ended up with is to use KEMP load balancer in the place of ELB, 

what is really important:

> fix MSDTC port on the container and expose to host.

> expose RPC port 135 to host using customer port number

> expose port 80 to host via custom port number.

the KEMP load balancer will then map the ordinary port number to the custom port number on the container host, we also need to create the hostname DNS to point to the KEMP load balancer IP.

the above scenario only involves one container instance, if we need run multiple replica then we will most likely need to multiple the setup but I assume it is all straight forward and no drama here.

and finally MSDTC is working fine and hopefully the PoC is a success