Sample notebook for blog post

samples/features/sql-big-data-cluster/bootstrap-sample-db.sql

@@ -15,6 +15,10 @@ GO
 
 USE sales;
 GO
+-- Create database master key (required for database scoped credentials used in the samples)
+IF NOT EXISTS(SELECT * FROM sys.databases WHERE name = DB_NAME() and is_master_key_encrypted_by_server = 1)
+	CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'sql19bigdatacluster!';
+
 -- Create default data sources for SQL Big Data Cluster
 IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlDataPool')
 	CREATE EXTERNAL DATA SOURCE SqlDataPool

samples/features/sql-big-data-cluster/data-pool/data-ingestion-spark.sql

@@ -26,7 +26,7 @@ IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'web_clickstreams_s
 ---- Main class: "FileStreaming"
 ---- Path to jar: /jar/mssql-spark-lib-assembly-1.0.jar
 ---- Arguments:
----- mssql-master-pool-0.service-master-pool 1433 sa %PASSWORD% sales web_clickstreams_spark_results hdfs:///clickstream_data csv false 
+---- --server mssql-master-pool-0.service-master-pool --port 1433 --user sa --password !yourPassword! --database sales --table web_clickstreams_spark_results --source_dir hdfs:///clickstream_data --input_format csv --enable_checkpoint false --timeout 380000
 
 -- The arguments to jar file are
 -- 1: server name - sql server to connect to read the table schema
@@ -38,6 +38,7 @@ IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'web_clickstreams_s
 -- 7: Source directory for streaming. This must be a full URI - such as "hdfs:///clickstream_data"
 -- 8: Input format. This can be "csv", "parquet", "json".
 -- 9: enable checkpoint: true or false
+-- 10: timeout - in milliseconds - how long to run for before stopping
 --
 
 -- After the Spark streaming job has been sucessfully submitted, you can run below query to view the results.
@@ -52,4 +53,4 @@ SELECT TOP 10 * FROM [web_clickstreams_spark_results];
 GO
 
 DROP EXTERNAL TABLE [dbo].[web_clickstreams_spark_results];
-GO
+GO

samples/features/sql-big-data-cluster/deployment/aks/README.md

@@ -17,7 +17,7 @@ Using this sample Python script, you will deploy a Kubernetes cluster in Azure u
         ```
     - Install mssqlctl CLI latest version using . Run the command below using elevated priviledges (sudo or admin cmd window):
         ```
-        pip3 install --index-url https://private-repo.microsoft.com/python/ctp-2.0 mssqlctl
+        pip3 install --extra-index-url https://private-repo.microsoft.com/python/ctp-2.2 mssqlctl
         ```
 1. Login into your Azure account. Run this command:
 ```

samples/features/sql-big-data-cluster/deployment/aks/deploy-sql-big-data-aks.py

@@ -81,7 +81,7 @@ def executeCmd (cmd):
 print("")
 print("SQL Server big data cluster connection endpoints: ")
 print("SQL Server master instance:")
-command="kubectl get service service-master-pool-lb -o=custom-columns=""IP:.status.loadBalancer.ingress[0].ip,PORT:.spec.ports[0].port"" -n "+CLUSTER_NAME
+command="kubectl get service endpoint-master-pool -o=custom-columns=""IP:.status.loadBalancer.ingress[0].ip,PORT:.spec.ports[0].port"" -n "+CLUSTER_NAME
 executeCmd(command)
 print("")
 print("HDFS/KNOX:")

samples/features/sql-big-data-cluster/deployment/kubeadm/README.md

@@ -53,5 +53,9 @@ To install docker, you can follow the steps below:
 1. Start a sudo shell context
 1. Execute [ubuntu/setup-k8s-prereqs.sh](ubuntu/setup-k8s-prereqs.sh/) script on each machine
 1. Execute [ubuntu/setup-k8s-master.sh](ubuntu/setup-k8s-master.sh/) script on the machine designated as Kubernetes master
-1. After successful initialization of the Kubernetes master, follow the kubeadm join commands output by the script on each agent machine
-1. Now, you can deploy SQL Server 2019 big data cluster using instructions [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions)
+1. After successful initialization of the Kubernetes master, follow the kubeadm join commands output by the setup script on each agent machine
+1. Execute [ubuntu/setup-volumes-agent.sh](ubuntu/setup-volumes-agent.sh/) script on each agent machine to create volumes for local storage
+
+**NOTE: Ensure there is sufficient local storage on your agents. Each volume will use up to 6GB by default. The script creates 25 volumes. Not all of the volumes will be used since it depends on the number of pods being deployed on each agent node. It is recommended to have at least 200 GB of storage on the agent nodes**
+1. Execute ***kubectl apply -f ubuntu/local-storage-provisioner.yaml*** against the Kubernetes cluster to create the local storage provisioner. You can now use persistent volumes by setting the USE_PERSISTENT_VOLUME environment variable to ***true*** and STORAGE_CLASS_NAME environment variable to ***local-storage*** during deployment
+1. Now, you can deploy the SQL Server 2019 big data cluster following instructions [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions)

samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu/local-storage-provisioner.yaml

@@ -0,0 +1,129 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: local-storage
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: local-volume-provisioner
+  namespace: local-storage
+  labels:
+    app: local-volume-provisioner
+spec:
+  type: ClusterIP
+  selector:
+    app: local-volume-provisioner
+  ports:
+  - name: metrics
+    port: 8080
+    protocol: TCP
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: local-storage
+provisioner: kubernetes.io/no-provisioner
+volumeBindingMode: WaitForFirstConsumer
+reclaimPolicy: Delete
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: local-provisioner-config 
+  namespace: local-storage
+data:
+  storageClassMap: |     
+    local-storage:
+       hostDir: /mnt/local-storage
+       mountDir:  /mnt/local-storage
+       blockCleanerCommand:
+         - "/scripts/shred.sh"
+         - "2" 
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: local-volume-provisioner
+  namespace: local-storage
+  labels:
+    app: local-volume-provisioner
+spec:
+  selector:
+    matchLabels:
+      app: local-volume-provisioner 
+  template:
+    metadata:
+      labels:
+        app: local-volume-provisioner
+    spec:
+      serviceAccountName: local-storage-admin
+      containers:
+        - image: "quay.io/external_storage/local-volume-provisioner:v2.1.0"
+          imagePullPolicy: "Always"
+          name: provisioner 
+          securityContext:
+            privileged: true
+          env:
+          - name: MY_NODE_NAME
+            valueFrom:
+              fieldRef:
+                fieldPath: spec.nodeName
+          volumeMounts:
+            - mountPath: /etc/provisioner/config 
+              name: provisioner-config
+              readOnly: true             
+            - mountPath:  /mnt/local-storage
+              name: local-storage
+              mountPropagation: "HostToContainer" 
+      volumes:
+        - name: provisioner-config
+          configMap:
+            name: local-provisioner-config         
+        - name: local-storage
+          hostPath:
+            path: /mnt/local-storage
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: local-storage-admin
+  namespace: local-storage
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: local-storage-provisioner-pv-binding
+  namespace: local-storage
+subjects:
+- kind: ServiceAccount
+  name: local-storage-admin
+  namespace: local-storage
+roleRef:
+  kind: ClusterRole
+  name: system:persistent-volume-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: local-storage-provisioner-node-clusterrole
+  namespace: local-storage
+rules:
+- apiGroups: [""]
+  resources: ["nodes"]
+  verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: local-storage-provisioner-node-binding
+  namespace: local-storage
+subjects:
+- kind: ServiceAccount
+  name: local-storage-admin
+  namespace: local-storage
+roleRef:
+  kind: ClusterRole
+  name: local-storage-provisioner-node-clusterrole
+  apiGroup: rbac.authorization.k8s.io

samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu/setup-volumes-agent.sh

@@ -0,0 +1,11 @@
+#!/bin/bash -e
+
+# num of persistent volumes
+PV_COUNT=25
+
+for i in $(seq 1 $PV_COUNT); do
+  vol="vol$i"
+
+  mkdir -p /mnt/local-storage/$vol
+  mount --bind /mnt/local-storage/$vol /mnt/local-storage/$vol
+done

samples/features/sql-big-data-cluster/machine-learning/spark/h2o/README.md

@@ -0,0 +1,19 @@
+# Automated Machine Learning using H2O in SQL Server 2019 Big Data Cluster
+
+## Machine Learning using Spark
+
+The new built-in notebooks in Azure Data Studio enable data scientists and data engineers to run Python, R, or Scala code against the cluster. This is a great way to explore the data and build machine learning models. Notebooks facilitate collaboration between teammates working on a shared data set.
+
+This sample uses the automated machine learning capabilities of the third party H2O package running in Spark in a SQL Server 2019 Big Data Cluster to build a machine learning model that predicts powerplant output.
+
+### Instructions
+
+1. From Azure Data Studio, connect to the SQL Server Big Data Cluster endpoint. Information about how you connect from Azure Data Studio can be found [here](https://docs.microsoft.com/en-us/sql/azure-data-studio/sql-server-2019-extension?view=sql-server-ver15).
+
+1. Download and save the notebook file [h2o-automl-powerplant.ipynb](h2o-automl-powerplant.ipynb/) locally.
+
+1. Open the notebook file in Azure Data Studio (right click on the SQL Server big data cluster  server name-> **Manage**-> Open Notebook).
+
+1. Wait for the “Kernel” and the target context (“Attach to”) to be populated. Set the “Kernel” to **PySpark3** and “Attach to” needs to be the IP address of your big data cluster endpoint.
+
+1. Run each cell in the Notebook sequentially using Azure Data Studio. Expect the first cell to take 20 sec to finish. Other cells downloading and installing H2O, downloading data, and performing automated machine learning may take several minutes to finish.