Add CI that tests each Snowflake sample on Snowflake Next

.github/workflows/main.yml

@@ -0,0 +1,87 @@
+name: Test samples
+
+on:
+  push:
+    paths-ignore:
+      - '**.md'
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  sample:
+    name: ${{ matrix.sample }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        sample:
+          - airflow-dbt-transformation
+          - citi-bike-data-app
+          - credit-card-fraud-detection-with-snowpark
+          - credit-scoring-with-snowpark
+          - glue-snowflake-integration
+          - lambda-snowpark-connector
+          - multi-container
+          - predicting-customer-spend
+          - soda-data-quality-checks
+          - streamlit-snowpark-dynamic-filters
+    defaults:
+      run:
+        working-directory: ${{ matrix.sample }}
+    env:
+      AWS_ACCESS_KEY_ID: test
+      AWS_SECRET_ACCESS_KEY: test
+      AWS_DEFAULT_REGION: us-east-1
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Setup Terraform
+        if: matrix.sample == 'glue-snowflake-integration'
+        uses: hashicorp/setup-terraform@v3
+        with:
+          terraform_wrapper: false
+
+      - name: Install lstk and the Snowflake and AWS CLIs
+        run: |
+          npm install -g @localstack/lstk
+          pip install snowflake-cli awscli-local terraform-local
+          for name in localstack local; do
+            snow connection add --connection-name $name --user test --password test \
+              --account test --host snowflake.localhost.localstack.cloud --no-interactive
+          done
+
+      - name: Install sample dependencies
+        run: make install
+
+      - name: Start LocalStack for Snowflake
+        env:
+          LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
+        # samples whose `start` target runs the app name this target start-localstack
+        run: |
+          if grep -q '^start-localstack:' Makefile; then make start-localstack; else make start; fi
+
+      - name: Run tests
+        run: make test
+
+      - name: Show LocalStack logs
+        if: failure()
+        run: make logs || true

README.md

@@ -2,13 +2,15 @@
 
 This repository contains sample projects that can be deployed on LocalStack for Snowflake to demonstrate local development and testing workflows for Snowflake data applications.
 
+The samples run on the next-generation Snowflake emulator, [`localstack/snowflake-next`](https://hub.docker.com/r/localstack/snowflake-next). It emulates Snowflake only, so the samples that also use AWS services run LocalStack AWS next to it with Docker Compose.
+
 ## Prerequisites
 
-- A valid [LocalStack for Snowflake license](https://localstack.cloud/pricing). Your license provides a [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/).
+- A valid [LocalStack for Snowflake license](https://localstack.cloud/pricing) with access to the Snowflake Next preview. Your license provides a [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/).
 - [Docker](https://docs.docker.com/get-docker/)
-- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli)
-- [`awslocal` CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/)
-- [LocalStack Snowflake emulator](https://snowflake.localstack.cloud/getting-started/installation/)
+- [`lstk`](https://docs.localstack.cloud/snowflake/developer-tools/lstk/) 1.3.0 or later, which starts the emulator with `lstk start --type snowflake --image localstack/snowflake-next:latest`
+- [Docker Compose](https://docs.docker.com/compose/) and the [`awslocal` CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/) for the samples that also use AWS services: `airflow-dbt-transformation`, `glue-snowflake-integration`, `lambda-snowpark-connector` and `multi-container`. Each of them has a `docker-compose.yml` that starts LocalStack AWS on port 4566 and Snowflake Next on a second port.
+- [Snowflake CLI](https://docs.snowflake.com/en/developer-guide/snowflake-cli/installation/installation) (`snow`) with a `local` and a `localstack` connection, see below
 
 ## Configuration
 
@@ -18,10 +20,15 @@ Set your auth token before running any sample:
 export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
 ```
 
-Alternatively, use the LocalStack CLI:
+Alternatively, log in with `lstk login`.
+
+Several samples use the Snowflake CLI with a connection named `local` or `localstack`. Create both, pointing at the emulator:
 
 ```bash
-localstack auth set-token <your-auth-token>
+for name in local localstack; do
+  snow connection add --connection-name $name --user test --password test \
+    --account test --host snowflake.localhost.localstack.cloud --no-interactive
+done
 ```
 
 ## Outline
@@ -39,6 +46,12 @@ localstack auth set-token <your-auth-token>
 | [soda-data-quality-checks](./soda-data-quality-checks) | Execute Soda data quality checks against local Snowflake tables. |
 | [streamlit-snowpark-dynamic-filters](./streamlit-snowpark-dynamic-filters) | Streamlit application with dynamic Snowpark-powered filters. |
 
+## Testing
+
+A fresh Snowflake Next instance only has the `SNOWFLAKE` database, as in a new Snowflake account, so each sample creates the database it uses before loading data.
+
+Every sample has `make start` (or `make start-localstack` where `make start` runs the app) to start LocalStack, and `make test` to check that the sample works. The [CI workflow](.github/workflows/main.yml) runs `make install`, the start target and `make test` for each sample on every pull request.
+
 ## Checking Out A Single Sample
 
 To check out only one sample directory:

airflow-dbt-transformation/Makefile

@@ -1,26 +1,48 @@
+AIRFLOW_CONTAINER = localstack-mwaa-000000000000-us-east-1-my-mwaa-env
+
 usage:		## Shows usage for this Makefile
 	@cat Makefile | grep -E '^[a-zA-Z_-]+:.*?## .*$$' | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'
 
 install:    ## Install dependencies
 	which awslocal || pip install 'awscli-local[ver1]'
+	which snow || pip install snowflake-cli
+
+start:      ## Start LocalStack for AWS and the Snowflake emulator, and wait until both are healthy
+	docker compose up -d --wait
+
+stop:       ## Stop both containers and remove the Airflow container
+	-docker rm -f $(AIRFLOW_CONTAINER)
+	docker compose down
+
+logs:       ## Show the logs of both containers
+	docker compose logs --tail 300
 
 init:      ## Initialize the Airflow environment in LocalStack MWAA
 	awslocal s3 mb s3://snowflake-airflow
-	docker rm -f localstack-mwaa-000000000000-us-east-1-my-mwaa-env
+	awslocal s3 cp requirements.txt s3://snowflake-airflow/
+	-docker rm -f $(AIRFLOW_CONTAINER)
 	awslocal mwaa create-environment --dag-s3-path /dags \
+		--requirements-s3-path /requirements.txt \
 		--execution-role-arn arn:aws:iam::000000000000:role/airflow-role \
 		--network-configuration {} \
 		--source-bucket-arn arn:aws:s3:::snowflake-airflow \
 		--airflow-configuration-options agent.code=007,agent.name=bond \
 		--name my-mwaa-env \
 		--endpoint-url http://localhost.localstack.cloud:4566
 
-deploy:     ## Deploy the DAG to the local Airflow instance
+deploy:     ## Deploy the DAG and the dbt project to the local Airflow instance
 	awslocal s3 cp requirements.txt s3://snowflake-airflow/
 	awslocal s3 cp packages.yml s3://snowflake-airflow/dags/
 	awslocal s3 cp dbt_project.yml s3://snowflake-airflow/dags/
 	awslocal s3 cp --recursive models s3://snowflake-airflow/dags/models
 	awslocal s3 cp --recursive seeds s3://snowflake-airflow/dags/seeds
+	awslocal s3 cp --recursive macros s3://snowflake-airflow/dags/macros
 	awslocal s3 cp airflow_dag.py s3://snowflake-airflow/dags/
 
-.PHONY: usage install init deploy
+restart:    ## Restart the Airflow services so they pick up the synced DAG and dependencies
+	docker exec $(AIRFLOW_CONTAINER) supervisorctl restart all
+
+test:       ## Create the Airflow environment, deploy the DAG, run it and check the result
+	./test.sh
+
+.PHONY: usage install start stop logs init deploy restart test

airflow-dbt-transformation/README.md

@@ -1,33 +1,44 @@
 # LocalStack Demo: Data Engineering with Apache Airflow, Snowflake, Snowpark, dbt & Cosmos
 
-This project illustrates how to use the LocalStack Snowflake+MWAA to run a data transformation pipeline entirely on your local machine.
+This project illustrates how to use LocalStack for AWS (MWAA and S3) together with the LocalStack Snowflake emulator ([`localstack/snowflake-next`](https://hub.docker.com/r/localstack/snowflake-next)) to run a data transformation pipeline entirely on your local machine.
 
 The code is based on the Snowflake Guide for [Data Engineering with Apache Airflow, Snowflake, Snowpark, dbt & Cosmos](https://quickstarts.snowflake.com/guide/data_engineering_with_apache_airflow).
 
 ## Prerequisites
 - A valid [LocalStack for Snowflake license](https://www.localstack.cloud/pricing). Your license provides a [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/).
-- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli) with [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/) environment variable set
+- [Docker](https://docs.docker.com/get-docker/) with Docker Compose
 - [`awslocal` CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/#localstack-aws-cli-awslocal)
-- [LocalStack Snowflake emulator](https://snowflake.localstack.cloud/getting-started/installation/)
+- [Snowflake CLI](https://docs.snowflake.com/en/developer-guide/snowflake-cli/installation/installation) (`snow`), used by `make test` to check the result
 
 ## Instructions
 
 ## Start LocalStack
 
-Start LocalStack with the custom Snowflake/Airflow networking flags:
+The Snowflake emulator only emulates Snowflake, so this sample runs two containers with Docker Compose (see `docker-compose.yml`):
+
+- `localstack-aws`: LocalStack for AWS, which runs MWAA and S3, on port 4566
+- `localstack-snowflake`: the Snowflake emulator, on port 4567
+
+Start both and wait until they are healthy:
 
 ```bash
-docker network create --attachable --subnet 172.20.0.0/24 localstack
-DOCKER_FLAGS='-e SF_LOG=trace --network localstack --name=localhost.localstack.cloud --network-alias=snowflake.localhost.localstack.cloud'
-  DEBUG=1 \
-  localstack start -s snowflake -d
+export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
+make start
 ```
 
+How the containers connect:
+
+- `MAIN_DOCKER_NETWORK=airflow-snowflake` makes LocalStack start the MWAA Airflow container on the Compose network, so the DAG reaches the emulator at `http://snowflake:4566`. It uses plain HTTP, since the emulator's TLS certificate is for `snowflake.localhost.localstack.cloud`, not the service name.
+- `SF_HOSTNAMES` lists the hostnames the emulator answers to: `snowflake.localhost.localstack.cloud` for clients on your machine, and `snowflake` for the Airflow container.
+- `init.sf.sql` creates the `test` database at startup. `SNOWFLAKE_INIT_ACCOUNT=TEST` runs it in the account the connections use (`account=test`).
+
+From your machine, connect to `snowflake.localhost.localstack.cloud` on port 4567, as in `profiles.yml`. Stop everything with `make stop`.
+
 ## Deploy the app
 
 The sample application provides Makefile targets to simplify the setup process.
 
-Run the following command to initialize the Airflow environment in LocalStack (this may take a couple of seconds):
+Run the following command to initialize the Airflow environment in LocalStack (this may take a couple of seconds). It uploads `requirements.txt` first, so the environment installs `astronomer-cosmos`, `dbt-snowflake` and the Snowflake provider:
 ```
 make init
 ```
@@ -46,6 +57,14 @@ Now use the following command to deploy the Airflow DAG with our dbt transformat
 make deploy
 ```
 
+LocalStack syncs the files from S3 into the Airflow container every 30 seconds. Once `airflow_dag.py` shows up in the container, restart the Airflow services so they load the DAG with the installed dependencies:
+```
+docker exec localstack-mwaa-000000000000-us-east-1-my-mwaa-env ls /opt/airflow/dags
+make restart
+```
+
+To run the whole flow and check the result in one go, use `make test`. It creates the environment, deploys the DAG, triggers a run through the Airflow CLI, waits for it to succeed and checks that the `transform.PREPPED_DATA` view has rows.
+
 ## Use the Airflow UI to trigger a DAG run
 
 Once the Airflow environment has spun up, and the DAG has been successfully deployed, you should be able to access the Airflow UI under http://localhost.localstack.cloud:4510/home

airflow-dbt-transformation/airflow_dag.py

@@ -1,11 +1,14 @@
+import json
 import os
+import shutil
 from datetime import datetime
 from pathlib import Path
 
-from airflow import settings
-from airflow.models import Connection
-from airflow.operators.empty import EmptyOperator
 from airflow.decorators import dag, task
+try:
+    from airflow.providers.standard.operators.empty import EmptyOperator
+except ImportError:  # Airflow 2
+    from airflow.operators.empty import EmptyOperator
 from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig, ExecutionConfig
 from cosmos.profiles.snowflake.user_pass import SnowflakeUserPasswordProfileMapping
 from snowflake import connector
@@ -17,39 +20,32 @@
 
 SnowflakeUserPasswordProfileMapping.airflow_param_mapping["host"] = "extra.host"
 SnowflakeUserPasswordProfileMapping.airflow_param_mapping["port"] = "extra.port"
+SnowflakeUserPasswordProfileMapping.airflow_param_mapping["protocol"] = "extra.protocol"
 
+# The Snowflake emulator runs in its own container (see docker-compose.yml). The
+# Airflow container shares its network and reaches it by the service name. Use
+# plain HTTP, since the TLS certificate is for snowflake.localhost.localstack.cloud.
 snowflake_connection_params = {
     "user": "test",
     "password": "test",
-    "host": "snowflake.localhost.localstack.cloud",
+    "host": "snowflake",
     "port": 4566,
+    "protocol": "http",
     "account": "test",
     "database": "test",
     "schema": "public",
 }
 
 
-def create_snowflake_connection():
-    conn = Connection(
-        conn_id="snowflake_local",
-        conn_type="snowflake",
-        login="test",
-        password="test",
-        description="LocalStack Snowflake",
-        extra=snowflake_connection_params
-    )
-    session = settings.Session()
-    conn_name = session.query(Connection).filter(Connection.conn_id == conn.conn_id).first()
-
-    if str(conn_name) == str(conn.conn_id):
-        return None
-
-    session.add(conn)
-    session.commit()
-    return conn
-
+# Airflow reads connections from AIRFLOW_CONN_<CONN_ID> environment variables.
+# Airflow 3 does not allow DAG files to write connections to the metadata database.
+os.environ["AIRFLOW_CONN_SNOWFLAKE_LOCAL"] = json.dumps({
+    "conn_type": "snowflake",
+    "login": "test",
+    "password": "test",
+    "extra": {key: value for key, value in snowflake_connection_params.items() if key not in ("user", "password")},
+})
 
-create_snowflake_connection()
 credentials = SnowflakeUserPasswordProfileMapping(
     conn_id="snowflake_local",
     profile_args={"database": "test", "schema": "public"})
@@ -59,13 +55,18 @@ def create_snowflake_connection():
     target_name="dev",
     profile_mapping=credentials)
 
-dbt_executable = f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt"
+# use a dedicated dbt virtualenv if the environment has one, otherwise the dbt
+# installed from requirements.txt
+dbt_venv_executable = f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt"
+dbt_executable = dbt_venv_executable if os.path.exists(dbt_venv_executable) else shutil.which("dbt")
 
 
-@dag(schedule_interval="@hourly",
+@dag(schedule="@hourly",
     start_date=datetime(2024, 6, 10),
     catchup=False,
     dag_id="dbt_snowpark",
+    # run one task at a time, to avoid concurrent CREATE TABLE statements
+    max_active_tasks=1,
 )
 def dbt_snowpark_dag():
     transform_data = DbtTaskGroup(
@@ -81,8 +82,8 @@ def dbt_snowpark_dag():
     @task
     def query_result_data():
         connection = connector.connect(**snowflake_connection_params)
-        # select rows from `PREPPED_DATA` view created by DBT transformation
-        result = connection.cursor().execute("SELECT * FROM PREPPED_DATA")
+        # select rows from the `PREPPED_DATA` view created by the dbt transformation
+        result = connection.cursor().execute("SELECT * FROM transform.PREPPED_DATA")
         result = list(result)
         print("-----")
         print(f"Query result ({len(result)} rows):")

airflow-dbt-transformation/dbt_project.yml

@@ -1,6 +1,6 @@
 name: test_dbt
 models:
-  my_project:
+  test_dbt:
     # Applies to all files under models/example/
     transform:
       schema: transform