Skip to content

Deploy Feast on Kubernetes

  • Offline Store: BigQuery
  • Registry: postgresql
  • Online Store: Redis

Architecture (Click to Enlarge)
minikube start --cpus=4 --memory=7000 --driver=docker

Build Feast Docker Image Locally

go to feast/ folder. 就會看到以下檔案

Dockerfile
# Use Python 3.10 slim as the base image
FROM python:3.10-slim

# Set the working directory
WORKDIR /app

# Install Python dependencies
COPY requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x entrypoint.sh

COPY feature_store.yaml /app/feature_store.yaml
COPY account_features.py /app/account_features.py

# Set the entrypoint to run Feast
ENTRYPOINT ["./entrypoint.sh"]
feature_store.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
project: test
registry: # https://docs.feast.dev/reference/registries/sql
  registry_type: sql
  path: postgresql+psycopg://postgres:password@registry.feast.svc.cluster.local:5432
  cache_ttl_seconds: 60
  sqlalchemy_config_kwargs:
    echo: false
    pool_pre_ping: true
entity_key_serialization_version: 3
provider: local 
online_store: # https://docs.feast.dev/reference/online-stores/redis
  type: redis
  connection_string: "online-store.feast.svc.cluster.local:6379"
offline_store: # https://docs.feast.dev/reference/offline-stores/bigquery
  type: bigquery
  dataset: feast
  project_id: mlops-437709
  billing_project_id: mlops-437709
  location: US
entrypoint.sh
#!/bin/bash
set -e

echo "Running feast apply..."
feast apply

echo "Starting feast server..."
exec feast serve --host 0.0.0.0 --port 8080

Build image

docker buildx build \
  --platform linux/amd64 \
  -t feast:v0.1.0 \
  -t feast:latest \
  .

我是使用MacOS，我們使用docker buildx，for amd64架構

確認是否建置成功

docker images --filter=reference="feast*"

REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
feast        v0.1.0    436abcf371e9   55 seconds ago   596MB

Load Image into Minikube Cluster

在Local進入minikube

minikube ssh

查看目前minikube裡有的images，沒有feast 

docker@minikube:~$ docker images

Expected Output

REPOSITORY                                TAG        IMAGE ID       CREATED         SIZE
registry.k8s.io/kube-apiserver            v1.30.0    181f57fd3cdb   12 months ago   112MB
registry.k8s.io/kube-controller-manager   v1.30.0    68feac521c0f   12 months ago   107MB
registry.k8s.io/kube-proxy                v1.30.0    cb7eac0b42cc   12 months ago   87.9MB
registry.k8s.io/kube-scheduler            v1.30.0    547adae34140   12 months ago   60.5MB
registry.k8s.io/etcd                      3.5.12-0   014faa467e29   15 months ago   139MB
registry.k8s.io/coredns/coredns           v1.11.1    2437cf762177   21 months ago   57.4MB
registry.k8s.io/pause                     3.9        829e9de338bd   2 years ago     514kB
gcr.io/k8s-minikube/storage-provisioner   v5         ba04bb24b957   4 years ago     29MB
docker@minikube:~$

在local開啟另個terminal，將local的docker image載入到minikube cluster裡

minikube image load feast:v0.1.0

回到minikube裡，查看images，有feast了

docker@minikube:~$ docker images

Expected Output

REPOSITORY                                TAG        IMAGE ID       CREATED          SIZE
feast                                     v0.1.0     436abcf371e9   33 minutes ago   596MB
registry.k8s.io/kube-apiserver            v1.30.0    181f57fd3cdb   12 months ago    112MB
registry.k8s.io/kube-controller-manager   v1.30.0    68feac521c0f   12 months ago    107MB
registry.k8s.io/kube-scheduler            v1.30.0    547adae34140   12 months ago    60.5MB
registry.k8s.io/kube-proxy                v1.30.0    cb7eac0b42cc   12 months ago    87.9MB
registry.k8s.io/etcd                      3.5.12-0   014faa467e29   15 months ago    139MB
registry.k8s.io/coredns/coredns           v1.11.1    2437cf762177   21 months ago    57.4MB
registry.k8s.io/pause                     3.9        829e9de338bd   2 years ago      514kB
gcr.io/k8s-minikube/storage-provisioner   v5         ba04bb24b957   4 years ago      29MB

K8S

Registry

registry.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: feast
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      containers:
        - name: registry
          image: postgres
          env:
            - name: POSTGRES_DB
              value: feast
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: password
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: storage
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: storage
          hostPath:
            path: /home/docker/data/feast/registry
            type: DirectoryOrCreate          
---
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: feast
spec:
  selector:
    app: registry
  type: ClusterIP
  ports:
    - port: 5432
      targetPort: 5432

Online Store

online-store.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: online-store
  namespace: feast
spec:
  replicas: 1
  selector:
    matchLabels:
      app: online-store
  template:
    metadata:
      labels:
        app: online-store
    spec:
      containers:
        - name: online-store
          image: redis
          ports:
            - containerPort: 6379
              protocol: TCP
          volumeMounts:
            - name: storage
              mountPath: /data
      restartPolicy: Always
      volumes:
        - name: storage
          hostPath:
            path: /home/docker/data/feast/online-store
            type: DirectoryOrCreate
---
apiVersion: v1
kind: Service
metadata:
  name: online-store
  namespace: feast
spec:
  selector:
    app: online-store
  type: ClusterIP
  ports:
    - port: 6379
      targetPort: 6379

Secret

Create GCP Service Account Key

確認Project id

gcloud config get-value project

Expected Output

mlops-437709

建立Service Account

gcloud iam service-accounts create feast-sa \
    --display-name "Feast Service Account"

Expected Output

Reauthentication required.
Please enter your password:
Reauthentication successful.
Created service account [feast-sa].

替Service Account加上BigQuery權限

gcloud projects add-iam-policy-binding mlops-437709 \
    --member="serviceAccount:feast-sa@mlops-437709.iam.gserviceaccount.com" \
    --role="roles/bigquery.admin"
Expected Output 
[1] EXPRESSION=request.time < timestamp("2025-04-09T07:42:05.596Z"), TITLE=cloudbuild-connection-setup
[2] None
[3] Specify a new condition
The policy contains bindings with conditions, so specifying a condition is required when adding a binding. Please specify a condition.:  2

Updated IAM policy for project [mlops-437709].
bindings:
- members:
  - serviceAccount:service-362026176730@gcp-sa-aiplatform-cc.iam.gserviceaccount.com
  role: roles/aiplatform.customCodeServiceAgent
- members:
  - serviceAccount:service-362026176730@gcp-sa-vertex-op.iam.gserviceaccount.com
  role: roles/aiplatform.onlinePredictionServiceAgent
- members:
  - serviceAccount:service-362026176730@gcp-sa-aiplatform.iam.gserviceaccount.com
  role: roles/aiplatform.serviceAgent
- members:
  - serviceAccount:service-362026176730@gcp-sa-artifactregistry.iam.gserviceaccount.com
  role: roles/artifactregistry.serviceAgent
- members:
  - serviceAccount:feast-sa@mlops-437709.iam.gserviceaccount.com
  role: roles/bigquery.admin
- members:
  - serviceAccount:362026176730@cloudbuild.gserviceaccount.com
  role: roles/cloudbuild.builds.builder
- members:
  - serviceAccount:service-362026176730@gcp-sa-cloudbuild.iam.gserviceaccount.com
  role: roles/cloudbuild.serviceAgent
- members:
  - serviceAccount:service-362026176730@containerregistry.iam.gserviceaccount.com
  role: roles/containerregistry.ServiceAgent
- members:
  - serviceAccount:362026176730-compute@developer.gserviceaccount.com
  role: roles/editor
- members:
  - serviceAccount:service-362026176730@gcp-sa-firestore.iam.gserviceaccount.com
  role: roles/firestore.serviceAgent
- members:
  - serviceAccount:362026176730@cloudbuild.gserviceaccount.com
  role: roles/iam.serviceAccountUser
- members:
  - serviceAccount:service-362026176730@cloud-ml.google.com.iam.gserviceaccount.com
  role: roles/ml.serviceAgent
- members:
  - user:edison@kcl10.com
  role: roles/owner
- members:
  - serviceAccount:service-362026176730@gcp-sa-pubsub.iam.gserviceaccount.com
  role: roles/pubsub.serviceAgent
- members:
  - serviceAccount:362026176730@cloudbuild.gserviceaccount.com
  role: roles/run.admin
- members:
  - serviceAccount:service-362026176730@serverless-robot-prod.iam.gserviceaccount.com
  role: roles/run.serviceAgent
- condition:
    expression: request.time < timestamp("2025-04-09T07:42:05.596Z")
    title: cloudbuild-connection-setup
  members:
  - serviceAccount:service-362026176730@gcp-sa-cloudbuild.iam.gserviceaccount.com
  role: roles/secretmanager.admin
etag: BwY0his6v7Y=
version: 3

建立Key

gcloud iam service-accounts keys create feast-gcp-key.json \
    --iam-account=feast-sa@mlops-437709.iam.gserviceaccount.com

Expected Output

created key [a2609fffff05f5fdf311de233f1a2e1e89288ab5] of type [json] as [feast-gcp-key.json] for [feast-sa@mlops-437709.iam.gserviceaccount.com]

Create K8S Secret

cat feast-gcp-key.json | base64

apiVersion: v1
kind: Secret
metadata:
  name: feast-gcp-key
  namespace: feast
type: Opaque
data:
  key.json: <base64 encoded string>

Online Feature Server

online-feature-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: online-feature-server
  namespace: feast
spec:
  replicas: 1
  selector:
    matchLabels:
      app: online-feature-server
  template:
    metadata:
      labels:
        app: online-feature-server
    spec:
      containers:
        - name: online-feature-server
          image: feast:v0.1.8
          ports:
            - containerPort: 8080
          env:
            - name: FEAST_FEATURE_SERVER_CONFIG_PATH
              value: /app/config/feature_store.yaml
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /var/secrets/google/key.json
          volumeMounts:
            - name: gcp-sa-key
              mountPath: /var/secrets/google
              readOnly: true
      volumes:
        - name: gcp-sa-key
          secret:
            secretName: gcp-sa-key
---
apiVersion: v1
kind: Service
metadata:
  name: online-feature-server
  namespace: feast
spec:
  selector:
    app: online-feature-server
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30000

Deploy Your Feast Architecture

install.sh
#!/bin/bash

echo "Step 0: Create feast Namespace"
kubectl create ns feast

echo "Step 1: Deploy Online Store"
kubectl apply -f online-store.yaml
kubectl rollout status deployment/online-store -n feast

echo "Step 2: Deploy Registry"
kubectl apply -f registry.yaml
kubectl rollout status deployment/registry -n feast

echo "Step 3: Deploy Feast Online Feature Server"
kubectl apply -f secret.yaml
kubectl apply -f online-feature-server.yaml
kubectl rollout status deployment/online-feature-server -n feast

./install.sh

Expected Output

Step 0: Create feast Namespace
namespace/feast created

Step 1: Deploy Online Store
deployment.apps/online-store created
service/online-store created
Waiting for deployment "online-store" rollout to finish: 0 of 1 updated replicas are available...
deployment "online-store" successfully rolled out

Step 2: Deploy Registry
deployment.apps/registry created
service/registry created
Waiting for deployment "registry" rollout to finish: 0 of 1 updated replicas are available...
deployment "registry" successfully rolled out

Step 3: Deploy Feast Online Feature Server
secret/gcp-sa-key created
deployment.apps/online-feature-server created
service/online-feature-server created
Waiting for deployment "online-feature-server" rollout to finish: 0 of 1 updated replicas are available...
deployment "online-feature-server" successfully rolled out

Test Your Feast Online Feature Server

首先要先將service 透過minikube service output給Local使用

minikube service online-feature-server -n feast

Expected Output

|-----------|-----------------------|-------------|---------------------------|
| NAMESPACE |         NAME          | TARGET PORT |            URL            |
|-----------|-----------------------|-------------|---------------------------|
| feast     | online-feature-server |        8080 | http://192.168.49.2:30000 |
|-----------|-----------------------|-------------|---------------------------|
🏃  Starting tunnel for service online-feature-server.
|-----------|-----------------------|-------------|------------------------|
| NAMESPACE |         NAME          | TARGET PORT |          URL           |
|-----------|-----------------------|-------------|------------------------|
| feast     | online-feature-server |             | http://127.0.0.1:52316 |
|-----------|-----------------------|-------------|------------------------|
🎉  Opening service feast/online-feature-server in default browser...
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

http://127.0.0.1:52316就是我們在localhost可以access得到的URL，待會測試時都會需要打這個URL

Materialize Features to Online Store

/materialize

Materialize features within a specified time range

curl -X POST http://127.0.0.1:52316/materialize \
     -H "Content-Type: application/json" \
     -d @request-materialize.json

where request-materialize.json is

{"start_ts": "2024-08-07T00:00:00Z", "end_ts": "2024-08-08T00:00:00Z"}

by running kubectl logs pod, you can see the progress of materialization:

Expected Output

Materializing 3 feature views from 2024-08-07 00:00:00+00:00 to 2024-08-08 00:00:00+00:00 into the redis online store.

acct_fraud_7d:
0it [00:00, ?it/s]
acct_profiles:
0it [00:00, ?it/s]
acct_num_txns_7d:
0it [00:00, ?it/s]
10.244.0.1:39638 - "POST /materialize HTTP/1.1" 200

/materialize-incremental

Incrementally materialize features up to a specified timestamp

curl -X POST http://127.0.0.1:52316/materialize-incremental \
     -H "Content-Type: application/json" \
     -d @request-materialize-incremental.json

where request-materialize-incremental.json is

{"end_ts": "2025-05-27T07:00:00"}

by running kubectl logs pod, you can see the progress of materialization:

Expected Output

acct_fraud_7d from 2025-05-11 11:56:07+00:00 to 2025-05-27 07:00:00+00:00:
100%|█████████████████████████████████████████████████████████| 9944/9944 [00:02<00:00, 3984.00it/s]
acct_profiles from 2025-05-11 11:56:07+00:00 to 2025-05-27 07:00:00+00:00:
100%|████████████████████████████████████████████████████████| 9944/9944 [00:00<00:00, 14760.64it/s]
acct_num_txns_7d from 2025-05-11 11:56:07+00:00 to 2025-05-27 07:00:00+00:00:
100%|████████████████████████████████████████████████████████| 9944/9944 [00:00<00:00, 17575.82it/s]
10.244.0.1:23621 - "POST /materialize-incremental HTTP/1.1" 200

Get Online Features

/get-online-features

Get online features from the feature store:

curl -X POST http://127.0.0.1:52316/get-online-features \
     -H "Content-Type: application/json" \
     -d @request-get-online-features.json | jq

where request-get-online-features.json is 

{
  "feature_service": "fraud_detection_v1",
  "entities": {
    "entity_id": ["v5zlw0", "000q95"]
  }
}

Expected Output

{
  "metadata": {
    "feature_names": [
      "entity_id",
      "has_fraud_7d",
      "num_transactions_7d",
      "account_age_days",
      "credit_score",
      "has_2fa_installed"
    ]
  },
  "results": [
    {
      "values": [
        "v5zlw0",
        "000q95"
      ],
      "statuses": [
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z"
      ]
    },
    {
      "values": [
        false,
        false
      ],
      "statuses": [
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "2025-05-26T22:00:37Z",
        "2025-05-26T22:00:37Z"
      ]
    },
    {
      "values": [
        7,
        6
      ],
      "statuses": [
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "2025-05-26T00:00:00Z",
        "2025-05-26T00:00:00Z"
      ]
    },
    {
      "values": [
        655,
        236
      ],
      "statuses": [
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "2025-05-19T22:00:27Z",
        "2025-05-19T22:00:27Z"
      ]
    },
    {
      "values": [
        480,
        737
      ],
      "statuses": [
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "2025-05-19T22:00:27Z",
        "2025-05-19T22:00:27Z"
      ]
    },
    {
      "values": [
        true,
        true
      ],
      "statuses": [
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "2025-05-19T22:00:27Z",
        "2025-05-19T22:00:27Z"
      ]
    }
  ]
}

https://docs.feast.dev/reference/feature-servers/python-feature-server