Semantic Traffic Management

Asegurarse de haber realizado la Instalación de un Redis Container

Lanzar un contenedor de Redis Server

docker run -d --name redis --network kong -p 6379:6379 redis:8.0

Revisar Containers en Ejecución

Revisar que todos los conteendores se encuentren en ejecución de forma correcta

docker ps -a

Configuraciones

1. Obtener un Personal Access Token dando click en el link que se muestra en la imágen a continuación:
2. Dar click en el botón +Generate Token
3. Colocar de nombre workshop y 30 days de Expiración.
4. Dar click en el botón Generate, copiar y guardar en un lugar seguro el Access Token generado:
5. Solicitar al instructor las siguientes ACCESS_KEYS: OPENAI_API_KEY y GROQ_API_KEY
6. Extraer el ID del Control Plane, el cual se encuentra en: Gateway Manager > workshop-itm > Overview Copiar y guardar en un lugar seguro.
7. Conectarse por medio de SSH al servidor Docker y exportal las siguientes variables de entorno, las cuales deberán de estar seteadas siempre que se necesite realizar configuraciones por medio de Konnect:

   export KONNECT_TOKEN=kpat_s1Yja.......
   export OPENAI_API_KEY=sk-proj-3j2bI...
   export GROQ_API_KEY=gsk_ebd...........
   export CONTROL_PLANE_ID=9346a65b-9....
   

8. Crear un nuevo servicio con el siguiente comando en el servidor Docker:

   curl -X POST https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/services \
     --header "accept: application/json" \
     --header "Content-Type: application/json" \
     --header "Authorization: Bearer $KONNECT_TOKEN" \
     --data '
   {
     "name": "semantic-load-balancing",
     "host": "localhost",
     "port": 80,
     "protocol": "http",
     "retries": 5,
     "connect_timeout": 60000,
     "read_timeout": 60000,
     "write_timeout": 60000,
     "tags": [
       "ai-manager-created",
       "ai-manager-name: semantic-load-balancing"
     ]
   }'
   

9. Extraer el nombre ID del servicio creado anteriormente, ejecutar el siguiente comando en el servidor Docker:

   SERVICE_ID=$(curl -s -X GET https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/services \
     --header "accept: application/json" \
     --header "Authorization: Bearer $KONNECT_TOKEN" \
     | jq -r '.data[] | select(.name == "semantic-load-balancing") | .id')
   

10. Revisar el ID del servicio:

    echo "$SERVICE_ID"
    

11. Crear una nueva Route con el siguiente comando en el servidor Docker:

    curl --request POST \
      --url https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/routes \
      --header "Accept: application/json" \
      --header "Authorization: Bearer $KONNECT_TOKEN" \
      --header "Content-Type: application/json" \
      --data "{
      \"name\": \"semantic-load-balancing\",
      \"paths\": [\"/anything\"],
      \"service\": {
        \"id\": \"$SERVICE_ID\"
      }
    }"
    

12. Configurar un plugin ai-proxy-advanced, ejecutar el siguiente comando en el servidor Docker:

    curl -X POST https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/services/$SERVICE_ID/plugins/ \
        --header "accept: application/json" \
        --header "Content-Type: application/json" \
        --header "Authorization: Bearer $KONNECT_TOKEN" \
        --data '
        {
          "name": "ai-proxy-advanced",
          "config": {
            "embeddings": {
              "auth": {
                "header_name": "Authorization",
                "header_value": "Bearer '$OPENAI_API_KEY'"
              },
              "model": {
                "provider": "openai",
                "name": "text-embedding-3-large",
                "options": {
                  "upstream_url": "https://api.openai.com/v1/embeddings"
                }
              }
            },
            "vectordb": {
              "dimensions": 3072,
              "distance_metric": "cosine",
              "strategy": "redis",
              "threshold": 0.75,
              "redis": {
                "host": "redis",
                "port": 6379
              }
            },
            "balancer": {
              "algorithm": "semantic"
            },
            "targets": [
              {
                "model": {
                  "name": "gpt-3.5-turbo",
                  "provider": "openai",
                  "options": {
                    "max_tokens": 826,
                    "temperature": 0
                  }
                },
                "route_type": "llm/v1/chat",
                "auth": {
                  "header_name": "Authorization",
                  "header_value": "Bearer '$OPENAI_API_KEY'"
                },
                "description": "Expert in Python programming."
              },
              {
                "model": {
                  "name": "tinyllama",
                  "provider": "llama2",
                  "options": {
                    "llama2_format": "ollama",
                    "upstream_url": "http://ollama:8080/api/chat"
                  }
                },
                "route_type": "llm/v1/chat",
                "description": "Expert in Docker Containers"
              },
              {
               "model": {
                 "name": "llama-3.1-8b-instant",
                 "provider": "openai",
                 "options": {
                   "upstream_url": "https://api.groq.com/openai/v1/chat/completions"
                 }
               },
               "route_type": "llm/v1/chat",
               "auth": {
                 "header_name": "Authorization",
                 "header_value": "Bearer '$GROQ_API_KEY'"
              },
              "description": "Expert in Kubernetes"
              },
              {
                "model": {
                  "name": "gpt-4o",
                  "provider": "openai",
                  "options": {
                    "max_tokens": 512,
                    "temperature": 0.3
                  }
                },
                "route_type": "llm/v1/chat",
                "auth": {
                  "header_name": "Authorization",
                  "header_value": "Bearer '$OPENAI_API_KEY'"
                },
                "description": "All IT support questions."
              },
              {
                "model": {
                  "name": "gpt-4o-mini",
                  "provider": "openai",
                  "options": {
                    "max_tokens": 256,
                    "temperature": 1.0
                  }
                },
                "route_type": "llm/v1/chat",
                "auth": {
                  "header_name": "Authorization",
                  "header_value": "Bearer '$OPENAI_API_KEY'"
                },
                "description": "CATCHALL"
              }
            ]
          }
        }
        '
    

13. Configurar un plugin ai-semantic-cache, ejecutar el siguiente comando en el servidor Docker:

    curl -X POST https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/services/$SERVICE_ID/plugins/ \
        --header "accept: application/json" \
        --header "Content-Type: application/json" \
        --header "Authorization: Bearer $KONNECT_TOKEN" \
        --data '
        {
          "name": "ai-semantic-cache",
          "config": {
            "embeddings": {
              "auth": {
                "header_name": "Authorization",
                "header_value": "Bearer '$OPENAI_API_KEY'"
              },
              "model": {
                "provider": "openai",
                "name": "text-embedding-3-large",
                "options": {
                  "upstream_url": "https://api.openai.com/v1/embeddings"
                }
              }
            },
            "vectordb": {
              "dimensions": 3072,
              "distance_metric": "cosine",
              "strategy": "redis",
              "threshold": 0.3,
              "redis": {
                "host": "redis",
                "port": 6379
              }
            }
          }
        }
        '
    

Pruebas de funcionamiento

Ejecutar los siguientes comandos en el servidor Docker.

gpt-4o-mini (General)

curl -X POST http://student-0-aio.34-82-238-123.nip.io:30080/anything \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "Tell me a joke about AI." }
    ]
  }'

gpt-3.5-turbo (Expert in Python programming.)

curl -X POST http://student-0-aio.34-82-238-123.nip.io:30080/anything \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "How to implement a custom iterator class in Python" }
    ]
  }'

gpt-4o (All IT support questions.)

curl -X POST http://student-0-aio.34-82-238-123.nip.io:30080/anything \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "How can I configure my corporate VPN?" }
    ]
  }'

llama-3.1-8b-instant (Expert in Kubernetes)

curl -X POST http://student-0-aio.34-82-238-123.nip.io:30080/anything \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "How can I create a Pod in Kubernetes?" }
    ]
  }'

TinyLlama (Expert in Docker Containers)

curl -X POST http://student-0-aio.34-82-238-123.nip.io:30080/anything \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "How can I create a volume in Docker?" }
    ]
  }'

TinyLlama (Expert in Docker Containers)

curl -X POST http://student-0-aio.34-82-238-123.nip.io:30080/anything \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "How can I create a persistent storage in Docker?" }
    ]
  }'