AI Pyramid OpenAI-Compatible API Chat Example

This example explains how to deploy a StackFlow chat model and the OpenAI-compatible API on AI Pyramid, then invoke the local model through cURL, the Python SDK, and Chatbox.

Preparation

Refer to AI Pyramid Software Package Update to complete the installation of the following model packages and software packages.

apt install lib-llm llm-sys llm-llm llm-openai-api

apt install llm-model-qwen2.5-1.5b-int4-ax650

Note

After installing a new model each time, you need to manually execute systemctl restart llm-openai-api to update the model list.

Curl Call

curl http://127.0.0.1:8000/v1/models \
  -H "Content-Type: application/json"

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-xxxxxxxx" \
  -d '{
    "model": "qwen2.5-1.5B-Int4-ax650",
    "messages": [
      {"role": "developer", "content": "You are a helpful home assistant."},
      {"role": "user", "content": "Write a one-sentence bedtime story about a unicorn."}
    ]
  }' 

Python Call

from openai import OpenAI
client = OpenAI(
    api_key="sk-",
    base_url="http://127.0.0.1:8000/v1"
)

client.models.list()
print(client.models.list()) 

from openai import OpenAI
client = OpenAI(
    api_key="sk-",
    base_url="http://127.0.0.1:8000/v1"
)

completion = client.chat.completions.create(
  model="qwen2.5-1.5B-Int4-ax650",
  messages=[
    {"role": "developer", "content": "You are a helpful home assistant."},
    {"role": "user", "content": "Turn on the light!"}
  ]
)

print(completion.choices[0].message) 