npuserver

Usage Examples

The npuserver provides a standard REST API that is fully compatible with OpenAI’s chat completions endpoint. This means you can use standard HTTP clients, the official openai Python package, or just raw curl commands to interact with it.

[!IMPORTANT] The server has strict memory management. You must explicitly load a model using the /load endpoint before you can send it chat completions requests.

1. Using the OpenAI Python Client

Since npuserver implements the /v1/chat/completions endpoint, you can drop it into any existing application that uses the OpenAI client by simply changing the base_url.

Prerequisites

pip install openai

Example

from openai import OpenAI
import requests

# 1. First, explicitly load the model into NPU memory via REST
requests.post("http://localhost:8080/load", json={
    "model": "durgasai299792458/Qwen3-0.6BOpenVINO"
})

# 2. Configure the OpenAI client to point to the local NPU server
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed" # API keys are not required for local use
)

# 3. Generate a chat completion (Streaming is fully supported!)
response = client.chat.completions.create(
    model="durgasai299792458/Qwen3-0.6BOpenVINO",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a short poem about Intel NPUs."}
    ],
    temperature=0.7,
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

2. Using standard Python Requests

If you don’t want to use the OpenAI client, you can just use requests. This is especially useful for the memory management endpoints (/load, /unload, /models).

import requests

base_url = "http://localhost:8080"

# List available models
response = requests.get(f"{base_url}/models")
print("Models:", response.json())

# Download and compile a model from Hugging Face (blocks until complete)
requests.post(f"{base_url}/download", json={
    "model": "durgasai299792458/Qwen3-0.6BOpenVINO",
    "max_prompt_len": 2048,
    "allow_download": True
})

# Load a model
requests.post(f"{base_url}/load", json={"model": "durgasai299792458/Qwen3-0.6BOpenVINO"})

# Chat Completion
payload = {
    "messages": [
        {"role": "user", "content": "Hello, world!"}
    ],
    "max_tokens": 100,
    "stream": False
}
response = requests.post(f"{base_url}/v1/chat/completions", json=payload)
print(response.json()['choices'][0]['message']['content'])

# Unload the model when done to free NPU memory
requests.post(f"{base_url}/unload")

# Delete only compiled files
requests.post(f"{base_url}/delete", json={"model": "durgasai299792458/Qwen3-0.6BOpenVINO", "compiled_only": True})

# Delete completely (compiled files, downloaded snapshots, and lock files)
requests.post(f"{base_url}/delete", json={"model": "durgasai299792458/Qwen3-0.6BOpenVINO", "compiled_only": False})

3. Using cURL

For quick tests in your terminal, curl is perfect.

Load a Model

curl -X POST http://localhost:8080/load \
     -H "Content-Type: application/json" \
     -d '{"model": "durgasai299792458/Qwen3-0.6BOpenVINO"}'

Generate a Response

curl -X POST http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "messages": [
         {"role": "user", "content": "Explain quantum computing in one sentence."}
       ],
       "max_tokens": 512,
       "stream": false
     }'

Unload Model

curl -X POST http://localhost:8080/unload

Delete Model (Compiled Only)

curl -X POST http://localhost:8080/delete \
     -H "Content-Type: application/json" \
     -d '{"model": "durgasai299792458/Qwen3-0.6BOpenVINO", "compiled_only": true}'

Delete Model (Complete snapshot & compiled)

curl -X DELETE http://localhost:8080/v1/models/durgasai299792458/Qwen3-0.6BOpenVINO

4. Using the npuserver Client Library

The package includes built-in client functions to manage model lifecycle programmatically from python without manually writing HTTP requests.

import npuserver

# Download and compile a model from Hugging Face.
# Blocks until complete — no timeout by default since large models can take many minutes.
npuserver.download_model("durgasai299792458/Qwen3-0.6BOpenVINO")

# Or with an explicit upper bound (e.g. 30 minutes)
npuserver.download_model("durgasai299792458/Qwen3-0.6BOpenVINO", timeout=1800)

# Load model into NPU memory
npuserver.load_model("durgasai299792458/Qwen3-0.6BOpenVINO")

# Unload model
npuserver.unload_model()

# Delete compiled files only
npuserver.delete_compiled("durgasai299792458/Qwen3-0.6BOpenVINO")

# Delete both compiled files and downloaded snapshots completely (including lock files)
npuserver.delete("durgasai299792458/Qwen3-0.6BOpenVINO")