What is SearXNG?

SearXNG is a free, open-source, self-hosted metasearch engine that aggregates results from multiple search engines without tracking users. It solves the problems of privacy invasion, vendor lock-in, API costs, and search bias by giving developers full control over search data, sources, and usage.

Here are the key benefits of SearXNG:

• Privacy-first: No user tracking, logging, or profiling.
• Self-hosted control: You own the data, configuration, and infrastructure.
• Multiple search engines: Aggregates results from many sources(Google,Bing,DuckDuckGo etc) in one query.
• Free & open source: No API costs, no vendor lock-in.
• API-friendly: JSON output works well for apps and LLM pipelines.
• Customizable: Enable/disable engines, formats, languages, and filters.
• Lightweight & fast: Runs efficiently with minimal resources.
• LLM-ready: Ideal for RAG, research agents, and private AI workflows.

Step 1: Install Docker and Docker Compose

First we need to install SerXNG, we will use docker based installation and use.

• For Windows users, Download docker desktop from docker’s official site
• Linux users should install docker and docker compose by following the guide

Step 2: Install SearXNG

Make a directory named “searxng”, and create file named docker-compose.yml inside this newly directory. Copy paste the below code.

version: "3.7"

services:
  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    ports:
      - "8080:8080"
    volumes:
      - ./searxng:/etc/searxng
    environment:
      - SEARXNG_BASE_URL=http://localhost:8080/
    restart: unless-stopped

Next run the below command :

docker-compose up

Wait for the installation to finish, and check the browser by operning the URL http://localhost:8080/.

Step 3: Enable API in SearXNG

To enable and use the API, first install and configure SearXNG, then add JSON support in settings. We need edit the searxng\searxng\settings.yaml file add the json format in the formats settings as shown below :

search:
  formats:
    - html
    - json  # Add this for API
server:
  secret_key: "your-random-secret-key" 

Save the settings file and restart the SearXNG container.

docker-compose restart

Now we can make API requests to SearXNG enpoints, for more details about API please refer here .

Conclusion

It is easier to setup SearXNGand expose its API endpoints, thanks to docker. Run Large Language Models Locally with Simple APIs

The post How to use SearXNG as a Private Search API – Step-by-Step Guide appeared first on NoloWiz.

First update the system by running the below commands :

sudo apt update
sudo apt upgrade

We need to disable Nouveau driver,Nouveau is an accelerated open-source graphics driver for NVIDIA GPUs, developed independently by the free software community to provide libre support without relying on NVIDIA’s proprietary drivers. Disable Nouveau driver by creating a blacklist file:

sudo bash -c "echo -e 'blacklist nouveau\noptions nouveau modeset=0' > /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
sudo update-initramfs -u

Next, reboot the system eithe reboot manually or run the below command :

sudo reboot

Install the prerequisites by running the command :

sudo apt install build-essential libglvnd-dev pkg-config

Add the graphics drivers PPA to get the latest official NVIDIA drivers:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update

Check available NVIDIA drivers and install the recommended one:

ubuntu-drivers devices

Install latest/recommended driver version, I’m using NVIDIA RTX 3060 graphics card so using the driver nvidia-driver-570 :

sudo apt install nvidia-driver-570

Alternatively, to auto install the CUDA driver run the below command.

sudo ubuntu-drivers autoinstall

Verify the installation by running the below command :

nvidia-smi

Conclusion

By following these simple steps it is easy to install CUDA driver on Ubuntu. Read about TP-LINK AC1300 Archer T3U Ubuntu Driver Installation .

The post How to Install CUDA Driver on Ubuntu 24.04 LTS or Higher – A step-by-step guide appeared first on NoloWiz.

Regression Bug

A regression bug is an error that appears in software after a change like a new feature, update, or fix causing previously working functionality to break.It indicates the system has “regressed” in quality. This can happen in typical software development.

What is Git Bisect?

Let’s understand the problem :

We know a feature was working before, now it’s broken. How do we find which commit broke it – without checking every commit?

The Solution is git bisect!

Git Bisect helps find the exact commit that introduced a bug using binary search.

The Core Idea Behind Git Bisect

Binary Search is a fast way to find something inside a sorted list. Instead of checking items one by one, it splits the list in half each time reducing search time from O(n) to O(log n).

Let’s see an example searching for 7 in a sorted list as shown below.

List: [1, 2, 3, 4, 5, 6, 7, 8]

Search for “7”
              ↓
Step 1 → middle = 4 (not found)
Search right half → [5,6,7,8]

Step 2 → middle = 6 (not found)
Search right half → [7,8]

Step 3 → middle = 7 (FOUND)

Instead of checking 8 items, it found the answer in just 3 checks.

The same technique is used in git bisect to search for the commit that caused the bug.

How to use git bisect

Suppose we have simple calculator app with following commits. (We introduced a bug in commit 57cd4b8 for this demo)

The latest code is shown below commit ID : ac6beeb

def add(a :int, b :int) -> int:
    return a - b

def subtract(a :int, b :int) -> int:
    return a - b    

def multiply(a :int, b :int) -> int:
    return a * b

def divide(a :int, b :int) -> float:
    if b == 0:
        raise ValueError("Cannot divide by zero.")
    return a / b

def modulus(a :int, b :int) -> int:
    return a % b

if __name__ == "__main__":
    result = add(3, 5)
    print(f"The sum of 3 and 5 is: {result}")

We can see that add function is wrong when we run the application we will get the below output

Now we have a regression bug, we need to find out the which commit introduced this bug. To finding this bug manually we need to test each commit for the bug, but git bisect will help us to find the buggy commit faster.

First run the below command to list all the commits with commit IDs.

git log --oneline

Run the below command to start the git bisect process :

git bisect start

Next we need to give one known good commit where the feature(additon) worked and bad commit. We are going to use use the first commit for this demo (You can give previous good release commit ID)

git bisect good 8714580

Finally, we need to give a bad commit for this demo Im using latest commit ID ac6beeb :

git bisect bad ac6beeb

From the above logs we can see that bisecting started current commit is “update readme”. Let’s check app output

It is working fine – the answser is correct, next we need to mark it as good by running the below command

git bisect good

We are now moved to another commit that is “Update readme with supported operations”, again run the application or run unit tests.

Yes we got bad commit, mark it using the below command.

git bisect bad

We finally found the commit that caused the bug that is “Implement multiplication”, commit ID : 57cd4b8.

Finally , we need to exit git from bisect mode by running the below command

git bisect reset

Conclusion

With Git Bisect, you let Git do the detective work for you – using the power of binary search to find the exact commit that introduced a bug. It’s fast, reliable, and perfect for regression issues. I hope this tutorial helpful for you. Learn the difference between git rebase and git merge.

The post Regression Bug? Learn Git Bisect to Find the Problem Commit appeared first on NoloWiz.

Project

In this demo, we are going to create a project named “Cool Project” . It has a FastAPI-based API server and client app. For app-to-app authentication/authorization, we will use the OAuth2 client credentials flow.

OAuth 2.0 Client Credentials Flow Specification – RFC 6749

The components of our project are :

• Resource server – FastAPI server
• Client – Client app ( For the simplicity of this demo, we will use Postman to call the APIs).
• Authorization Server – Keycloak

We will use the client ID and client secret of the client app to get an access token from the Keycloak authorization server. We will use this access token to call the FastAPI server APIs.

Keycloak

Keycloak is an open-source identity and access management solution that provides single sign-on (SSO), authentication, and authorization for web and mobile applications. It supports modern security protocols like OAuth2 and OpenID Connect, integrates with LDAP and social logins, and offers centralized user management and customization. Keycloak supports the OAuth2 Client Credentials flow, which can be used for app-to-app authentication and authorization.

Start Keycloak

First, we need to download Keycloak.

Download the Keycloak zip file from here and extract the zip file. We are using Keycloak version 26.3.

After extracting the zip, enter the keycloak directory and run the following command in the command prompt:

bin\kc.bat start-dev

Please note that we are running Keycloak in development mode. Open the URL http://localhost:8080/ in a browser. For this demo, we’re going to create an admin user.

Create an admin user with a username admin and password as adminpass Then log in to Keycloak using the admin user.

Create a realm for the project

Next, we need to create a realm for our project. A realm in Keycloak is an isolated security and administrative domain where you manage users, applications, roles, and groups. In simple terms, we can think of it as a project with all the details.. Each realm controls authentication and authorization for its contained resources, ensuring that users and configurations are separated from other realms, enabling multi-tenancy and organized management.

Let’s create a cool project realm. Click on “Manage realms” on the left sidebar and create a realm as shown below.

Create client

Next, we need to create a client in the coolproject realm.

Click next, and on the next page, we need to turn on “Client authentication” and “Service account roles” as shown below :

The above settings enable us to use the OAuth2 client credentials flow.

Click next and leave everything as it is, and click save. Now, click on clients again on the left sidebar, as we can see the newly created client “client-app” listed as shown below :

Client ID and Client Secret

In the OAuth2 Client credentials flow, we need the client ID and client secret.

Open the client-app from the left sidebar and we can see that Client ID is “client-app” as shown below :

Next, we need the client secret. Click the credentials tab.

Open the http://localhost:8080/realms/coolproject/.well-known/openid-configuration in a browser. We can see the token endpoint as shown below. We will use this URL in Postman to get an access token for the client app.

Add audience claim

In the Client Credentials flow, there is no user; it’s an application-to-application (machine-to-machine) authentication. The audience (aud) claim in the access token typically refers to the resource server, that is, the application or API that will receive and validate the token.

We need to add the audience claim to the access token payload. Open clients, then client scopes, and click one client-app-dedicated (a client scope that is ending dedicated).

Then turn off “Full scope allowed” (It will remove the default audience “account” ) as shown below.

Next, we will create a new client scope for the audience with value as the resource server name.

Next, open the client and add the newly created “api-audience” scope to the client-app. Open clients and client scopes, add api-audience as shown below:

Ensure that api-audience assigned type is “Default” as shown below.

Now our client app configuration is ready.

Start Resource Server

Next, we need to start the resource server. In this example, we are going to use a simple FastAPI application.

from fastapi import FastAPI, Depends, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from jose import jwt
import httpx

app = FastAPI()
security = HTTPBearer()


KEYCLOAK_REALM = "coolproject"
KEYCLOAK_URL = "http://localhost:8080"
KEYCLOAK_OPENID_CONFIG = (
    f"{KEYCLOAK_URL}/realms/{KEYCLOAK_REALM}/.well-known/openid-configuration"
)

JWKS = None
from jose.utils import base64url_decode
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.backends import default_backend


async def verify_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
    token = credentials.credentials
    jwks = await get_public_key()

    try:
        unverified_header = jwt.get_unverified_header(token)
        kid = unverified_header.get("kid")
        if not kid:
            raise Exception("Missing 'kid' in token header")

        key = next(k for k in jwks["keys"] if k["kid"] == kid)

        n = int.from_bytes(base64url_decode(key["n"].encode()), byteorder="big")
        e = int.from_bytes(base64url_decode(key["e"].encode()), byteorder="big")

        public_key = rsa.RSAPublicNumbers(e, n).public_key(default_backend())

        payload = jwt.decode(
            token,
            key=public_key,
            algorithms=["RS256"],
            audience="apiserver",
            issuer=f"{KEYCLOAK_URL}/realms/{KEYCLOAK_REALM}",
        )
        print(payload)
        return payload

    except Exception as e:
        raise HTTPException(status_code=401, detail=f"Invalid token: {str(e)}")


async def get_public_key():
    global JWKS
    if JWKS is None:
        async with httpx.AsyncClient() as client:
            r = await client.get(KEYCLOAK_OPENID_CONFIG)
            jwks_uri = r.json()["jwks_uri"]
            r2 = await client.get(jwks_uri)
            JWKS = r2.json()
    return JWKS


@app.get("/secure-data")
async def secure_data(user=Depends(verify_token)):
    return {"message": "You have access!", "client": user["sub"]}


if __name__ == "__main__":
    import uvicorn

    uvicorn.run("app:app", host="127.0.0.1", port=8000, reload=True)

Create a requirements.txt and install it

fastapi
httpx
python-jose[cryptography]
uvicorn

In the JWT decode, we need to give the audience as the “apiserver” that we configured

payload = jwt.decode(
            token,
            key=public_key,
            algorithms=["RS256"],
            audience="apiserver",
            issuer=f"{KEYCLOAK_URL}/realms/{KEYCLOAK_REALM}",
        )

Start the FastAPI server. Next, we will get an access token using the /token request as shown below :

Finally, we will use this access token to call the APIs as shown below :

We can see that the access token payload contains the “aud” claim.

Conclusion

In conclusion, it is easier to set up an authorization server using client credential flow in Keycloak. You can read about How to serve LLM using the OpenVINO Model Server on Windows.

The post App-to-App Authentication with Keycloak – Step by Step Guide appeared first on NoloWiz.

We know that OpenVINO is an open-source toolkit by Intel for optimizing and deploying deep learning models. It enhances model performance on Intel hardware, supports frameworks like PyTorch and TensorFlow, and offers tools for efficient inference across multiple platforms, including CPUs, GPUs, NPUs and VPUs.

Ensure the following dependencies are available on your Windows 11 machine :

• Python 3.11 or higher
• git
• curl (optional)

Model Conversion

We need to convert the model LLM model to OpenVINO. Create a folder and a Python virtual environment, for that open command prompt and run the below commands :

mkdir model_converter
cd model_converter
python -m venv env
env\Scripts\activate

Next, install the dependencies by running the below command :

pip install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt

Then download the export_model.py file to “model_converter” folder or use the below command :

curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py  --ssl-no-revoke -o export_model.py

Create a folder named “models” by running the below command :

mkdir models

Next, we need to convert the TinyLlama-1.1B-Chat-v1.0 model to OpenVINO using the below command :

python export_model.py text_generation --source_model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --config_file_path models/config.json --model_repository_path models 

After conversion, the OpenVINO TinyLlama model is available in the “models” directory. We need the full path of “config.json” to serve the model with the OpenVINO model server. The converted OpenVINO model weights are int4 quantized.

Deploy model with OpenVINO Model Server

OpenVINO Model Server is a high-performance system for serving optimized machine learning models over standard network protocols like REST and gRPC. Built with C++ and optimized for Intel hardware, it leverages the OpenVINO toolkit for accelerated inference. OVMS offers gRPC and REST APIs, compatible with TensorFlow Serving and KServe, simplifying integration. It supports diverse frameworks and model formats, enabling scalable and efficient deployment for AI applications like image recognition and NLP. OVMS is open-source, flexible, and designed for production environments.

Download the OpenVINO model server for Windows from the GitHub releases page here . We are using OpenVINO Model Server 2025.0 for this tutorial, extract the zip(ovms_windows.zip) file.

Open the command prompt enter into the “ovms_windows” folder and run the below commands:

setupvars.bat

Next, we can start the OVMS to serve our model by running the below command :

ovms --config_path "<<Your converted models config.js path>>"  --rest_port 8000

Now the OVMS running at http://localhost:8000. Verify model serving is ok by requesting the API endpoint http://localhost:8000/v1/config

As we can see that model is available since the state is “available” in the above API response.

Use LLM

OpenVINO model server provides chat completions API endpoint similar to OpenAI.The chat completion endpoint of OVMS is :

http://localhost:8000/v3/chat/completions

Install OpenAI package:

pip install openai

We can test the LLM using the below code :

from openai import OpenAI

client = OpenAI(
  base_url="http://localhost:8000/v3",
  api_key="unused"
)

stream = client.chat.completions.create(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    messages=[{"role": "user", "content": "What is opencv?"}],
    stream=True,
    temperature=0.1,
)
for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

Output

Conclusion

In short, deploying your LLM model on Windows is simplified with the OpenVINO model server. Just remember to convert your chosen model with the appropriate quantization for optimal performance.

The post How to serve LLM using the OpenVINO Model Server on Windows appeared first on NoloWiz.

Step 1: Download and Install FastSDCPU

In this demo, we are using a Windows 11 machine to test, first download FastSD CPU release. Extract the zip file and install it by following the steps :

• You need to install Python 3 and uv – fast package manager for python.
• Double click install.bat (It will take some time to install, depending on your internet speed.)
• After the installation close this command prompt window
• Double click the start-webserver.bat file to start FastSD CPU in webserver mode

Step 2: Download and Install ComfyUI

Next, we will download ComfyUI from this official repo and extract the zip file.

Step 3: Install ComfyUI extensions

Enter into ComfyUI_windows_portable\ComfyUI\custom_nodes folder and clone the following extensions :

• BetaDoggo/ComfyUI-FastSDCPU
• pythongosssss/ComfyUI-Custom-Scripts

Run the following command to install extensions:

git clone https://github.com/pythongosssss/ComfyUI-Custom-Scripts.git
git clone https://github.com/BetaDoggo/ComfyUI-FastSDCPU

Step 4: Faster generation of images using OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit for optimizing and deploying deep learning models on Intel hardware, including CPUs, GPUs, NPUs, VPUs, and FPGAs. FastSD uses OpenVINO to improve inference speeds on CPU and AIPCs.

Next, we can start comfyUI by double-clicking run_cpu.bat file.

Open the FastSD ComfyUI extension page and drag and drop the first image to your ComfyUi running at http://127.0.0.1:8188/

Write a prompt, select the model rupeshs/sd-turbo-openvino , and run. (if you get a NAN error change token_merging value to 0)

The post How to use ComfyUI with FastSDCPU and OpenVINO appeared first on NoloWiz.

What is Short Circuiting?

Short-circuiting is a technique many programming languages use when evaluating boolean logic expressions to save computing power by skipping unnecessary parts of the expression.

Python provides 2 short-circuit operators :

• or
• and

The or Operator

Python evaluates the left-hand side of the or expression first. If it is True, Python doesn’t evaluate the right-hand side because the entire expression will be True regardless of the right-hand side.

 50 > 20 or 0 > 1

Here, the first expression starting from left 50>20 is true. So Python will skip evaluating expressions on the right-hand side (0>1).

Another example :

a = True
b = False
result = a or b  # Python only evaluates `a`, and the result is `True`

This avoids unnecessary computation on the right-hand side of the “or” operator.

The and Operator

For “and” operator, If the left-hand side expression is False, Python doesn’t evaluate the right-hand side because the entire expression will be False regardless of the right-hand side.

Let’s check the below example :

1 > 20 and 30 > 4

The left-hand side expression( 1>20) is False so Python will skip evaluating right-hand side expressions.

How can we confirm that short-circuiting is skipping the computation?

The expression 1/0 raises an error(Division by zero) as shown below:

Python skips the 1/0 expression evaluation since 1>20 is False.

Similarly, we can confirm or operator short-circuiting too.

Here also division by zero is not raised because 1/0 is not evaluated.

Advantages of Short Circuiting in Python

1. Avoiding errors

Short-circuiting can help avoid errors by skipping unnecessary evaluations.

user = {'name': 'John', 'age': 30}
if user and user['address']:
    print(user['address']['city'])

If user is None, accessing user[‘address’] would raise an error. But with short circuiting, the second part is only evaluated if user is true(Not None), preventing the error.

2. Improving Performance

Short-circuiting can improve performance by avoiding unnecessary computation..

if is_even(num) and is_prime(num):
    print("Number is both even and prime")

If num is odd, checking if it’s prime is unnecessary, so short circuiting skips it.

3. Concise Conditional Logic

Short circuiting allows writing concise conditional logic. For example:

is_loaded = True
content = is_loaded and get_content()

If is_loaded is False, get_content() is never called, but content is still assigned False.

4. Assigning Default Values

Short circuiting is commonly used to assign default values:

user_name = user.get('name') or 'Anonymous'

If user.get(‘name’) returns None, user_name is assigned the default value ‘Anonymous’.

Conclusion

In summary, short circuiting in Python provides a way to write efficient, concise, and error-resistant code by skipping unnecessary computations based on the result of earlier parts of a boolean expression.

The post Short-Circuiting in Python appeared first on NoloWiz.

What is Anomalib?

Anomalib is a deep-learning Python library designed to collect and benchmark state-of-the-art anomaly detection algorithms for both public and private datasets.

Some key features are :

• Python API and CLI available for model training, inference, and benchmarking
• Simple and modular API
• Models can be exported to OpenVINO for faster inference on Intel hardware

Let’s start by creating a Python virtual environment:

python -m venv env

Copy the path to the activate.bat file in the env folder and run :

F:\demo\anomalydetection\env\Scripts\activate.bat

The above command will activate the virtual environment

Now install the required Python dependencies by running the following commands:

pip install torch==2.2.2 --index-url https://download.pytorch.org/whl/cpu 
pip install anomalib[full]==1.1.0

Now the development environment is ready.

Download MVTec anomaly detection dataset

We will use MVTec anomaly detection dataset. It contains different categories of images train and test data.

Download the full MVTec anomaly detection dataset.

You can read more about the MVTech anomaly detection data research paper here.

For this tutorial, we will use bottle data. The bottle folder has the following directories :

• train – This folder contains all good bottle images (209 images).
• test – This folder contains all good and defective bottle images (83 images).
• ground_truth – Contains segmented images

Let’s check good and defective bottle images.

Workflow of Anomalib

The typical workflow of anomalib has the following steps :

• Training – Training the model to recognise the normal pattern.
• Validation/Test – Validating the model and metrics and testing model performance on the test dataset.
• Optimization (optional) – Optimizing model to an efficient form for example FP16 or int8.
• Deployment – The trained model can be deployed using tools Torch, Lightning, ONNX, OpenVINO, and Gradio.

Training

Data modules are responsible for providing the data for training, validation, and testing. We need to prepare a datamodule let’s initialize it.

datamodule = MVTec(
        root=r"F:\demo\mvtec_anomaly_detection",
        category="bottle",
        task=TaskType.CLASSIFICATION,
        val_split_mode=ValSplitMode.SYNTHETIC,  # synthetically generate validation data
        image_size=(256, 256),
        val_split_ratio=0.2,
        train_batch_size=32,
        eval_batch_size=32,
        num_workers=2,
    )

MVTec constructor arguments are :

• root – Root path of extracted MVTec anomaly dataset
• category – bottle
• task – Classification
• val_split_mode – No validation data is available we need to use synthetic data
• image_size – Images resized to 256 x 256
• train_batch_size – training batch size
• eval_batch_size – evaluation batch size
• num_workers – Number of workers

Next, we need to initialize a model and engine. Here we are using the Padim model for the demo.

 #Model and engine
model = Padim()
engine = Engine()

Train the model by calling the engine.fit method.

# Train the model
engine.fit(
        datamodule=datamodule,
        model=model,
    )

After the training, a results folder will be created and the trained lighting model will be saved F:\demo\anomalydetection\results\Padim\MVTec\bottle\v0\weights\lightning\model.ckpt. We will use this model path in the next steps.

The full source code of the training is available here (train.py) .

Test the trained model

To test the model we need a datamodule, model, and engine.

 test_results = engine.test(
        model=model,
        datamodule=datamodule,
        ckpt_path=r"F:\demo\anomalydetection\results\Padim\MVTec\bottle\v0\weights\lightning\model.ckpt",
    )
    print(test_results)

Test output as shown below with image_AUROC and image_F1Score. The full source code of testing is available here (test.py).

Metrics

The following metrics are used image_F1Score and image_AUROC.

F1 Score

The F1 score is a measure of the performance of a classification model, combining precision and recall into a single value. It is the harmonic mean of precision and recall. It ranges from 0 to 1, with higher values indicating better performance.

Precision tells us how many of the predicted positive cases are actually positive. It is a measure of the accuracy of the positive predictions. For example, high precision is essential in spam detection.

Recall indicates how many of the actual positive cases were captured by the model. It measures the model’s ability to identify all relevant cases. High recall is crucial in scenarios where missing a positive case is costly. medical diagnosis, failing to identify a disease can have serious consequences.

AUROC

The AUROC (Area Under the Receiver Operating Characteristic Curve) metric evaluates a binary classifier’s performance. It measures the area under the ROC curve, which plots the true positive rate against the false positive rate at various threshold settings. A higher AUROC indicates better model discrimination between classes.

For more details about Anomalib metrics here.

Export model to OpenVINO

We can export the model to OpenVINO for faster inference on Intel hardware.

openvino_model_path = engine.export(
        model=model,
        export_type=ExportType.OPENVINO,
        export_root=r"F:\demo\anomalydetection\models",  # OpenVINO model will be saved here
        ckpt_path=r"F:\demo\anomalydetection\results\Padim\MVTec\bottle\v0\weights\lightning\model.ckpt",
    )
print(f"OpenVINO Model saved to {str(openvino_model_path)}")

The OpenVINO model will be exported to “F:\demo\anomalydetection\models” directory.

Full source code of OpenVINO export here (ov_export.py).

OpenVINO model inference

Now we do inference on saved model as shown below:

from typing import Any

from anomalib import TaskType
from anomalib.data.utils import read_image
from anomalib.deploy import OpenVINOInferencer
from anomalib.utils.visualization.image import ImageVisualizer, VisualizationMode
from PIL import Image


def get_predictions(
    image_path: str,
    metadata_path: str,
    model_path: str,
) -> Any:
    image = read_image(path=image_path)
    inferencer = OpenVINOInferencer(
        path=model_path,  # Path to the OpenVINO IR model.
        metadata=metadata_path,  # Path to the metadata file.
        device="CPU",
    )
    predictions = inferencer.predict(image=image)
    return predictions

The get_predictions functions will predict the result. Let’s test a broken bottle image.

if __name__ == "__main__":
    predictions = get_predictions(
        r"F:\demo\mvtec_anomaly_detection\bottle\test\broken_large\000.png",
        r"F:\demo\anomalydetection\models\weights\openvino\metadata.json",
        r"F:\demo\anomalydetection\models\weights\openvino\model.bin",
    )

From the above, we can see that the image is anomalous(82%) in nature. Finally, test a normal image.

Full source code of the OpenVINO inference and visualization here (ov_infer.py).

Conclusion

In conclusion, using anomalib we can easily create anomaly detection model. OpenVINO inference helps to speed up inference on intel hardware. The full source code is available as a GitHub repo please check it out.

The post Anomaly Detection Using Anomalib and OpenVINO – Step by Step by Guide appeared first on NoloWiz.

For this demo, I’m using the OpenVINO repository. This method can be used when you already created PR in GitHub and it is out of sync with the master. Clone a copy of your repository and run the following command :

git remote add upstream https://github.com/openvinotoolkit/openvino.git

remote: A subcommand used to manage set of repositories tracked by your local repository.

add: Adds a new remote repository.

upstream: The name given to the new remote repository. This is a common convention, often used to refer to the original repository(OpenVINO) from which a fork was created.

Typically, the above command is used in the context of working with forks. For example, if you have forked the OpenVINO repository to your own GitHub account and cloned it to your local machine, you would use this command to keep track of the original OpenVINO repository (referred to as upstream) so you can pull in the latest changes from the original project.

Next, run the following command :

git fetch upstream

The command git fetch upstream is used in Git to fetch updates from a remote repository named upstream.

When you execute git fetch upstream, Git will:

1. Connect to the remote repository named upstream.
2. Download any changes (commits, branches, tags, etc.) that have been made in the upstream repository since you last fetched.
3. Update your local repository’s remote-tracking branches for upstream, typically upstream/master, upstream/develop, etc.

Now “checkout” your feature branch onnx-fe-bitwise-not :

git checkout onnx-fe-bitwise-not

Now rebase with upstream/master :

git rebase upstream/master

When you run git rebase upstream/master, Git will:

1. Fetch the latest changes from the upstream remote repository’s master branch (if you haven’t already).
2. Move the base of your current branch to the tip of the upstream/master branch.
3. Replay (reapply) your local commits on top of the upstream/master branch.

Finally, we can push the changes and update the MR.

git push origin onnx-fe-bitwise-not --force

The post How to rebase the GitHub pull request appeared first on NoloWiz.

Let’s first dive into the concept of AI PC.

What is an AI PC?

An AI PC (Artificial Intelligence Personal Computer), in simple terms, is a new type of personal computer designed with specific components to efficiently run powerful artificial intelligence (AI) software.

AI PC is a relatively new term in the field of computers. Let’s see its definition by Intel.

An AI PC has a CPU, a GPU and an NPU, each with specific AI acceleration capabilities. An NPU, or neural processing unit, is a specialized accelerator that handles artificial intelligence (AI) and machine learning (ML) tasks right on your PC instead of sending data to be processed in the cloud.

– Intel

Microsoft’s requirements for an AI PC

Microsoft’s definition of an AI PC :

AI PC should have a CPU, GPU, NPU, and it should support Copilot, also have a Copilot key.

– Microsoft

The Copilot key is a new addition to Windows PC keyboards introduced by Microsoft as part of their AI push for 2024. This key provides quick access to Microsoft’s AI-powered Windows Copilot experience directly from the keyboard. 

CPU, NPU, GPU

Let’s understand the core processing components of an AI PC:

CPU ( Central Processing Unit)

We all know that a Central Processing Unit(CPU) is an Integrated circuit that handles general-purpose computing. The CPU’s performance is affected by factors such as clock speed, core count, cache size, and architecture design. Higher clock speeds, more cores, and larger cache sizes generally result in better performance. Artificial intelligence needs more computing power. Even though the CPU can handle AI workloads it will be slower when compared with a GPU or NPU.

NPU (Neural Processing Unit)

Neural Processing Unit(NPU) is a specialized hardware accelerator that can handle Artificial intelligence (AI) and Machine learning (ML) workloads.NPUs are crucial for AI tasks, particularly in the realm of on-device generative AI, as they are optimized to handle neural network computations efficiently. NPU is especially good at low-power AI calculations. In 2017, Huawei was the first company to integrate NPU into smartphone CPUs.

Advantages of NPU over GPU

Here are some important advantages of NPU over GPU :

• NPU is specifically optimized for neural networks; it can perform better than CPU or GPU.
• NPU is specially designed for IoT(Internet of Things) AI or edge AI.
• NPUs are energy-efficient(long battery life in devices like smartphones and laptops) so they can be used for continuous AI workloads (For example real-time face recognition).
• NPUs integrate storage and computation through synaptic weights (synaptic weights refer to the connections between neurons in a neural network), improving operational efficiency compared to CPUs and GPUs.This integration results in higher computational performance and efficiency, making them more suitable for small devices and mobile devices.
• NPUs can complete the processing of a set of neurons with just one or a few instructions, while CPUs and GPUs require thousands of instructions to complete neuron processing.
• GPUs are primarily designed for graphics rendering whereas NPUs are best suited for AI processing.

GPU (Graphics Processing Unit)

Graphics Processing Unit (GPU), is a specialized chip designed to accelerate computer graphics and image processing tasks. It is a programmable processor that is essential for rendering images on a computer screen, particularly for 3D animations and video. GPUs are also used for AI workloads, they are powerful and faster. Deep neural networks can be trained using GPU. GPU can handle generative AI workloads.

Intel Core Ultra Processors and AI PC

Intel Core Ultra processors represent the latest generation of Intel’s mobile processors, introduced in December 2023 as part of the 14th-generation Meteor Lake series. One of the key advantages of Intel Core Ultra processors is their support for artificial intelligence (AI) tasks. These processors are equipped with a neural processing unit (NPU) that provides low-power support for AI tasks, such as real-time language translation, audio transcription, image generation, and automation inferencing

If you are curious about the specifications of the Intel Core Ulta Proccesor check here.

Intel Core Ultra processor is best suitable for AI PCs as it focuses on :

• Intel AI Boost – Built-in AI acceleration with NPU and GPU
• Immersive Graphics – Gaming and video content generation using built-in Intel Arc GPUs
• Longer battery life – Efficient NPU and Arc GPU saves battery power

Intel Core Ultra processors are:

• Intel® Core Ultra Processor 9
• Intel® Core Ultra Processor 7
• Intel® Core Ultra Processor 5

Here is the comparison of these processors by Intel.

Use cases of AI PC

AI PCs can be used for normal workloads like playing music, word processor but AI PC excels in the following use cases :

• Locally running AI assistant
• Local stable diffusion (Text to image generation)
• AI-assisted image editing
• AI-assisted content creation
• Audio effects using AI ( For example, Audacity already introduced OpenVINO AI effects)

OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit for optimizing and deploying deep learning models on Intel hardware, including CPUs, GPUs, NPUs, VPUs, and FPGAs. Intel initially released the OpenVINO toolkit on May 16, 2018. It provides a comprehensive set of tools and libraries to streamline the deployment process, ensuring efficient inference performance across various Intel platforms while maintaining high accuracy.

Using OpenVINO we can efficiently handle AI workloads on an AI PC. For example, consider the following use cases :

• Create copyright-free image generation locally on AI PC using OpenVINO (FastSD CPU)
• Locally running personal assistant using OpenVINO’s fast inference
• Chat locally with your documents using OpenVINO RAG (Retrieval Augmented Generation)
• AI-assisted Image, audio, and video editing

How does OpenVINO works?

OpenVINO converts models from various frameworks into an intermediate representation(IR) and then applies model optimization techniques like quantization, pruning, and graph transformations. The optimized model is loaded onto the OpenVINO Inference Engine, which automatically selects the best execution path based on the available hardware. It supports heterogeneous execution across multiple devices, enabling seamless switching between CPUs, NPUs, GPUs, VPUs, and FPGAs for optimal performance.

OpenVINO supports the following frameworks:

• Pytorch
• Tensorflow
• ONNX
• mxnet
• Keras
• PaddlePaddle

Here are some useful links :

• OpenVINO GitHub repository
• OpenVINO Jupyter notebooks
• OpenVINO quick start guide
• Awesome OpenVINO – A curated list of OpenVINO-based AI projects.
• OpenVINO good first issues – If you are interested in open-source project contributions please contribute to the OpenVINO repository.

Where to buy an AI PC?

We can buy Intel AI PCs from the following online shopping sites, please check the below links and view the available AI PCs:

• Amazon
• Flipkart
• Croma

Conclusion

Now we have a clear idea of AI PC. The combination of powerful AI PCs and Intel’s OpenVINO toolkit unlocks a world of possibilities for computer vision and generative AI applications.

The post AI PC and OpenVINO – Quick and Simple Guide appeared first on NoloWiz.