RunPod is a cloud platform that lets you rent powerful GPUs on demand for AI workloads such as running LLMs, image generation, video generation, and model training. It provides access to GPUs like NVIDIA RTX 5090, A100, and H100 without needing to own the hardware.

First create and account in rupod and recharge. Next we need to create a pod for that we need to select a GPU for this demo I’m going to use RTX 5090 GPU(32 GB VRAM). A Pod in RunPod is a rented cloud computer with a GPU. You choose a GPU, CPU, RAM, storage, and an image/template, then RunPod creates the Pod where you can run AI models, Docker containers, or other applications.

Next we will keep pod template as it is. Change the storage configuration to Volume disk. Refer the configuration screenshot below :

Next click the “Deploy-On-Demand” button and you will see a deploying screen as shown below.

After the pod initalized, we need edit port number click the edit pod by clicking the pods hamberger menu on the right side. Change HTTP port to 8080 and save.

Enable web terminal and open web terminal in browser.

Install llama.cpp by running the below command :

curl -LsSf https://llama.app/install.sh | sh

Now we have installed llama.cpp. Lets check the GPU by running nvidia-smi command.

Next we will Qwen 3.8 27B model run the below command to download it.

curl -L -C - -o Qwen3.8-27B-Q4_K_M.gguf https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-Q4_K_M.gguf

Run the below command to start the llama

~/.llama-app/llama serve -m Qwen3.8-27B-Q4_K_M.gguf -ngl 99 --host 0.0.0.0 --port 8080 -c 8192

Next we need to copy the pod ID by opening pod page :

Now we can access the llama.cpp webui on your browser :

https://{YOUR_RUNPOD_ID}-8080.proxy.runpod.net/

In my case my runpod ID is psyvpxd4isy2vu and I can access llama.cpp in https://psyvpxd4isy2vu-8080.proxy.runpod.net/

We can see the llama.cpp webui in browser :

After the use we can stop and terminate pod.

Conclusion

Running llama.cpp on RunPod provides a simple and cost-effective way to run large language models on powerful NVIDIA GPUs without requiring local GPU hardware. In this guide, we created a RunPod Pod with an RTX 5090, installed llama.cpp, downloaded a GGUF model, configured GPU offloading, and exposed the llama.cpp web interface through RunPod’s proxy.

The post How Run llama.cpp on Runpod – Step by step Guide appeared first on NoloWiz.

In this article we’ll clear up the confusion, explain what each service actually does, and give you a practical framework for deciding what to use for your own workloads.

The Short Answer

EC2 (Elastic Compute Cloud) gives you virtual servers in the cloud. You get an operating system, CPU, memory, and storage, and you’re responsible for everything you run on it.

ECS (Elastic Container Service) is a container orchestration service. It decides where and how your Docker containers run, restarts them when they crash, and scales them up and down.

The key thing most comparison articles bury: ECS is not an alternative to EC2 ECS runs on top of compute, and one of its two options for that compute is EC2 itself. The real decision isn’t usually “ECS or EC2.” It’s “should I manage my containers directly on EC2 by hand, or let ECS orchestrate them for me?”

What Is EC2?

EC2 is AWS’s Infrastructure-as-a-Service (IaaS) offering. When you launch an EC2 instance, you’re renting a virtual machine. You choose:

• An instance type (how much CPU and RAM – e.g. t3.micro, m5.large)
• An AMI (the base image / operating system, like Amazon Linux or Ubuntu)
• Storage (EBS volumes)
• Networking (VPC, security groups, public/private IPs)

Once it boots, that server is yours to manage. You SSH in, install software, deploy your application, patch the OS, and handle scaling. That flexibility is EC2’s biggest strength and its biggest burden you can run anything, but you also maintain everything.

EC2 is a good fit when you need:

• Full control over the operating system and environment
• Traditional (non-containerized) applications
• Specialized software that expects a full server
• Long-running workloads where you want to fine-tune the machine

What Is ECS?

ECS is a container orchestration platform. If your application is packaged as Docker containers, ECS handles the operational hard parts for you:

• Scheduling – placing containers onto available compute
• Health checks and self-healing – restarting failed containers automatically
• Scaling – running more or fewer copies based on load
• Service discovery and load balancing – integrating with an Application Load Balancer
• Rolling deployments – updating your app without downtime

In ECS you define a task definition (a blueprint describing your container image, CPU/memory, ports, and environment variables), then run it as a task or a long-running service.

Here’s a simplified task definition to make it concrete:

{
  "family": "my-web-app",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "web",
      "image": "123456789.dkr.ecr.us-east-1.amazonaws.com/my-web-app:latest",
      "cpu": 256,
      "memory": 512,
      "portMappings": [
        { "containerPort": 80, "protocol": "tcp" }
      ]
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512"
}

Notice there’s nothing in there about which server this runs on. That’s the point of orchestration you describe what you want running, and ECS figures out where.

The Part That Confuses Everyone: ECS Launch Types

ECS still needs actual compute to run your containers on. It offers two ways to provide it, and this is where EC2 re-enters the picture.

1.ECS on EC2 (the EC2 launch type)

You run a cluster of EC2 instances that you own and manage, and ECS packs your containers onto them. You’re still responsible for the instances patching, scaling the cluster, right-sizing but ECS handles the container-level orchestration on top.

Choose this when you want lower cost at scale, need specific instance types (GPUs, high memory), or want fine-grained control over the underlying hosts.

2.ECS on Fargate (the Fargate launch type)

Fargate is serverless compute for containers. There are no EC2 instances for you to see or manage you just specify how much CPU and memory each task needs, and AWS provisions the compute invisibly. You pay only for the resources your tasks use while they run.

Choose this when you want zero server management, variable or bursty workloads, and the simplest possible operations, and you’re willing to pay a bit more per unit of compute for that convenience.

So Which One Should You Use?

Instead of picking “ECS vs EC2” as rivals, walk through these questions:

Is your application containerized (Docker)? If no, and you don’t plan to containerize it, plain EC2 is your natural home. Run it like a traditional server.

If yes, you almost certainly want an orchestrator, which means ECS (or EKS if you specifically need Kubernetes).

If you’re using ECS, how much do you want to manage the servers?

• Want AWS to handle the compute entirely and value simplicity – ECS on Fargate
• Want lower cost at scale or need special instance types, and are comfortable managing a cluster – ECS on EC2

A quick rule of thumb:

• Single traditional app, or you need OS-level control – EC2
• Containers, and you want the least operational work – ECS + Fargate
• Containers at large scale where cost matters – ECS + EC2

A Note on Pricing

Pricing is where the trade-offs become tangible, and AWS pricing changes over time, so always confirm current rates in the AWS pricing calculator. In general terms:

• EC2 – you pay for the instance whether it’s busy or idle. Reserved Instances and Savings Plans cut costs significantly for steady workloads.
• ECS itself – there’s no extra charge for the ECS control plane. You only pay for the underlying compute.
• ECS on EC2 – you pay standard EC2 prices, so you can hit lower per-unit costs at scale but you pay for idle capacity.
• ECS on Fargate – you pay per vCPU and GB of memory that your tasks consume, billed by the second. Simpler and often cheaper for spiky or low-utilization workloads, but usually pricier per unit for constant, high-utilization ones.

Frequently Asked Questions

Is ECS a replacement for EC2? No. ECS orchestrates containers and can even run on EC2. They solve different problems and frequently work together.

Can I run containers on EC2 without ECS? Yes, you can install Docker on an EC2 instance and run containers manually. But you’d be hand-building the scheduling, health checks, and scaling that ECS gives you for free.

What about EKS? EKS is AWS’s managed Kubernetes service. It’s an alternative to ECS for orchestration (not to EC2), and it also runs on EC2 or Fargate. Choose EKS if you specifically want Kubernetes; choose ECS for a simpler, more AWS-native experience.

What’s the difference between Fargate and EC2 for ECS? Both provide the compute ECS runs on. With EC2 you manage the servers; with Fargate you don’t manage any servers at all.

Conclusion

“ECS vs EC2” is really a comparison between a building block (EC2, raw virtual servers) and a management layer (ECS, container orchestration) and ECS often sits directly on top of EC2. If you’re running traditional applications and want full control, EC2 is your foundation. If you’re running Docker containers and want automated scheduling, scaling, and healing, ECS is the tool, and your next choice is simply whether to back it with EC2 (more control, cheaper at scale) or Fargate (zero server management). Get that mental model right and the “versus” mostly disappears you’ll know exactly which piece you need, and when to combine them.

The post ECS vs EC2: What’s the Difference and Which One Should You Use? appeared first on NoloWiz.

1. Stash your changes.
2. Checkout main.
3. Fix the bug.
4. Commit.
5. Checkout back to your feature branch.
6. Stash pop. Sound tedious? You’re not alone.

But there’s a better way. Today, I’m going to introduce you to one of Git’s most underrated features: git worktree!

What Is Git Worktree?

In simple terms, git worktree lets you work on multiple branches of the same repository simultaneously, each in its own directory.

Normally, Git only lets you be on one branch at a time. To switch branches, you either commit, stash, or discard your current work. With worktrees, you get separate working directories that all share the
same .git folder meaning you can be on feature-x in one window and main in another, without any stashing or switching.

The Problem It Solves

Imagine you are working on a feature branch (feature-x) and a critical bug appears on the main branch.

• Without worktree: You stash your changes, checkout main, fix the bug, commit, and then stash pop back to your feature.
• With worktree: You simply open a second terminal or directory for main, fix the bug, and leave your feature branch untouched in the first directory.

That’s it. No stashing. No context switching. No lost mental momentum.

How to Use It

1. Create a new worktree

Run the following command :

git worktree add -b fix-bug ../fix-bug main

This creates a new folder (../fix-bug), creates a new branch named fix-bug based on main, and checks it out for you automatically.

2. List All Worktrees

Run the below command to list all worktrees :

git worktree list

Output :

/path/to/repo       (master)
/path/to/fix-bug    (fix-bug)

3. Work on the new branch

Navigate into your new directory (cd ../fix-bug). Here, you can make commits, run tests, or mess with dependencies completely independently from your original repository. Your main repo’s working directory remains completely untouched.

4. Remove a Worktree

When you are finished with your changes and have pushed or merged your work, you can clean up:

git worktree remove ../fix-bug

Note: This safely deletes the ../fix-bug directory. It does not delete the branch itself from Git history. If you want to completely delete the branch too, you can now safely run git branch -d fix-bug from your main repository directory.

Common Use Cases

• Fixing urgent bugs – Work on main for a hotfix while keeping your feature branch clean in the main terminal.
• Testing – Test a specific branch without having to switch your current working state.
• Code reviews – Pull down a colleague’s PR in a separate folder without merging it into your local repo.
• Documentation – Work on both code and docs in parallel without switching branches.
• AI Agents & Copilots – In 2026, we see a massive spike in worktree usage driven by coding AI agents (like Claude Code, GitHub Copilot Workspace, or OpenAI Codex). These tools often spin up isolated background git worktrees to write features, run automated tests, and fix bugs without disrupting the developer’s active screen.

Git Worktree Anti-Patterns to avoid

1. Checking Out the Same Branch Twice

git worktree add ../fix main   # fails if already on main

Why: Git hard-blocks this. Each worktree must be on a unique branch.

2. Rebasing/Amending a Branch Another Worktree Is Using

# Worktree A is on feature-x
# In main worktree — DON'T do this:
git rebase -i HEAD~3 feature-x   # corrupts worktree A's HEAD

Why: History rewrite orphans the other worktree’s commits.

3. Putting Worktrees Inside the Repo Folder

git worktree add ./fix-bug -b fix-bug main   # inside repo

Why: Causes .gitignore headaches, accidental commits of worktree folders, and confusing git status output. Always place worktrees outside or as siblings:

git worktree add ../fix-bug -b fix-bug main  # Correct - sibling folder

4. Forgetting to Remove Stale Worktrees

# Manually deleted the folder but never ran:
git worktree remove fix-bug

Why: Git still tracks it internally. Builds up ghost entries. Always clean up:

git worktree prune    # removes stale entries
git worktree list     # verify

5. Running Git Commands That Affect Shared State From Wrong Worktree

# From fix-bug worktree — dangerous:
git branch -D feature-x     # deletes branch another worktree is on
git tag -f v1.0              # moves a tag others depend on
git gc --aggressive          # can corrupt shared object store mid-work

Why: All worktrees share the same .git – destructive commands affect everyone.

6. Using Worktrees With Submodules Carelessly

git worktree add ../feature -b feature main
# Submodules are NOT automatically initialized in new worktree

Why: Submodule state is not automatically carried over. You must manually run:

cd ../feature
git submodule update --init

Final Thoughts

The git worktree is one of those features that, once you start using it, you wonder how you ever lived without it. It’s lightweight, it’s built into Git (no plugins needed), and it saves you from the stashing dance every time.

The post Mastering Git Worktree: How to Work on Multiple Branches at Once appeared first on NoloWiz.

What is SearXNG?

SearXNG is a free, open-source, self-hosted metasearch engine that aggregates results from multiple search engines without tracking users. It solves the problems of privacy invasion, vendor lock-in, API costs, and search bias by giving developers full control over search data, sources, and usage.

Here are the key benefits of SearXNG:

• Privacy-first: No user tracking, logging, or profiling.
• Self-hosted control: You own the data, configuration, and infrastructure.
• Multiple search engines: Aggregates results from many sources(Google,Bing,DuckDuckGo etc) in one query.
• Free & open source: No API costs, no vendor lock-in.
• API-friendly: JSON output works well for apps and LLM pipelines.
• Customizable: Enable/disable engines, formats, languages, and filters.
• Lightweight & fast: Runs efficiently with minimal resources.
• LLM-ready: Ideal for RAG, research agents, and private AI workflows.

Step 1: Install Docker and Docker Compose

First we need to install SerXNG, we will use docker based installation and use.

• For Windows users, Download docker desktop from docker’s official site
• Linux users should install docker and docker compose by following the guide

Step 2: Install SearXNG

Make a directory named “searxng”, and create file named docker-compose.yml inside this newly directory. Copy paste the below code.

version: "3.7"

services:
  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    ports:
      - "8080:8080"
    volumes:
      - ./searxng:/etc/searxng
    environment:
      - SEARXNG_BASE_URL=http://localhost:8080/
    restart: unless-stopped

Next run the below command :

docker-compose up

Wait for the installation to finish, and check the browser by operning the URL http://localhost:8080/.

Step 3: Enable API in SearXNG

To enable and use the API, first install and configure SearXNG, then add JSON support in settings. We need edit the searxng\searxng\settings.yaml file add the json format in the formats settings as shown below :

search:
  formats:
    - html
    - json  # Add this for API
server:
  secret_key: "your-random-secret-key" 

Save the settings file and restart the SearXNG container.

docker-compose restart

Now we can make API requests to SearXNG enpoints, for more details about API please refer here .

Conclusion

It is easier to setup SearXNGand expose its API endpoints, thanks to docker. Run Large Language Models Locally with Simple APIs

The post How to use SearXNG as a Private Search API – Step-by-Step Guide appeared first on NoloWiz.

First update the system by running the below commands :

sudo apt update
sudo apt upgrade

We need to disable Nouveau driver,Nouveau is an accelerated open-source graphics driver for NVIDIA GPUs, developed independently by the free software community to provide libre support without relying on NVIDIA’s proprietary drivers. Disable Nouveau driver by creating a blacklist file:

sudo bash -c "echo -e 'blacklist nouveau\noptions nouveau modeset=0' > /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
sudo update-initramfs -u

Next, reboot the system eithe reboot manually or run the below command :

sudo reboot

Install the prerequisites by running the command :

sudo apt install build-essential libglvnd-dev pkg-config

Add the graphics drivers PPA to get the latest official NVIDIA drivers:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update

Check available NVIDIA drivers and install the recommended one:

ubuntu-drivers devices

Install latest/recommended driver version, I’m using NVIDIA RTX 3060 graphics card so using the driver nvidia-driver-570 :

sudo apt install nvidia-driver-570

Alternatively, to auto install the CUDA driver run the below command.

sudo ubuntu-drivers autoinstall

Verify the installation by running the below command :

nvidia-smi

Conclusion

By following these simple steps it is easy to install CUDA driver on Ubuntu. Read about TP-LINK AC1300 Archer T3U Ubuntu Driver Installation .

The post How to Install CUDA Driver on Ubuntu 24.04 LTS or Higher – A step-by-step guide appeared first on NoloWiz.

Regression Bug

A regression bug is an error that appears in software after a change like a new feature, update, or fix causing previously working functionality to break.It indicates the system has “regressed” in quality. This can happen in typical software development.

What is Git Bisect?

Let’s understand the problem :

We know a feature was working before, now it’s broken. How do we find which commit broke it – without checking every commit?

The Solution is git bisect!

Git Bisect helps find the exact commit that introduced a bug using binary search.

The Core Idea Behind Git Bisect

Binary Search is a fast way to find something inside a sorted list. Instead of checking items one by one, it splits the list in half each time reducing search time from O(n) to O(log n).

Let’s see an example searching for 7 in a sorted list as shown below.

List: [1, 2, 3, 4, 5, 6, 7, 8]

Search for “7”
              ↓
Step 1 → middle = 4 (not found)
Search right half → [5,6,7,8]

Step 2 → middle = 6 (not found)
Search right half → [7,8]

Step 3 → middle = 7 (FOUND)

Instead of checking 8 items, it found the answer in just 3 checks.

The same technique is used in git bisect to search for the commit that caused the bug.

How to use git bisect

Suppose we have simple calculator app with following commits. (We introduced a bug in commit 57cd4b8 for this demo)

The latest code is shown below commit ID : ac6beeb

def add(a :int, b :int) -> int:
    return a - b

def subtract(a :int, b :int) -> int:
    return a - b    

def multiply(a :int, b :int) -> int:
    return a * b

def divide(a :int, b :int) -> float:
    if b == 0:
        raise ValueError("Cannot divide by zero.")
    return a / b

def modulus(a :int, b :int) -> int:
    return a % b

if __name__ == "__main__":
    result = add(3, 5)
    print(f"The sum of 3 and 5 is: {result}")

We can see that add function is wrong when we run the application we will get the below output

Now we have a regression bug, we need to find out the which commit introduced this bug. To finding this bug manually we need to test each commit for the bug, but git bisect will help us to find the buggy commit faster.

First run the below command to list all the commits with commit IDs.

git log --oneline

Run the below command to start the git bisect process :

git bisect start

Next we need to give one known good commit where the feature(additon) worked and bad commit. We are going to use use the first commit for this demo (You can give previous good release commit ID)

git bisect good 8714580

Finally, we need to give a bad commit for this demo Im using latest commit ID ac6beeb :

git bisect bad ac6beeb

From the above logs we can see that bisecting started current commit is “update readme”. Let’s check app output

It is working fine – the answser is correct, next we need to mark it as good by running the below command

git bisect good

We are now moved to another commit that is “Update readme with supported operations”, again run the application or run unit tests.

Yes we got bad commit, mark it using the below command.

git bisect bad

We finally found the commit that caused the bug that is “Implement multiplication”, commit ID : 57cd4b8.

Finally , we need to exit git from bisect mode by running the below command

git bisect reset

Conclusion

With Git Bisect, you let Git do the detective work for you – using the power of binary search to find the exact commit that introduced a bug. It’s fast, reliable, and perfect for regression issues. I hope this tutorial helpful for you. Learn the difference between git rebase and git merge.

The post Regression Bug? Learn Git Bisect to Find the Problem Commit appeared first on NoloWiz.

Project

In this demo, we are going to create a project named “Cool Project” . It has a FastAPI-based API server and client app. For app-to-app authentication/authorization, we will use the OAuth2 client credentials flow.

OAuth 2.0 Client Credentials Flow Specification – RFC 6749

The components of our project are :

• Resource server – FastAPI server
• Client – Client app ( For the simplicity of this demo, we will use Postman to call the APIs).
• Authorization Server – Keycloak

We will use the client ID and client secret of the client app to get an access token from the Keycloak authorization server. We will use this access token to call the FastAPI server APIs.

Keycloak

Keycloak is an open-source identity and access management solution that provides single sign-on (SSO), authentication, and authorization for web and mobile applications. It supports modern security protocols like OAuth2 and OpenID Connect, integrates with LDAP and social logins, and offers centralized user management and customization. Keycloak supports the OAuth2 Client Credentials flow, which can be used for app-to-app authentication and authorization.

Start Keycloak

First, we need to download Keycloak.

Download the Keycloak zip file from here and extract the zip file. We are using Keycloak version 26.3.

After extracting the zip, enter the keycloak directory and run the following command in the command prompt:

bin\kc.bat start-dev

Please note that we are running Keycloak in development mode. Open the URL http://localhost:8080/ in a browser. For this demo, we’re going to create an admin user.

Create an admin user with a username admin and password as adminpass Then log in to Keycloak using the admin user.

Create a realm for the project

Next, we need to create a realm for our project. A realm in Keycloak is an isolated security and administrative domain where you manage users, applications, roles, and groups. In simple terms, we can think of it as a project with all the details.. Each realm controls authentication and authorization for its contained resources, ensuring that users and configurations are separated from other realms, enabling multi-tenancy and organized management.

Let’s create a cool project realm. Click on “Manage realms” on the left sidebar and create a realm as shown below.

Create client

Next, we need to create a client in the coolproject realm.

Click next, and on the next page, we need to turn on “Client authentication” and “Service account roles” as shown below :

The above settings enable us to use the OAuth2 client credentials flow.

Click next and leave everything as it is, and click save. Now, click on clients again on the left sidebar, as we can see the newly created client “client-app” listed as shown below :

Client ID and Client Secret

In the OAuth2 Client credentials flow, we need the client ID and client secret.

Open the client-app from the left sidebar and we can see that Client ID is “client-app” as shown below :

Next, we need the client secret. Click the credentials tab.

Open the http://localhost:8080/realms/coolproject/.well-known/openid-configuration in a browser. We can see the token endpoint as shown below. We will use this URL in Postman to get an access token for the client app.

Add audience claim

In the Client Credentials flow, there is no user; it’s an application-to-application (machine-to-machine) authentication. The audience (aud) claim in the access token typically refers to the resource server, that is, the application or API that will receive and validate the token.

We need to add the audience claim to the access token payload. Open clients, then client scopes, and click one client-app-dedicated (a client scope that is ending dedicated).

Then turn off “Full scope allowed” (It will remove the default audience “account” ) as shown below.

Next, we will create a new client scope for the audience with value as the resource server name.

Next, open the client and add the newly created “api-audience” scope to the client-app. Open clients and client scopes, add api-audience as shown below:

Ensure that api-audience assigned type is “Default” as shown below.

Now our client app configuration is ready.

Start Resource Server

Next, we need to start the resource server. In this example, we are going to use a simple FastAPI application.

from fastapi import FastAPI, Depends, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from jose import jwt
import httpx

app = FastAPI()
security = HTTPBearer()


KEYCLOAK_REALM = "coolproject"
KEYCLOAK_URL = "http://localhost:8080"
KEYCLOAK_OPENID_CONFIG = (
    f"{KEYCLOAK_URL}/realms/{KEYCLOAK_REALM}/.well-known/openid-configuration"
)

JWKS = None
from jose.utils import base64url_decode
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.backends import default_backend


async def verify_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
    token = credentials.credentials
    jwks = await get_public_key()

    try:
        unverified_header = jwt.get_unverified_header(token)
        kid = unverified_header.get("kid")
        if not kid:
            raise Exception("Missing 'kid' in token header")

        key = next(k for k in jwks["keys"] if k["kid"] == kid)

        n = int.from_bytes(base64url_decode(key["n"].encode()), byteorder="big")
        e = int.from_bytes(base64url_decode(key["e"].encode()), byteorder="big")

        public_key = rsa.RSAPublicNumbers(e, n).public_key(default_backend())

        payload = jwt.decode(
            token,
            key=public_key,
            algorithms=["RS256"],
            audience="apiserver",
            issuer=f"{KEYCLOAK_URL}/realms/{KEYCLOAK_REALM}",
        )
        print(payload)
        return payload

    except Exception as e:
        raise HTTPException(status_code=401, detail=f"Invalid token: {str(e)}")


async def get_public_key():
    global JWKS
    if JWKS is None:
        async with httpx.AsyncClient() as client:
            r = await client.get(KEYCLOAK_OPENID_CONFIG)
            jwks_uri = r.json()["jwks_uri"]
            r2 = await client.get(jwks_uri)
            JWKS = r2.json()
    return JWKS


@app.get("/secure-data")
async def secure_data(user=Depends(verify_token)):
    return {"message": "You have access!", "client": user["sub"]}


if __name__ == "__main__":
    import uvicorn

    uvicorn.run("app:app", host="127.0.0.1", port=8000, reload=True)

Create a requirements.txt and install it

fastapi
httpx
python-jose[cryptography]
uvicorn

In the JWT decode, we need to give the audience as the “apiserver” that we configured

payload = jwt.decode(
            token,
            key=public_key,
            algorithms=["RS256"],
            audience="apiserver",
            issuer=f"{KEYCLOAK_URL}/realms/{KEYCLOAK_REALM}",
        )

Start the FastAPI server. Next, we will get an access token using the /token request as shown below :

Finally, we will use this access token to call the APIs as shown below :

We can see that the access token payload contains the “aud” claim.

Conclusion

In conclusion, it is easier to set up an authorization server using client credential flow in Keycloak. You can read about How to serve LLM using the OpenVINO Model Server on Windows.

The post App-to-App Authentication with Keycloak – Step by Step Guide appeared first on NoloWiz.

We know that OpenVINO is an open-source toolkit by Intel for optimizing and deploying deep learning models. It enhances model performance on Intel hardware, supports frameworks like PyTorch and TensorFlow, and offers tools for efficient inference across multiple platforms, including CPUs, GPUs, NPUs and VPUs.

Ensure the following dependencies are available on your Windows 11 machine :

• Python 3.11 or higher
• git
• curl (optional)

Model Conversion

We need to convert the model LLM model to OpenVINO. Create a folder and a Python virtual environment, for that open command prompt and run the below commands :

mkdir model_converter
cd model_converter
python -m venv env
env\Scripts\activate

Next, install the dependencies by running the below command :

pip install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt

Then download the export_model.py file to “model_converter” folder or use the below command :

curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py  --ssl-no-revoke -o export_model.py

Create a folder named “models” by running the below command :

mkdir models

Next, we need to convert the TinyLlama-1.1B-Chat-v1.0 model to OpenVINO using the below command :

python export_model.py text_generation --source_model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --config_file_path models/config.json --model_repository_path models 

After conversion, the OpenVINO TinyLlama model is available in the “models” directory. We need the full path of “config.json” to serve the model with the OpenVINO model server. The converted OpenVINO model weights are int4 quantized.

Deploy model with OpenVINO Model Server

OpenVINO Model Server is a high-performance system for serving optimized machine learning models over standard network protocols like REST and gRPC. Built with C++ and optimized for Intel hardware, it leverages the OpenVINO toolkit for accelerated inference. OVMS offers gRPC and REST APIs, compatible with TensorFlow Serving and KServe, simplifying integration. It supports diverse frameworks and model formats, enabling scalable and efficient deployment for AI applications like image recognition and NLP. OVMS is open-source, flexible, and designed for production environments.

Download the OpenVINO model server for Windows from the GitHub releases page here . We are using OpenVINO Model Server 2025.0 for this tutorial, extract the zip(ovms_windows.zip) file.

Open the command prompt enter into the “ovms_windows” folder and run the below commands:

setupvars.bat

Next, we can start the OVMS to serve our model by running the below command :

ovms --config_path "<<Your converted models config.js path>>"  --rest_port 8000

Now the OVMS running at http://localhost:8000. Verify model serving is ok by requesting the API endpoint http://localhost:8000/v1/config

As we can see that model is available since the state is “available” in the above API response.

Use LLM

OpenVINO model server provides chat completions API endpoint similar to OpenAI.The chat completion endpoint of OVMS is :

http://localhost:8000/v3/chat/completions

Install OpenAI package:

pip install openai

We can test the LLM using the below code :

from openai import OpenAI

client = OpenAI(
  base_url="http://localhost:8000/v3",
  api_key="unused"
)

stream = client.chat.completions.create(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    messages=[{"role": "user", "content": "What is opencv?"}],
    stream=True,
    temperature=0.1,
)
for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

Output

Conclusion

In short, deploying your LLM model on Windows is simplified with the OpenVINO model server. Just remember to convert your chosen model with the appropriate quantization for optimal performance.

The post How to serve LLM using the OpenVINO Model Server on Windows appeared first on NoloWiz.

Step 1: Download and Install FastSDCPU

In this demo, we are using a Windows 11 machine to test, first download FastSD CPU release. Extract the zip file and install it by following the steps :

• You need to install Python 3 and uv – fast package manager for python.
• Double click install.bat (It will take some time to install, depending on your internet speed.)
• After the installation close this command prompt window
• Double click the start-webserver.bat file to start FastSD CPU in webserver mode

Step 2: Download and Install ComfyUI

Next, we will download ComfyUI from this official repo and extract the zip file.

Step 3: Install ComfyUI extensions

Enter into ComfyUI_windows_portable\ComfyUI\custom_nodes folder and clone the following extensions :

• rupeshs/ComfyUI-FastSDCPU
• pythongosssss/ComfyUI-Custom-Scripts

Run the following command to install extensions:

git clone https://github.com/pythongosssss/ComfyUI-Custom-Scripts.git
git clone https://github.com/BetaDoggo/ComfyUI-FastSDCPU

Step 4: Faster generation of images using OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit for optimizing and deploying deep learning models on Intel hardware, including CPUs, GPUs, NPUs, VPUs, and FPGAs. FastSD uses OpenVINO to improve inference speeds on CPU and AIPCs.

Next, we can start comfyUI by double-clicking run_cpu.bat file.

Open the FastSD ComfyUI extension page and drag and drop the first image to your ComfyUi running at http://127.0.0.1:8188/

Write a prompt, select the model rupeshs/sd-turbo-openvino , and run. (if you get a NAN error change token_merging value to 0)

The post How to use ComfyUI with FastSDCPU and OpenVINO appeared first on NoloWiz.

What is Short Circuiting?

Short-circuiting is a technique many programming languages use when evaluating boolean logic expressions to save computing power by skipping unnecessary parts of the expression.

Python provides 2 short-circuit operators :

• or
• and

The or Operator

Python evaluates the left-hand side of the or expression first. If it is True, Python doesn’t evaluate the right-hand side because the entire expression will be True regardless of the right-hand side.

 50 > 20 or 0 > 1

Here, the first expression starting from left 50>20 is true. So Python will skip evaluating expressions on the right-hand side (0>1).

Another example :

a = True
b = False
result = a or b  # Python only evaluates `a`, and the result is `True`

This avoids unnecessary computation on the right-hand side of the “or” operator.

The and Operator

For “and” operator, If the left-hand side expression is False, Python doesn’t evaluate the right-hand side because the entire expression will be False regardless of the right-hand side.

Let’s check the below example :

1 > 20 and 30 > 4

The left-hand side expression( 1>20) is False so Python will skip evaluating right-hand side expressions.

How can we confirm that short-circuiting is skipping the computation?

The expression 1/0 raises an error(Division by zero) as shown below:

Python skips the 1/0 expression evaluation since 1>20 is False.

Similarly, we can confirm or operator short-circuiting too.

Here also division by zero is not raised because 1/0 is not evaluated.

Advantages of Short Circuiting in Python

1. Avoiding errors

Short-circuiting can help avoid errors by skipping unnecessary evaluations.

user = {'name': 'John', 'age': 30}
if user and user['address']:
    print(user['address']['city'])

If user is None, accessing user[‘address’] would raise an error. But with short circuiting, the second part is only evaluated if user is true(Not None), preventing the error.

2. Improving Performance

Short-circuiting can improve performance by avoiding unnecessary computation..

if is_even(num) and is_prime(num):
    print("Number is both even and prime")

If num is odd, checking if it’s prime is unnecessary, so short circuiting skips it.

3. Concise Conditional Logic

Short circuiting allows writing concise conditional logic. For example:

is_loaded = True
content = is_loaded and get_content()

If is_loaded is False, get_content() is never called, but content is still assigned False.

4. Assigning Default Values

Short circuiting is commonly used to assign default values:

user_name = user.get('name') or 'Anonymous'

If user.get(‘name’) returns None, user_name is assigned the default value ‘Anonymous’.

Conclusion

In summary, short circuiting in Python provides a way to write efficient, concise, and error-resistant code by skipping unnecessary computations based on the result of earlier parts of a boolean expression.

The post Short-Circuiting in Python appeared first on NoloWiz.