License Plate Recognition with YOLO and OCR: A Complete Pipeline

Automatic License Plate Recognition (ALPR, also called ANPR) powers parking systems, toll gates, access control and traffic monitoring. A modern ALPR system is a pipeline of deep learning models: one finds the plate, another reads it. This article walks through building one with YOLO, with special attention to Iranian (Persian) plates, which have their own format and character set.

The pipeline

camera frame → [1] plate detection (YOLO) → crop & rectify → [2] character recognition → [3] validation & tracking → result

1. Detection — locate plates in the image as bounding boxes.
2. Recognition — read the characters inside each plate crop.
3. Post-processing — validate the format, combine readings across video frames, and store the result.

Splitting the problem this way lets each model focus on one task and be improved independently.

Step 1: collect and label data

Model quality depends mostly on data. Collect images that match your deployment conditions:

• the actual camera angle and height (gate camera vs street camera),
• day, night, infrared, rain, glare and motion blur,
• different vehicle types, dirty or partially occluded plates,
• motorcycle plates if relevant.

A few thousand well-labeled images from your real cameras beat tens of thousands of random internet images.

Label with a tool like CVAT, Label Studio or Roboflow, exporting in YOLO format — one .txt per image with class x_center y_center width height normalized to 0–1:

0 0.512 0.734 0.186 0.061

Split into train/validation/test (e.g. 80/10/10), making sure images from the same video or location don't leak across splits — otherwise validation scores look better than reality.

# plates.yaml
path: datasets/plates
train: images/train
val: images/val
test: images/test
names:
  0: plate

Step 2: train the plate detector

With the Ultralytics package, training a custom YOLO detector is a few lines:

pip install ultralytics

from ultralytics import YOLO

model = YOLO("yolo11n.pt")          # small pretrained model; try "s" or "m" for more accuracy
model.train(
    data="plates.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
    patience=20,                    # early stopping
    degrees=5, translate=0.1, scale=0.5, fliplr=0.0,   # no horizontal flip: it mirrors characters
)
metrics = model.val()
print(metrics.box.map50, metrics.box.map)

Key points:

• Start from pretrained weights (transfer learning) — it converges much faster with less data.
• Disable horizontal flip augmentation; mirrored plates don't exist in reality.
• Watch mAP@50 and recall on the validation set: for ALPR, missing a plate (low recall) is usually worse than an occasional false box, which the recognition step can reject.
• Small, distant plates benefit from a larger imgsz (e.g. 960 or 1280), at the cost of speed.

Step 3: crop and normalize the plate

Before recognition, crop each detected plate with a small margin, then:

• resize to a fixed height (e.g. 64 px) keeping the aspect ratio,
• optionally correct perspective (detect the four corners with a keypoint/OBB model and warp to a rectangle),
• convert to grayscale and improve contrast (CLAHE) if your recognizer was trained that way.

import cv2

def crop_plate(frame, box, pad=0.08):
    x1, y1, x2, y2 = box
    w, h = x2 - x1, y2 - y1
    x1, y1 = max(0, int(x1 - w * pad)), max(0, int(y1 - h * pad))
    x2, y2 = min(frame.shape[1], int(x2 + w * pad)), min(frame.shape[0], int(y2 + h * pad))
    return frame[y1:y2, x1:x2]

Step 4: reading the characters

There are two popular approaches.

Option A: characters as detection classes

Train a second YOLO model where each character is a class (digits 0–9 plus the Persian plate letters). Sort detected characters by x-coordinate to form the string. This works very well for fixed-format plates and is robust to spacing and fonts.

The standard Iranian private-car plate has the layout:

[2 digits] [letter] [3 digits] | [2-digit region code]

For example ۱۲ ب ۳۴۵ - ۶۷. Since the region code sits in a separate box on the right, assign characters to the main part or the region part by position.

Option B: sequence recognition (CRNN / transformer OCR)

A CRNN with CTC loss, or a small transformer OCR model, reads the whole crop as a sequence. It handles variable-length plates and needs only the plate text as a label, not per-character boxes, but it usually needs more training data.

Warning: General-purpose OCR engines (like Tesseract out of the box) perform poorly on plates: unusual fonts, the Persian letter set, low resolution and dirt. A model trained on your plates performs dramatically better.

Step 5: validate the format

Use the known plate structure to reject impossible readings and fix common confusions:

import re

PERSIAN_LETTERS = "بجدسصطقلمنوهیالفپتثزشعژ"   # adapt to the letters your model outputs
PATTERN = re.compile(rf"^(\d{{2}})([{PERSIAN_LETTERS}])(\d{{3}})(\d{{2}})$")

def normalize(text: str) -> str:
    return text.translate(str.maketrans("۰۱۲۳۴۵۶۷۸۹", "0123456789"))

def validate(raw: str):
    m = PATTERN.match(normalize(raw))
    if not m:
        return None
    two, letter, three, region = m.groups()
    return {"plate": f"{two}{letter}{three}", "region": region}

Format rules filter out a large share of errors (e.g. a letter predicted in a digit position).

Step 6: video, tracking and voting

On video, don't trust a single frame. Track each vehicle across frames (ByteTrack or BoT-SORT are built into Ultralytics) and vote across readings:

from collections import Counter, defaultdict

readings = defaultdict(Counter)

for result in model.track(source="rtsp://camera/stream", stream=True, persist=True):
    for box in result.boxes:
        if box.id is None:
            continue
        track_id = int(box.id)
        text = recognize(crop_plate(result.orig_img, box.xyxy[0].tolist()))
        if (plate := validate(text)) and box.conf > 0.5:
            readings[track_id][plate["plate"] + plate["region"]] += 1

# when a track ends, the most common reading wins
best, votes = readings[track_id].most_common(1)[0]

Voting across 5–15 frames turns a 90% single-frame accuracy into a much higher final accuracy and removes flicker.

Measuring accuracy properly

Measure what matters to the business:

Evaluate on a test set from your cameras, separated by day/night and weather, and inspect failures — they tell you exactly what data to add next.

Deployment and serving

Export the trained models to ONNX (or TensorRT on NVIDIA, OpenVINO on Intel CPUs):

model.export(format="onnx", imgsz=640, dynamic=False, simplify=True)

Common architectures:

• Python service (FastAPI) running the models, called by your main application over HTTP or a message queue.
• Direct inference in .NET with Microsoft.ML.OnnxRuntime — no Python in production:

using var session = new InferenceSession("plate-detector.onnx");
var input = new DenseTensor<float>(Preprocess(frame), [1, 3, 640, 640]);
using var results = session.Run([NamedOnnxValue.CreateFromTensor("images", input)]);
var output = results[0].AsTensor<float>();
var plates = PostProcess(output, confidenceThreshold: 0.4f, iouThreshold: 0.5f); // decode boxes + NMS

An ASP.NET Core backend can then store readings, match them against an allow-list (for gates), push live events with SignalR, and expose reports.

Practical tips

• Camera setup is half the accuracy: shutter speed fast enough to avoid motion blur, IR illumination at night, plates at least ~100–150 px wide in the frame.
• Process only a region of interest (the lane) to reduce false detections and CPU time.
• Keep a human-review queue for low-confidence readings in critical systems (tolls, fines).
• Respect privacy laws: plates are personal data — limit retention, secure storage and access.

Summary

An ALPR system combines a YOLO plate detector, a character recognizer trained on your plate format, rule-based validation and multi-frame voting. Data from real deployment conditions is the biggest factor in accuracy; transfer learning, appropriate augmentation and format validation do the rest. Export to ONNX to run efficiently on edge devices or directly inside a .NET application.

FAQ

How many images do I need?

For the plate detector, 1,000–3,000 varied, well-labeled images from your cameras is a solid start. Character recognition needs more examples per character, especially for rare letters.

Can it run in real time on a CPU?

Yes, with a small model (YOLO "n"/"s"), a reduced input size, a region of interest and ONNX Runtime or OpenVINO, multiple frames per second on a modern CPU is realistic. A GPU or edge accelerator handles many streams.

Why is night-time accuracy much lower?

Low light causes noise and blur, and headlights cause glare. Use IR cameras and illumination, and add night images to training data.