← Seungheon Lee

Computer Vision Model Optimization for NPU Inference

Sep – Dec 2024 · AI Semiconductor

featured github.com/Githarold ↗ yolov8s → warboy YOLOv8s FP32 baseline ONNX export · BN folding · graph simplify 0.449 mAP Post-training quantization, FP32 → INT8 MIN_MAX_ASYM of 5 methods · 500 images 0.427 mAP −5% Furiosa Warboy NPU runtime dataloader · grid search · NMS · async scheduling NPU utilization vs. baseline +533%
Grand Prize · Minister of Science & ICT Award · KRW 40M

Grand Prize at the 1st AI Semiconductor Competition (Minister of Science & ICT Award). Collaborated with Furiosa AI in a 2-member team and achieved up to 533% NPU utilization improvement over baseline while simultaneously increasing quantization accuracy. Established an MLOps pipeline with W&B and MLflow for systematic experiment tracking and model management.

PyTorchQuantization Furiosa NPUComputer Vision Benchmarking
Problem
The SDK's default runner was synchronous with a batch size of 1. Only 3% of the 16 GB memory was in use, and NPU compute utilization sat at 24.9%.
Approach
ONNX export with BN folding and graph simplification, INT8 post-training quantization, uint8 inputs. Profiling showed NMS taking 14.25% of inference time; moving it to torchvision.ops cut that to 1.37%. The synchronous runner was replaced with a batched async queue, and dataloader settings were grid-searched.
My part
MLOps and proof-of-concept work in a two-person team: W&B and MLflow tracking for 1,000+ reproducible experiments, a shared experiment script, and early validation of the optimization ideas.
Result
Up to +533% NPU utilization on YOLOv8s and +252% on YOLOv8s-pose over the baseline. 1st of 244 teams.
GitHub ↗