Twice the training speed on a BC-250 from tuned matrix kernels
In my training benchmark, LoRA on MiniCPM5-2B ran at 193 tokens a second on my BC-250. The GPU should manage more than that, so I measured where the time went. Almost all of it was in matrix multiplication, running on generic fallback kernels at a fraction of what the chip can do. ROCm doesn't ship tuned kernels for this GPU. After I tuned a set for the shapes the model actually uses, the same benchmark runs at 419 tokens a second in bf16 and 494 in fp16, and the losses match the old run.
The BC-250 is an AMD board with an 8-core Zen 2 CPU and a gfx1013 GPU, an RDNA1-family part. With a patched driver it runs 40 compute units at 2 GHz, which is about 10 TFLOPS of FP32. CPU and GPU share 14 GiB of GDDR6. I'd already installed ROCm 7.2 and built rocBLAS and PyTorch from source for it; this post is about making that stack fast. The scripts, kernel files and full results are on GitHub.
Checking the hardware first
Before blaming software I checked that the GPU runs at full speed. The sysfs clock files said the shader clock was 55 MHz and the memory clock 450 MHz during training, which can't be right at 140 W. So I wrote a small HIP kernel that reads the shader's cycle counter and a fixed 100 MHz reference counter and divides one by the other. Under load it reads 1,990 to 2,015 MHz. A large copy measures 312 GB/s and an add with three streams 480 GB/s. Clocks and memory were fine.
The GPU does get hot. A MiniCPM5 training step draws 137 to 147 W and the die reaches 79 °C, one degree below the point where the clock governor starts to throttle. That matters for long runs and for benchmarks, which I'll come back to.
Where the time went
PyTorch on ROCm sends matrix multiplications to rocBLAS, which picks a kernel from a library that Tensile, AMD's GEMM kernel generator, builds for each GPU. The library has tuned kernels for the GPUs ROCm supports. For gfx1013 it has none, only the generic fallback kernels that rocBLAS keeps for any problem without a tuned entry.
Logging the rocBLAS calls of one training step shows what those kernels have to do. Each linear layer runs three matrix products: the forward pass, the gradient for its input and the gradient for its weight. Each one is a different memory layout, and the fallback kernels handle them very differently. In FP32 the forward pass ran at 2.4 TFLOPS and the weight gradient at 6.2. In fp16 and bf16 the weight gradient ran at 0.10 TFLOPS, sixty times slower than FP32. That explained an odd number from earlier: full training of SmolLM2-135M in bf16 was 3.6 times slower than in FP32.
Tuning the kernels
Tensile can benchmark a range of kernel designs for a list of matrix sizes and write the winners into a logic file that rocBLAS reads. The Tensile in my rocBLAS build already knew gfx1013's instruction set; it just had no logic files for it. So I listed the sizes training uses for MiniCPM5-2B and SmolLM2-135M: every layer at 1,024, 2,048 and 4,096 tokens, the rank-16 LoRA matrices, and the batched products inside attention. I tuned three types: FP32, bf16 and fp16 (both with FP32 accumulation, which is what PyTorch asks for).
The first attempt tried 80 kernel designs per type and would have taken hours. The winners came from a few families, so I cut the search to 32 designs and dropped sizes training never runs. The full tune then took 18 minutes for all three types, and every kernel passed Tensile's own validation. rocBLAS loads the result from a separate directory, so the original kernels stay untouched and one environment variable switches between them.
GEMM throughput, MiniCPM5-2B shapes
2,048 tokens, measured from PyTorch- Fallback kernels
- Tuned kernels
fp16 forward products now run faster than FP32 ones, because they move half the data. The best large products reach 8.8 TFLOPS, close to the GPU's limit. Two FP32 weight-gradient shapes got slower, because I left them out of the tune and the nearest tuned size picks a poor kernel. LoRA never runs those, but they're on the list to fix.
What got in the way
The clock readings were wrong55 MHz at 140 W
The sysfs clock files report values that cannot be true under load, so they say nothing about throttling.
Fix: a HIP kernel that compares the shader cycle counter with the 100 MHz reference counter.
The Tensile client crashed at start-upstd::invalid_argument: stoi
Its hardware monitor tries to parse the same clock files and fails on them.
Fix: turn the hardware monitor off in the tuning config and compute TFLOPS from kernel times. The client also reports throughput per CU as if the GPU had 20.
Tuned kernels could have been ignoreda CUEfficiency tag in every logic file
The tuning config ranks kernels by CU efficiency, and that tag ends up in the library. The stock logic files have no tag, and rocBLAS looks kernels up by its default metric.
Fix: strip the tag when building the library, then confirm the speed-up from PyTorch.
One tuning run stopped making progress15 minutes on a small matrix
Killing the client freed the GPU with no driver errors, and the same problem then ran in seconds. I could not reproduce it.
Fix: a watchdog that kills the client after five minutes without output.
Benchmarks varied by more than a third5.5 to 7.5 s for the same step
Runs that started straight after other GPU work were already near the 80 °C throttle point.
Fix: the benchmark waits for the GPU to cool to 55 °C before each shape, and the fan runs at full speed during measurements. I measured the old kernels again the same way, so both sides of every comparison start cool.
That last one changed the baseline. My earlier benchmark said 145 tokens a second for MiniCPM5 LoRA, but that run went from shape to shape without cooling and was partly throttled. Measured the same way as the tuned runs, the old kernels manage 193. All the comparisons below use the cool-start numbers.
Results
The benchmark trains on random tokens with a fixed seed and records the loss, so a speed-up only counts if the loss matches. For MiniCPM5-2B with LoRA on all linear layers and gradient checkpointing:
MiniCPM5-2B LoRA training
Tokens per second, 2,048 tokens per step- Fallback kernels, bf16
- Tuned kernels, bf16
- Tuned kernels, fp16, chunked loss
With bf16 weights the loss after the timed steps is 13.5012 with the old kernels and 13.5017 with the new ones. The fastest setting adds three changes on top of the kernels. The base weights are fp16 instead of bf16, which is faster here because PyTorch's fp16 products run fastest. The output layer and loss run in chunks of 1,024 tokens: MiniCPM5 has a 130,560-token vocabulary, and the standard loss holds 1.6 GB of logits per 2,048 tokens, so chunking saves that memory. Attention keeps fp16 instead of converting to FP32. Peak memory drops from 9.0 to 7.4 GB.
fp16 changes the numbers slightly, so I checked it on a real fine-tune: the same data, settings and learning rate as my last MiniCPM5 run, 50 steps, with a small eval set.
| Setup | Seconds per step | Tokens/s | Eval loss |
|---|---|---|---|
| Fallback kernels, standard loss, bf16 | 3.35 | 116 | 7.02 → 1.12 after 30 steps |
| Tuned kernels, chunked loss, bf16 | 2.15 | 178 | 7.02 → 0.69 after 50 steps |
| Tuned kernels, chunked loss, fp16 | 2.01 | 190 | 7.02 → 0.59 after 50 steps |
The run with the old kernels reproduces the training losses of my earlier run exactly, and the tuned runs follow the same curve. The real fine-tune gains less than the benchmark, 1.5 to 1.6 times, because its sequences are short and padded and more of each step is host-side work.
SmolLM2-135M full fine-tuning gains less from the kernels, because its matrices are small. FP32 goes from 3,345 to 4,141 tokens a second at 256 tokens × 16 with identical losses. A model that size fits without gradient checkpointing at the shorter sequences, and with the chunked loss and PyTorch's fused AdamW it reaches 5,685, 1.7 times the old speed. bf16 is now 4.8 times faster than before but ends with a higher loss than FP32, because bf16 weights lose small optimiser updates, so FP32 stays the better choice for full training.
Checking the results are right
A faster wrong answer is worse than a slow right one, and the tuned kernels are new code generated for an unsupported GPU. Beyond Tensile's validation, I added a check to my training validation script: FP32, fp16 and bf16 matrix products in all three layouts, over 11 shapes from 1×1×1 up to training sizes and batched attention, each compared with a float64 product on the CPU. It passes, and so do the existing checks of gradients, normalisation layers and 50 optimiser steps against the CPU.
What's left
After tuning, a MiniCPM5 step still spends 62% of its GPU time in matrix products. The best large ones run near the GPU's limit, but long products with 2,048 or 6,144 inner dimensions only reach 4.7 to 5.8 TFLOPS in FP32. The next 15% is data type conversion and attention: PEFT keeps the LoRA weights in FP32, so activations are converted for every adapter, and attention converts to FP32 by default. A wider Tensile search, a fused attention kernel and fusing the small element-wise operations are the obvious next steps.
Heat is the other limit. At 79 °C during a short benchmark, a long training run will reach the throttle point unless the fan curve or the throttle setting changes.
Running it yourself
On a BC-250 with the same ROCm and PyTorch build, scripts/install-tuned-gemm.sh builds the tuned kernels from the logic files in the repository, installs them next to rocBLAS and runs the validation. run-training then uses them by default. To tune for a different model, add its layer shapes to perf/tensile/gen-config.py; the steps are in the repository's skill file.
Inference doesn't go through ROCm at all. A trained adapter converts to a GGUF LoRA file with llama.cpp's converter and loads in its Vulkan server on the Q4_K_M base model, which decodes at 157 tokens a second, or 121 with the adapter applied at run time.