Using NVENC HEVC video encoding - Part 1

09 October 2025 at 11:35:00

Introducing the problem

I'm going on a video encoding journey. There's a few problems that guide me on the journey I am about to go through:

• I have a relatively limited amount of hard drive storage, and I don't have the money to buy more hard drives.
• I have a decent GPU, but it's too old for AV1 encoding, and I don't have the money to buy a new GPU.
• I don't want to delete anything because I am a data hoarder.

There's one more important aspect: I enjoy maintaining my own streaming infrastructure.

Let's find a solution

To my benefit, all the hardware I use supports HEVC hardware decoding. I have a GPU that supports HEVC hardware encoding. HEVC allows you to compress H.264 (and older) video formats for a nice space savings. Most of my video is H.264, because I only really store 1080P media. So, it seems like the answer is pretty clear: it's time to investigate HEVC encoding.

NVIDIA provides a solution in their GPUs in the form of NVENC (there is also a solution to decode video, called NVDEC). NVENC can handle H.264, H.265, and in very recent GPUs, AV1. Since this is dedicated silicon just to the task of encoding video, it can much more efficiently do the task than the CPU can. Offloading your computation to high efficiency, high performance dedicated silicon sure is magic!

Now, I've done plenty of HEVC encoding before, but I just eyeballed it. Mess with settings by hand in FFmpeg until I've gotten something passable. This works if you are encoding something specific, or don't mind edge cases where your settings suck, but I'm encoding a pretty large library. I'd like to find settings that work as well as possible, while also giving myself good space savings.

Some previous knowledge

From my previous playing around, I've found when CPU encoding, around 1Mbps tends to do fine for 1080P cartoons. I've also found that the GPU tends to be slightly less efficient. However, using CPU encoding takes a lot longer, and I want to encode a lot of videos, so the trade off should be worth it to me.

We'll start with investigating lower bandwidth ranges, from about 500Kbps to 3Mbps. I also know some shows are a lot harder to encode than others. For reference, your typical cartoon episode from a service like Disney+ is around 6Mbps in H.264. So the compressed video having a bitrate of 3Mbps would roughly cut the file size in half (ignoring audio tracks, subtitle tracks, and container overhead, which we aren't going to deal with here).

Preliminary results

Gravity Falls S01E01

First, a little explainer: there are two independent variables (bitrate and preset), and then a dependent variable, the VMAF score (the number in each cell). Bitrate controls the file size, it's how much information the encoder can store per second. The preset is essentially time, how long the encoder can spend on compressing the frame. The VMAF score is a "perceptual video quality assessment" algorithm that Netflix published to help people assess the quality of their video compression.

I have generally considered a VMAF score of 92 to be the lowest acceptable score, and here we reach that at 1250Kbps with preset 4. That's some nice savings! However, this episode of Gravity Falls could just be really easy to encode. Some other things to note. Presets 1, 2, and 3 are all identical. Preset 6 and 7 are practically identical. 250Kbps is so far from ever scoring well that I may as well not test it in the future, which I did expect, but I figured I'd try it.

I remember a different show was a lot harder to encode when I was playing with AV1...

Primos S01E01

The first occurance of a score of 92 is 1750Kbps with preset 6. That's a fairly significant increase from the Gravity Falls episode. As you can see, I refined my search a little here. I dropped the 250Kbps bandwidth tests, and I only went up to 2500Kbps. I actually originally only went to 2000Kbps because of how well the Gravity Falls episode went, but I was unhappy with my highest score only being 92.9, so I went on to 2250Kbps and 2500Kbps.

It's seeming like we can refine our search further by dropping presets 1 and 2, and dropping preset 7, since they're all still (practically) identical to other presets. But, I'd like to at least see a result with a live action show first. Maybe the presets become more distinguished in 3D.

Slow Horses S01E01

Well. That didn't match my expectations. NVENC apparently found this easier to encode than the 2D cartoons, according to VMAF. Even 1000Kbps preset 1 scored 92.7. The difference between the presets is much lower than the cartoons, too. It does at least confirm we can elimate presets 1, 2, and 7 from our future testing. We can also safely eliminate 500Kbps.

A "conclusion"

Clearly, more data is needed. The optimal preset is all over the place between the three episodes of TV I tested. I could go for 1750Kbps preset 6, but I would like some more data. I suppose it's time to encode and test a bunch more TV. Thankfully, we did eliminate a decent number of settings to test, so it should help compensate for having to do this dozens more times.

Off to go encode! See you soon, hopefully.