It seems strange that your ability to use CUDA hardware acceleration varies from movie to movie. Which encoding profile are you using? I typically use MKV.4K.H265.10bit, but other H265 rips have taken advantage of my NVIDIA GeForce GTX 1070 without a problem.
Here are my current A/V Codec settings:

Have you tried resetting to defaults in DVDFab and trying again?
Good luck!
- Dave