Yes, there are a couple of settings you can select to improve the quality.
To help you locate the choices, I need to know for sure if you are using v8 or v9.
From your title it looks like you are using v9 ?? OK, here's the first step.
1. Open DVDFab and then click on the small wheel, (looks like a gear) found at the top right of your page. It's right
next to the question mark.
2. When that page opens, select the blue tab at the top that reads: Blu-ray.
3. Then on the left side of the page click on the title: "Blu-ray Copy"
4. Then notice at the bottom you have a choice of encoding settings: "Normal quality" or "High Quality"
Try this first and see if it helps. Takes a bit longer to encode, but it can make a quality difference.