For optimal quality DVR captures are at a max 1/4 of your screen size by default.
See Settings > DVR > Request Best Output Size - tool tips for more details. If you switch that off you can set the output width and size. Have a look at the tooltips so you can understand its effects.
As you are running in a VM you will no doubt want to make that screen as large as possible.