in the guest system, do a Start -> Run -> dxdiag
click on DirectX Files Tab, if you see yellow exclamation marks mean the guest D3D should be working.
The Test Direct3D at the Display tab shouldn't be grey out as well.
btw this is for xp
Actually for the newer Guest Additions 3.24 it works without safe mode, but iz still better to do it in safe mode.