Hello there, from France ! I know I am years too late, but since I managed to use cel shading in this game, I thought I might leave you a message.
As you can see, the game looks stunning (picture enclosed) !
First of all, and as you suspected, you have to make the game run under a later version of directx : I encountered the same problems as you, but Dg Vodoo did the trick for me : you can download it here : Dg Voodoo 2
Just extract the "exe" and the "conf" files in your executable folder, as well as the 3 files contained in the "MS" folder of the zipped file (D3D8.dll, D3Dlmm.dll, and DDraw.dll).
Now, as you launch the game, you should see a "Dg Voodoo" watermark on the corner of your screen (you can disable it later through Dg Voodoo).
Download and install ReShade : ReShade
When asked for the shaders you want to download, choose (at least) :
In addition, download the "Outline" shader here : Outline shader link
Finally, download the UIMask.png here : My UI Mask
Replace the one in the "textures" folder by this one
Now, when starting the game, you should get the following screen, confirming reshade is in use :
Press the "'home" key (the one with an arrow above your "end" key), and Reshade should open.
Put the shaders in the following order and check them :
Technicolor 2 (to get vivid colors, as Voodoo seems to "wash out" the colors a bit),
Levels (Emphasizes the contrast without messing the cel shading, in my opinion),
UIMask_Top (Tells Reshade not to apply "outline" to UI),
Outline (the actual cel shading effect),
SMAA (to smooth up the picture, so that the cel shading will look less "PS2" like).
On the picture below are my presets : I think it's best not to push "outline" too far, as the picture gets all messed up. Usually, a look at the characters' eyes will tell you if you put it too far, as cel shading will make their eyes black.
Levels : leave default,
Outline : "normal-depth edge detection" + edge detection (1) + edge slope (1.600) + outline opacity (0.666)
Technicolor2 : Saturation (1.100), leave the rest to default,
UIMask : leave default,
SMAA : leave defaut.
Feel free to try others presets, or use another shaders for colors and AA.
As for me, I know I just COULDN'T play this game without celshading anymore.
Hope I could help after all these years.
See ya !