So the Tagtool Server is running on my Laptop and my iPad is connected to the Tagtool session. I launched the Tagtool Server in a window mode with 1024 x 768 resolution. When you ALT-Click the Tagtool Server MacOSX gives you some options before the app launches.
I am not very familiar with OBS - just downloaded it as well. So i simply select the window where the The New Tagtool is running and i can start the recording. My laptop is not very good so even with 1024 x 768 resolution i barely it the 30 FPS mark.
Within the settings of OBS Studio you can tweak the output resolution as well and as far as i can tell the main purpose of this software is optimised for video streaming. To get a decent quality you need good hardware as well.