Have you tried... Changing their token?
To perhaps, one that is itself functionally invisible?
Perhaps a fully transparent image?
And perhaps make it really really small?

In my experience as a player, in 5e, and FG Unity, with the light sources and line of sight extensions active,
I believe our DM has been able to effectively make my toekn invisible to the party.
I believe, I get a ghostly toekn effect myself?

So, that might be the solution for you. Consider using the Line of Sight extensions if you aren't. It might solve your problem.

(I know when I'M Blinded, I can't see Jack squat.