Dakadin,
That does get messy. If you were to do it that way, the modifiers would start with 0 on the "96" table and increase to +4 on the "100" table, because the open-ended roll result for 96 would actually start at 97 (96+1). Doing it this way would also change the statistical probability of getting another high end roll, so that also would need to be taken into consideration.
I am not a statistician, so I don't know the math well enough, but if I were putting the tables together, I think I would live with the error of treating 96-100 the same and not adding the 2nd roll to the 1st roll. It may decrease the probability of 1-5 on the subsequent table (97-101 on a manual 2nd roll), but it simplifies the process.
This is really a FGC issue, at least for high open-ended d100 rolls, because FGU can handle them with the exploding die expression. A standard high open-ended roll would be expressed as "/die d100e96". Low open-ended rolls are still an issue.
I found a statistical discussion of open-ended rolls here:
https://www.guildcompanion.com/scrol...cearticle.html
Bale Nomad