They've never made an RS8. If you've seen one, it's probably a rebadged car.
Typically, the RS models have major improvements and increases in horsepower over the S models.
B4 S2: 217 HP => B4 RS2: 311 HP
B5 S4: 261 HP => B5 RS4: 375 HP
C5 S6: 335 HP => C5 RS6: 444 HP
B7 S4: 340 HP => B7 RS4: 420 HP
C6 S6: 430 HP => C6 RS6: 575 HP
A5 TTS: 265HP => A5 TTRS: 335 HP
If RS8 (based on the A8/S8) were to exist, I'd imagine it would have to have at least 580 HP given the S8 has 444 and the RS6 has 575.
At one point in it's development the V10 R8 was called RS8. (It was going to have a twin turbo V10, but unfortunately the engine got too hot in the mid engine layout, so they went with the NA V10).
I believe they changed the RS8 to R8 5.2 to avoid confusion.