drm/radeon/kms: fix r100->r500 CS checker for compressed textures. (v2)

This adds support for compressed textures to the r100->r500 CS
checker, it lets me run openarena and the demos in mesa fine.

Thanks to Maciej Cencora for initial comments.

Changes since v1:
fix calculations with Maciej formulas

Reviewed-by: Maciej Cencora <m.cencora@gmail.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>