jsm: clean up "serial: jsm: correctly support 4 8 port boards"

Remove unneeded casts.

Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Alan Cox <alan@linux.intel.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>