ASoC: Blackfin: Fix AD1980/1 build with MMAP support disabled

clean up redudent code and correct building problem in non-mmap mode

Signed-off-by: Cliff Cai <cliff.cai@analog.com>
Signed-off-by: Bryan Wu <cooloney@kernel.org>
Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>