As I understand it, horse riding would not be vegan since it is using an animal. I don't know enough about it to say whether or not it's necessarily cruel- I can imagine someone having a horse, riding him or her, and still genuinely caring about that horse's wellbeing- but I'm pretty sure that the possibility of cruelty will always be there so long as the horses are seen primarily as something to be used.
It's true that horses get much more consideration from humans generally than cows, chickens, or pigs. Remember the outcry that happened over a planned horse slaughterhouse someplace in the US to deal with horses who were no longer ridable (or maybe it was going to be for feral horses on public lands)?