Practically, if you're buying vegetables I think you can consider them vegan regardless of how they were grown. Bees are exploited for several crops which many vegans eat. Many crops are grown using pesticides and insecticides which harm animals around those crops.
Now, the use of manure for fertilser can be problematic for health reasons, as it's a great way to spread harmful bacteria and contaminate those crops. If you have the knowledge and choice, it would probably be best to stay away from manure fertilised crops. But very few of us have that ability, and we're still vegan.
If you're growing your own food, I would say it isn't vegan to buy manure for fertiliser. That usually comes from cows or horses who were exploited, and buying their poop supports the system. Soil made from compost actually makes excellent fertiliser, and the only animals used are earth worms, who are doing what they'd normally do and I don't think are being exploited in the process.