I do find your phrase "social sub-group" somewhat demeaning. but maybe I'm reading into it.
I mean it in the sense that if veganism were just a diet, all we have is a social in-group. One can be a member just by not eating meat. There is not necessarily any moral backing for that choice. There might be some kind of sentimentalism at work, but even that isn't necessary. On the other hand, if you think veganism is talking about rights and justice, then you have a moral matter. And moral matters apply to all of us; what's always in question is the extent to which we think a moral matter should be applied in practice.
My metaphor is that just as veganism is on a spectrum of ethics, red is on the visible light spectrum. just as its a little hard to say where red ends and orange begins, red does end before orange begins. Same with veganism which ends before Carnism starts. Just like red, where you can add extra white and black to get your pinks and magentas, we have the different schools of veganism too.
I'm not sure I quite follow your metaphor. Colours don't really exist, they are equipment dependent responses to electromagnetic radiation that give us information about the world. And they depend upon there being the full spectrum because the information they give is primarily about differences. Red is not green, for example. If someone could only see red colours (ie everything were shades of red), the whole point of colour would be lost.
In similar vein, veganism as a moral attitude about the world depends on the full range of moral beliefs/attitudes about relations between individuals and groups. Before cities, states and agriculture, there was no veganism. The idea wasn't needed to make sense of the world, but some kind of moral take was nonetheless present. People enslaved, killed, fought with, dispossessed and otherwise treated each other in unpleasant ways. Yet they also had rules about how to treat others depending on circumstance and context so that life wasn't always about killing and fighting etc. The same with other animals.
But as population grew and people began to live in larger and more stable groups, rules were needed about how to treat diverse elements within groups. As intra-group relations began to reflect more than simple trade relations so too did the need for rules to manage relations with other groups.
Philip Kitcher describes ethics as the efforts of humans to resolve failures of altruism. That may be right. In small groups, all we need are ways to manage internal relations to maximise altruism. That gets harder with larger and more diverse groups. Moral rules about how to treat others become more complicated and address more circumstances and contexts. Eventually we have reached the point where contexts can extend to include other species, partly informed by ideas about what's right and partly by science.
The point though - as I see it - is that the ethical project Kitcher describes and into which vegan moral beliefs fall consists of a continuum of ethical ideas. Veganism isn't a thing apart, it is an intrinsic part of everyday moral belief formation and ethical practice. The tenets of veganism align exactly with how we have come to shape the best ways to act justly and fairly to each other - in a real sense veganism is the idea we take these beliefs and practices beyond our own species. And it doesn't even require that we do so for all species. It depends, again, on circumstance and context. Just as our intra-human relations would rapidly change if society fell into disrepair.
Simply put, do unto others as you'd have them do unto you, whenever you can, even when "others" includes other species.