Is Vegetarianism Arbitrary?

Eating a vegetarian dinner at a restaurant,
it was surprising to hear almost everyone at the table
speaking about the arbitrariness of vegetarianism.

If vegetarianism doesn’t matter at all,
why are we Buddhists here, eating what we are,
instead of devouring flesh elsewhere?

I asked this,
but there was no reply,
before the subject changed.

If we all already know, deep down,
that vegetarianism is on higher moral ground,
why think it is arbitrary, especially for the animals?