If my ISP allows it, is there any disadvantage to setting my email priority in MX records to a low number?

That is, to a high priority.
If my ISP allows it, is there any disadvantage to setting my email priority in MX records to a low number?
Not really. You have it right - the lower the #, the higher the priority. The only thing is, that the high priority ones get most of the load. So if you want load balancing, consider using same preference MX records - or - use multiple A records, and fall back on the DNS default Round Robin approach. That way, for n servers you have a share of 1/n for each.