I realize it has been a while since I opened this ticket, but wanted to update as I forgot. I was expecting the low number to have priority, and it is actually the higher number that takes priority
For example,
I changed the Approve button property to <Property name="display_order_priority" value="10"/>
and the Return button to <Property name="display_order_priority" value="5"/>
and the Reject button to <Property name="display_order_priority" value="1"/>