They seem to have thought of that: "Unless you elect otherwise, Dash Button responds only to your first press until your order is delivered"