So, now, you send the motor somewhere, then read the switches, and finally, compute where the servo should be. You're getting closer, but it still isn't right. The LAST thing that should happen is the motor.write() call.
You need to have a look at debouncing, next.