The short answer: They used to be a nation of farmers.
The long answer: Congress chose November because the harvest was over and the weather wouldn't be bad enough to prevent people from traveling.
As for Tuesday, people used to have to travel overnight to their polling location. (In 1845, horse was the preferred method of transport.) In an effort to avoid religious days of rest, Congress chose Tuesday, leaving Monday and Wednesday as travel days. Tuesday was voting (and horse-resting) day.