That's right: For the first 122 years of Wisconsin's existence, if Wisconsinites were sick of their governor they could get rid of him in the second year. For our first 122 years, we wouldn't have needed to recall Walker this summer, because we could just vote him out this fall.
In fact, Wisconsin's founders even considered having a Governor serve only a one year term! You see, it was really important to both the Wisconsin and United States founders that state-level elected leaders follow the wishes of the people verrrrry closely-- hence, the short terms and the elections always just around the corner.
Unfortunately, Wisconsin as part of a national movement toward longer Governor terms, adopted a constitutional amendment to lenthen the governors term to four years in 1968 and gave little thought to when it would start: 1970... a non-presidential, off-year and low-turnout election year; and ensuring that every governor election thereafter would also be a non-presidental, off-year and low-turnout election. (And that's just what happened in 2010, where only about half of eligible voters participated in the election and Walker only got a little over half of those-- meaning that Walker got a stamp of approval from only about a quarter of eligible voters.)
In other words, up until 1970, many more eligible Wisconsin voters participated in the selection of their governor and, as an accountability safeguard, they did it every two years. Then, they changed it, with the backers of the longer governor term arguing that accountability would still be intact: if we do get a governor that goes haywire after we elect him or her, we can always recall them.
And that, boys and girls, is why holding an election for Walker in 2012 not only doesn't go against the wishes of our founders-- its what they intended.