Lets look at some facts, rather than getting hung up on the name "Google".
There are some protocols for two factor authentication. Namely TOTP and HOTP. These protocols are open and well documented*. These things work based on some interesting math (cryptography). It calculates a 4 digit "one time password" based on your "key" and the current time (or a login counter). These codes are not sent by mail or SMS or anything like that, they are calculated by your "client". A client could be an app on your phone, a hardware token (USB thing) or even a desktop application.
The Google Authenticator is only ONE implementation of these protocols. It's open source, and if you don't trust it, because it has the name "Google" attached to it, then you have a host of other apps at your disposal. Such as Authy or FreeOTP to name two alternatives. So, you can see, Google can not get hold of this data.
Yubikeys are hardware tokens that can (some models) do that. But some Yubikeys implement a different two factor authentication scheme called U2F, which is also great. Of course there are other alternatives for such hardware tokens.
The point is, maybe you don't want the authentication to run over the same channel (e.g. E-Mail) that is also used to login or RESET YOUR PASSWORD! Because if you do lose control of your mail account, then it's game over. That's why I'd support the idea of supporting TOTP/HOTP/U2F protocols for two factor authentication (at least I'd prefer it over sending authentication codes by mail).
*) RFC 4226 (HOTP) and RFC 6238 (TOTP)
Post edited October 17, 2016 by jug