Very true. The Koch brothers are very involved in libertarian causes. If they controlled the tea party organizations, you wouldn't see some of them talking against gay marriage, open borders, etc. Instead, you'd see them fighting against drug prohibition, occupational licensing, marriage licensing, military intervention and presence around the world, etc (in addition to fiscal issues).
How about wikipedia? I'm most familiar with this brother even though I haven't heard a great deal.
en.wikipedia.org/wiki/David_H._Koch
Take a look at "Political Career" section, but almost the entire thing tells of his politics and what he stands for.