Here I hope are the most clear and insightful reasons as to why this hasn't been historically realised, or even simply hasn't happened. Further insights and ideas, as always, are welcome.
Bad Install Experience
Customer experiences on games for linux aren't typically great. First off is how you get the game - in the past, the most typical method has been to get Linux users to buy a copy of the Windows version of the game, then download the Linux binary. This got around traditional support worries like producing a boxed Linux version that publishers may have not wanted to risk putting significant money and resources behind, but nevertheless this produces a worse experience, and thus less incentive to buy.
First you have to buy the Windows version that indeed, will show up as a Windows version sale (more on that later), download the actual installer or executable separately, meaning issues regarding customer knowledge of where to put said executable, and in case of an installer having to go through the terminal with ./game-installer.run which then asks you where you want to install the files (why should I have to know that? Surely you're the developer and know better?), copy the data files from the CD into the directory you installed it, possible dependency problems and other such things. Even worse is if the Linux installer is hosted on a more obscure FTP site the developer hosts that isn't immediately or prominently advertised, and may indeed simply point to a torrent file.
Aside from more obtuse installers that have to be run from the command line, you may very well end up with a Windows like install wizard that looks like a bad approximation of Windows 95. It does the job, but doesn't fit in with the rest of the environment at all and has other usability issues.
Contrast this to World of Goo, the most prominent example of successful Linux game sales - native .deb and .rpm that subsequently integrates well with just about every distro out there with easy 1/2 click installs, allowing even easy reinstall and uninstall, alongside an extract once .tar.gz archive. Any dependency problems can be handled automagically by the package manager.
Actual Audience Figures Are Hidden
As noted previously, major games put out for Linux have often distributed the actual game data (textures, sound etc.) separately from the installer or executable. Guess how they do that? By getting people to buy the Windows version for the game data, then download the executable. Whilst download figures are better than nothing, they can again be unreliable. People downloading in cases of hardware failure, or perhaps returning to the game after a long period mean downloads may be duplicated. The existence of WINE may also mean that people who've bought the Windows version but own and use Linux simply run it via that method, to some extent meaning "hidden" sales that could have counted towards Linux audience figures.
This also produces less incentive to buy - what's the point of supporting Linux games if they're just going to show up as Windows sales? Indeed it may be better to simply wait a little while and get it second hand, which may be a perfectly feasible option by the time the Linux installer/executable arrives, and for Linux users used to waiting before buying.
Again, contrast this to World of Goo and Mystic Mine. Buying for any particular platform counts specifically towards that platform, in a way that's more reliable and inherently compilable into easily understandable data for each one.
Disinteresting and Unoriginal Games
Just because Linux has a lack of overall commercial games doesn't mean someone who produces games has no competition. Unreal Tournament and Quake have plenty of clones and competition, with one or two particularly well known and stand out examples. Even if in certain areas they may be behind in overall visual quality or consistency doesn't mean people aren't happy with them. Even better is that they're free and can integrate with distributions and package management tools.
One thing we don't need is more UT and Quake style games, even if developers happen to think it's the largest known audience. World of Goo did well because it had innovative and fun gameplay, fantastic production values and a unique visual style. It's style and gameplay really made it stand out, with the production values being the icing on the cake, especially the music. Even with competition from free online games and other standard puzzle titles, none came close.
Do not focus on the free aspect however. Saying you can't compete with free is saying you can't compete at all:
In other words, companies look to add some value to the goods that makes their goods better than the competition in some way -- and that unique value helps them command a profit. But, the nature of the competitive market is that it's always shifting, so that everyone needs to keep on innovating, or any innovation will be matched (and usually surpassed) by competitors. That's good for everyone. It keeps a market dynamic and growing and helps out everyone.
So, let's go back to the "can't compete with free" statement. Anyone who says that is effectively saying that they can't figure out a way to add value that will make someone buy something above marginal cost -- but it's no different if the good is free or at a cost. Let's take a simple example. Say I own a factory that cost me $100 million to build (fixed cost) and it produces cars that each cost $20,000 to build (marginal cost). If the market is perfectly competitive, then eventually I'm going to be forced to sell those cars at $20,000 -- leaving no profit. Now, let's look at a different situation. Let's say that I want to make a movie. It costs me $100 million to make the movie (fixed cost) and copies of that movie each cost me $0 (marginal cost -- assuming digital distribution and that bandwidth and computing power are also fixed costs). Now, again, if the market is competitive and I'm forced to price at marginal cost, then the scenario is identical to the automobile factory. My net outlay is $100 million. My profit is zero. Every new item I make brings back in cash exactly what it costs to make the copy -- so the net result is the same. It's no different that the good is priced at $0 or $20,000 -- so long as the market is competitive.
Small Markets Don't Feed Large Companies
This goes off further into the more complex side of business and economics, but I shouldn't think this aspect is difficult to understand. Companies that are already established and large have certain minimum needs in the revenue they gain and their pressure for growth from investors will inherently forbid them from entering that market. Irrespective of whether the market is indeed large enough to support a business or profitable even for businesses already in that market, the current size of it makes it difficult for larger companies seeking growth to commit themselves to it.
This is typically the situation with the likes of Ubisoft, EA and others. They are already large companies, in which markets like OS X or Linux will at most be typically nice side lines, but not large enough to demand serious commitment. This is in contrast to smaller companies and indies who have trouble getting attention, no marketing department and little established credibility or franchises. They're in a situation where they're more likely to take whatever they can get, and options like being cross platform seem much more like advantages and take precedent over pure practical reasons like simplicity of setup that larger developers and publishers will be more worried about. This is something put across quite well by Wolfire Games:
1. It's good to be a big fish in a small pondAs a pretty niche independent game, Lugaru was never covered by PC Gamer, IGN, and other behemoth media publications. However, it was just large enough to get covered in a variety of Mac journals. One website, Inside Mac Games fell in love with it and posted about it all the time.
If you support Macs, even a small indie video game can rapidly spread throughout the community whereas the huge Windows market might just ignore you.
2. More platforms means more opportunitiesAs an indie video game studio, we don't have many resources in the way of marketing. It's embarrassing how much we rely on serendipity, for example, getting posted on fun-motion right when the admin decides to take a break, or getting called up by Igromania randomly.
We have had more than our fair share of Mac serendipity though. A recent example: Lugaru was promoted in MacHeist's Giving Tree. This generated the equivalent of like three diggings worth of traffic. We were up to 30 requests / second at its peak and dwarfed any event in the history of our Google Analytics. If we didn't have a Mac OS X build, we simply wouldn't have had this opportunity.
3. Vocal minoritiesHaving a Linux build meant coverage on Slashdot. This of course generated huge interest in not just the Linux version of Lugaru, but the Windows and Mac versions too. Lugaru also made an appearance in a few Linux magazines. A lot of people heard about and supported Lugaru simply because we had a Linux build.
4. You can't choose your evangelistsIf you're familiar with Guy Kawasaki's philosophy of evangelism, this shouldn't be any surprise. You'll notice that a small minority of your users will go crazy with your game and spread it all over the place. On the internet, all it takes is one thread on a popular forum, and you've literally got hundreds or thousands of new visitors. Basically, a small amount of your users can make a huge difference for you, and they might be Mac and Linux users.
A notable number of the Wolfire fans who hang out in IRC and spread Overgrowth like crazy are Mac users and we would be much smaller without them.
5. You can't choose your power usersIn the same vein as the above, you never know who the movers and shakers are going to be in your community. In Wolfire's case, we are forever indebted to Wolfire forum regular, Silb. He actually reverse-engineered portions of Lugaru and made a kick-ass, extremely popular replacement campaign for the game, providing a huge amount of extra content to other people. His single, epic thread has been viewed over a hundred thousand times.
Oh yeah. He's a Mac user.
This is why small porting outlets can also survive, even as the major publishers themselves that hand over the rights to put out these games continue to generally ignore or lack commitment to other OS and hardware platforms.
Lack Of Marketing
Due to what was mentioned previously, this also means Linux versions have typically come out a lot later due to not being any kind of priority. By the time they do come out, the publisher doesn't care any more. They've had their big Hollywood styled blockbuster launch, and now their advertising has died down or non-existent, even just a few weeks after the game launched.
You can see this very well in sales and 2nd hand trends - within a fairly short period, the game just drops out of the charts, whilst the market is quickly flooded with second hand copies being traded in after a game has been completed. This also means that Linux buyers at that point buy 2nd hand copies either out of price or availability, meaning the audience figures continue to be hidden. This can lead into my next point...
As noted above, this is never the main point, but does become an issue with larger games that regularly cost $50 - $60 new, especially if that game already has plenty of existing competition in the same style and genre. Even worse if the game has already been out a while and has reduced prices for the Windows version or large availability of 2nd hand copies, whilst the Linux version either as a download or otherwise is still much more expensive.
A good case of this is the site Tuxgames where many Linux versions of games are still very expensive. Buying Doom 3 for example costs $49 - an 8 year old game widely available cheaply for Windows, and Shadowgrounds Survivor is $33 for a 2 year old game. Other options like buying directly from LGP aren't always much better - their RRP for X3: Reunion is listed at £40 for a 3 year old game.
Correction: I hadn't realised I had pointed to the collectors edition of X3: Reunion, which is a bad example due to real value for money like a t-shirt, copy of the book Farnhams Legend and being signed by the LGP CEO with a limited 500 copy run. Though the normal edition is still £30
Once again, this hides real quantifiable audience figures and serves to further make Linux users feel worthless.
Suffice to say, almost every action taken by previous game publishers in how willing they are towards supporting Linux has been self defeating. Linux can only grow with at least half decent support from third parties, and third party ecosystems can only grow if they support Linux in a way that actually gives people a good experience. Yet oddly this becomes at least partially understandable when viewed from the context of the largest publishers and developers, where the incentive for growth naturally points them towards large or rapidly growing markets.
This doesn't mean the Linux games market is doomed however. It actually means the the biggest opportunity for commercial games is for small to medium sized publishers and developers, where there is at least some real continual incentive relative to their own size. Who knows, the ones that properly treat their customers with respect and timely support could properly nurture and cultivate a thriving place for innovative games with great experiences all the way from install to final boss.
I guess it's no different from any other market then.