Notebook of Dan

Google Revealed: The IT Strategy That Makes It Work
A unique mix of internally developed software, open source, made-to-order hardware, and people management is the secret behind the search engine.

In Building 43 at Google's Mountain View, Calif., headquarters is a video screen that depicts the world as seen in Google Earth. Across a revolving globe, streams of colorful pixels, like sparks from a Roman candle, mark the geographic origin of queries coming in to Google's search engine. It's a real-time representation of Google as the nexus of human curiosity.

Google is different. And it's different not only because its thinking is original and its applications unique--witness search queries morphed into a lobby display of bursting color--but because the company's unconventional IT strategy makes it so. Commodity hardware and free software hardly seem like the seeds of an empire, yet Google has turned them into an unmatched distributed computing platform that supports its wildly popular search engine, plus a burgeoning number of applications. We used to call them consumer applications, but Google changed that. Businesses also use them because, well, Google is different.

The IT infrastructure behind Google's Web services doesn't matter much to the millions of people conducting searches, but it's everything to the hundreds of engineers dedicated to Google's mission of organizing the world's information and making it "universally accessible and useful." That calls for an IT plan that matches the company's business vision in scope and ambition.

Google managers tend to be reticent on the subject of IT strategy, they're loath to talk about specific vendors or products, and they clam up when asked about their servers and data centers. But a day spent with some of the company's IT leaders reveals there's more to Google's IT operations than a search engine running on a massive server farm. Behind the seeming simplicity is a mash-up of internally developed software, made-to-order hardware, artificial intelligence, obsession with performance, and an unorthodox approach to people management.
There's a lesson in Google's IT philosophy for other companies: Shun the herding instinct that leads toward the same systems and software everyone else is using. There may well be competitive advantages in doing things your own way.
"Culture drives the way you do things," says Douglas Merrill, VP of engineering and Google's de facto CIO. "To the extent, like us, your organizational culture is unusual in important ways, you will have to build different ways of running your traditional systems."

TAKE OUR POLL
Google's A Do-It-Yourselfer.
Should You Be Too?
Google's great IT advantage is its ability to build high-performance systems that are cost efficient (we didn't say cheap) and that scale to massive workloads. Because of that, IT consultant Stephen Arnold argues, Google enjoys huge cost advantages over competitors such as Amazon, eBay, Microsoft, and Yahoo. Google's programmers are 50% to 100% more productive than their peers at other Web companies, a result of the custom libraries Google developed to support programming of massively parallel systems, Arnold says. He estimates the company's competitors have to spend four times as much to keep up.
Pimp My Server
How does Google do it? For one thing, Merrill says, "we build hardware." Google doesn't manufacture computer systems, but it does order them to its own specifications, then installs and tunes them like something out of MTV's Pimp My Ride. "What it comes down to is we're very good at buying commodity servers and using them to their fullest, to the point where they're almost so damn hot they'll melt," open source program manager Chris DiBona says.
That hands-on approach, born of the frugality of a garage startup, persists because Google's scale demands it. Google has between 200,000 and 450,000 servers spread among up to 65 data centers, depending on how you define them and who's doing the counting. And those numbers continue to rise.
The company won't discuss these estimates; it considers such numbers to be a competitive advantage. In fact, one of the things Google likes about open source software is that it facilitates secrecy. "If we had to go and buy software licenses, or code licenses, based on seats, people would absolutely know what the Google infrastructure looks like," DiBona says. "The use of open source software, that's one more way we can control our destiny."
Scale works in Google's favor. The marginal advantage of custom-built servers becomes significant when multiplied by hundreds of thousands of machines. The company is constructing a 30-acre data center along the Columbia River in The Dalles, Ore., where it can get low-priced hydroelectric power for computing and cooling (see story, "Google Goes Its Own Way In The Data Center").

Open source software lets Google control its own destiny, DiBona says.

Google organizes its machines, which run Linux, into "cells," which DiBona describes as a kind of disk drive for Internet services. (Not to be confused with Gdrive, the long-rumored Google hosted storage service. "There is no Gdrive," a spokeswoman insists.) Software programs reside on racks of inexpensive computers, and programmers decide how much redundancy to give them. The cells take the place of commercial storage equipment; DiBona says Google's cells are cheaper to create and maintain, and he hints they can handle more data, too.
No level of minutiae escapes Google's attention. For years, the company's engineers have studied the inner workings of microprocessors, and as Google continues to scale up, chips tuned to its unique needs could become a necessity. In a paper published in an industry journal last year, distinguished engineer Luiz Barroso said key workloads at Google suffered from single-core designs in recent years. Many server-side apps, such as serving the Google search index, don't process in parallel well at the instruction level on such chips.
The arrival of more chip-level parallelism as Advanced Micro Devices, Intel, and Sun Microsystems build multiple cores onto their chips has been a boon, says Barroso, a former chip designer at Digital Equipment and Compaq.
Google has even considered designing its own computer chips, but such a bold move may be unnecessary given industry trends. "Designing a microprocessor is a complex and costly task," says Urs Holzle, senior VP of operations. Google prefers to work with chip manufacturers to make sure they understand its applications and design chips that are a good fit. It's been advocating designs that focus on aggregate throughput and performance per watt rather than single-thread peak performance. "Recent trends in multicore CPUs very much follow that direction," says Holzle.
Custom Tailored
To wring every ounce of performance from its hardware, Google writes custom software--lots of it. Major innovations include MapReduce, a programming model to simplify processing and create large data sets; BigTable, a system for storing and managing massive amounts of data; Sawzall, an interpreted programming language for analyzing large data sets in a distributed computing environment; Google File System, a distributed file system for data-intensive applications; and Google Workqueue, a system that groups queries and schedules them for distributed processing.
It's in tools like Sawzall that Google's obsessive focus on computational efficiency becomes clear. Not every company tackles productivity at such a fundamental level, but for Google it made sense to develop a programming language specifically to deal with data sets that are too large to fit in a conventional relational database. Even though other programming tools could be used to deal with the problem, Google engineers developed a custom solution for the sake of efficiency. Google engineers contend Sawzall programs are a fraction the size of equivalent MapReduce programs in C++ and significantly easier to write.
Such concerns explain why Google isn't content with the standard Linux kernel; it runs a modified kernel tuned to its needs. By tinkering with the low-level behavior of Linux, Google engineers have solved data corruption and bottleneck problems, while increasing overall system reliability. The kernel alterations also made Google's computer clusters faster by making them communicate more efficiently. Of course, Google experiences the occasional system glitch, and, when it does, millions of users can be affected. Three years ago, a system failure hobbled 20% of search traffic for 30 minutes.
Google created its own Web server instead of using the open source Apache Web server, which underpins more than 60% of Web sites. Google's Web server can run on more machines and balance workloads among servers more effectively than Apache for Google's large code base, which contains lots of dependencies among programs, DiBona says. The company's approach to software like the Common Gateway Interface standard for linking databases to dynamic Web pages may be harder to use than with Apache, but it runs faster. "If we can eke out 10% to 20% better performance, we can save a lot of power, AC, and people," DiBona says.
Google built its own CRM system to support its business of selling Internet ads billed by a mixture of bid price and click-throughs. But Google isn't dogmatic about building its own tools. For accounting, it uses Oracle Financials.
Sometimes, value can be bought off the shelf, Merrill says, holding up a plastic fork as an example. But there are times when off-the-shelf software won't do. "Our culture is pretty deeply embedded in many of our processes," he says. "So what we don't want to do is buy a tool, which by extension changes the cultural aspects of the way we do things."
Keep It Interesting
Google doesn't disclose how much it spends on IT. Susquehanna Financial Group estimates Google invested about $300 million in IT in the first half of this year, or about 30% of the company's overall capital expenditures during that period, according to analyst Marianne Wolk. For the past few years, she says, Google was spending roughly 50% of capital expenditures on IT, but that percentage has diminished as the company increases spending in other areas, including land, as it expands.
Google's unorthodox approach to managing its Ph.D.s drove its decision not to budget research and development separately, as most tech companies do. "You end up in many companies with this divide between research and engineering," explains Alan Eustace, senior VP of engineering and research. By dividing those budgets, he says, "you're pretty much guaranteeing institutionally that you won't be solving interesting problems."
IT management at Google is decentralized. The company has neither a CIO nor CTO, but it's brimming with senior-level engineers and other technologists. They include Bill Coughran, VP of engineering for systems infrastructure, who oversees the distributing computing programs that power Google's online applications, and Eustace, who's responsible for product R&D. Sergey Brin isn't just Google's co-founder--his day-to-day job is president of technology. Merrill, brought in as senior director of IS three years ago, now is responsible for internal engineering and worldwide support.
Google employs a matrix management system where managers have many direct reports, and engineers report to multiple people. Engineers get most of their direction and critiques from the project leaders with whom they work. Since engineers can change projects every three months, Google eschews traditional approaches to project management and performance appraisals. Like other things at Google, artificial intelligence and computer automation perform some of the grunt work. "Our goal is to automate as many things as we can because it makes unfun things not happen," Merrill says. "Nobody wants to have a boring job, right?"
A tracking system automatically pulls information on job applicants, gives a hiring manager a job candidate's resumé, offers questions to ask, and sends the manager an e-mail after the interview asking what he or she thought of the candidate. Job interviews can involve logic questions, writing code, talking about software architecture, and generally proving to Google's brain trust that the applicant is a fast learner, since the company doesn't keep people working on the same problems for very long.
Lots of small, short-lived projects mean traditional project management software based on task lists isn't right for Google. For one thing, techies aren't very good at cataloging how they spend their hours. What they are good at, it turns out, is writing up a few short sentences or snippets about what they do each day. Those get compiled in a database along with periodic updates from project leaders about a team's deliverables. The project system tags the input by topic and routes to the appropriate people. "This is not hard AI," Merrill says. Still, who else manages workers like this?
Performance reviews are handled in a similarly technocratic way. Google's "Perf" system lets managers write e-mails--again read by a computer before any human--describing what a worker did on a project that was good or bad. Come review time, peers get an e-mail asking to compare the employee to other Google people. Perf breaks up the answers, measures who's being compared with whom, and--get this--makes the answers public. The way Merrill figures it, techies like open-air back patting. Presumably, the process airs some dirty laundry, too, but Merrill says that would happen anyway. "We have to protect our culture as we're growing fast," he says. "That's what keeps us up nights."
Google's approach isn't without its detractors. One marketing person who came to Google in 2004 as part of an acquisition grew frustrated by lack of resources and support, and quit. "I think Google is a great place to be from an IT engineer standpoint," the former employee says. "I don't know that it's quite as good as people think from the business and marketing side."
The company has a ways to go before its marketing savvy matches its engineering. Apart from its search engine and advertising system, Google's wide variety of online applications have seen only modest adoption (see story, "In Depth: Google Aims At Microsoft Office's Weak Spot With Desktop Suite"). Its Gmail service has yet to seriously challenge long-established free e-mail services from Yahoo and Microsoft. The same is true for its online financial portal, Google Finance. And Google Maps remains a distant third to MapQuest and Yahoo Maps.
Culture Of Choice
Google employees use Linux, Mac OS, and Windows on desktop computers, depending on their needs and desires. Many use homegrown programs such as Google Desktop, Google Earth, the acquired Writely word processor, and the recently launched Google Spreadsheets. In general, if an employee wants certain software, he or she can request it through the company intranet without jumping through a lot of hoops for approval.
Merrill is evasive when asked what kinds of commercial PC software are used at Google. "More important than what we put on each desktop is how we think about what to put on each desktop," he says obliquely. "Goo-gle's philosophy is that choice is always better than control. Tightly centralized control gets in the way of innovation."
He then takes a jab at CIOs--which he describes as a title used by "old-world companies"--at other companies. "Most people in my job try to control. 'Here are the three things you can buy.'" Merrill explains. "I try to control as a little as I possibly can but make it easy to work within parameters that I know how to work with."
Merrill sees a distinction between tools that tell you something and tools that stop you from doing something. For example, he observes that some financial services institutions block instant messaging because of they way they interpret regulations. "We don't think that's the right approach here," he says.
The right approach, as Merrill sees it: Talk a lot; use data, not intuition; automate wherever you can.
Collective Insight
VP of engineering Adam Bosworth last year wrote that Google's success in making a more relevant search was based on "leveraging the wisdom of crowds," referring to the company's PageRank algorithm. (James Surowiecki's book, The Wisdom Of The Crowds, was published in 2004 by Random House.) Company founders Larry Page and Sergey Brin built the business on PageRank, which analyzes the human-generated link structure of the Web to determine the relative importance of a Web page. As PageRank sees it, the more people link to a given page, the more important that page is likely to be.
This turns out to be the perfect division of labor between man and machine: Evaluating content is easy for people, and analyzing large data sets is easy for computers. By marrying collective intelligence with automation, Page and Brin built a company fueled by artificial intelligence. "AI is a great tool for helping people make better decisions," Merrill says. "It's not so good at making complex decisions."
The wisdom of the crowd, farmed and refined by machine, remains critical to Google. As Merrill puts it, "All of us together are smarter than any of us individually." That insight may not be as surprising now that it has been reinforced by the likes of Wikipedia and Digg.com, but it's still mostly lip service at many other companies.
At Google, however, examples abound, such as the way the company decides on new employees. "No one can hire anyone here," Merrill insists. "Hiring decisions are made by public groups. We all hire everyone."
That faith in group intelligence manifests itself in the lunch line. Google provides free meals to employees partly as a perk and to enhance productivity, but also to encourage interaction. It's about the pollination of ideas over salads and sandwiches. "If you want people to talk, if you want people to engage, how do you do that?" Merrill asks. "You give them lunch."
Google has an expression to describe this open discourse: Live out loud. "Everything that's done privately is done publicly here," he says. (As if to make the point, Merrill took off his T-shirt during our photo shoot, showing off his tattoos.) "We make decisions in public. We expect people to debate. You're supposed to engage. You're supposed to disagree."
TAKE OUR POLL
Google's A Do-It-Yourselfer.
Should You Be Too?

There are, of course, limits. Merrill concedes that some things need to remain private. "Customer data privacy is obviously critical to us, so that stuff is protected a lot," he says. "But our belief is that some of the things that are private in normal businesses aren't really private."
Normal businesses? That would be 99% of other companies. The challenge for Google is to remain different--which is part of its competitive advantage--while staying true to its mission to organize the world's information and make it universally, rather than selectively, accessible and useful. Because you can be sure that other companies are watching and learning from Google's every move.
-- with Aaron Ricadela and Charles Babcock

Google's Brew Of Open Source And Custom Code
Special tools are needed to manage the company's highly customized IT infrastructure.
By Charles Babcock, InformationWeek
Aug. 28, 2006
URL: http://www.informationweek.com/story/showArticle.jhtml?articleID=192300293
Reliance on open source code runs deep among Google's developers, and the company uses open source in its production systems, too. The software has been Google-ized--something new added to make it fit Google's way of doing things.
A majority of Google's engineering desktops are Linux machines. They're typically loaded with tools such as the Free Software Foundation's Gnu C Compiler; Make, a Unix utility for assembling files into a C program; and Apache Ant, open source code for assembling Java applications from Java files.
In that sense, Google looks "very much like a typical IT shop in a lot of ways," says Chris DiBona, open source program manager.
But when it comes to the Apache Web Server, one of the most common pieces of open source code, Google eschews the standard issue. "We use very little Apache around here," DiBona says. Google's internally developed Web server has been tuned to provide monitoring and health information for system reliability.
Google developers use a homegrown system to generate a "build" out of complex software projects; the system is capable of integrating code in different languages from distributed developers. "We have a very large code base, and there are often a lot of dependencies," DiBona says. The system can heed those dependencies and enforces use of previously established, tested code when it fits files together.
Google talks up its ties to open source, but the company's IT infrastructure is so customized, such as its "cell" data storage system, that specialized tools are needed to manage the code that taps such resources.
Ajax, which powers Google Maps and Gmail, is one area in which Google has given some of its own development expertise back to the open source community. Google contributed $3 million to support interns for 600 open source projects this summer and last.

Google Goes Its Own Way In The Data Center
With patents on cooling baffles and fan mounts, its next big step is hydropowered electricity.
By Thomas Claburn, InformationWeek
Aug. 28, 2006
URL: http://www.informationweek.com/story/showArticle.jhtml?articleID=192300294
How many servers does Google have in how many data centers? Google's mum on that. But counting servers reveals as much about Google's infrastructure as tallying teeth tells you about a bear; better to worry about what the bear will eat next.
VP of engineering Douglas Merrill acknowledges Google uses a standard server setup--what he calls an "office in a box"--for provisioning IT services as it expands. It's possible, even likely, that Google has adopted a common design for data centers, too.
George Mason University professor Paul Strassmann suggested in a lecture last December that Google's Linux-based infrastructure is considerably cheaper to buy and maintain than a comparable setup of Sun Microsystems servers or Windows servers would cost. For IT shops that spend half their budgets just keeping machines up and patched, the implications are significant. Strassmann said IT pros know where they need to go--toward a Google-style architecture.

Organizational growth has been the No. 1 issue, Holzle says.

Some of the new servers going into Google's data centers are probably equipped with AMD Opteron multicore processors. Google won't confirm that, but one of the reasons AMD chips are selling so well to other companies is that they don't throw off as much heat as older alternatives. Google engineers, who pay close attention to microprocessor efficiency and heat dissipation, must find AMD chips hard to ignore. Intel is racing to improve the performance-per-watt of its own processor line.
Google has tackled some of the tough issues of data center management with internally developed technology. The company has patented homegrown designs for a better cooling baffle and fan mount for its rack servers. The U.S. Patent and Trademark Office lists 23 patents granted to Google, with 14 patent applications in the pipeline. That doesn't include patents obtained through acquisitions or filed outside the United States.
Though Google doesn't build its own fans or power supplies, it cares about such components. The company requests specific high-efficiency designs from manufacturing partners, says Urs Holzle, senior VP of operations.
Google's newest data center, under construction in The Dalles, Ore., has become a focus of media attention. An aerial photo of the data center appeared on the front page of The New York Times on June 14. Why Oregon? "We're always looking for candidate sites to host our infrastructure--selecting a site always means balancing a number of factors, including the availability of land and power, as well as a suitable workforce," Holzle explains in an e-mail interview. "The Dalles was one of the sites we found that met our needs, plus it's a beautiful area to live."
The availability of cheap power--widely cited as the main reason Google is building on a river in Oregon--isn't an issue for most companies. Edward Koplin, a principal in the Baltimore engineering firm of Jack Dale Associates, says that among his firm's clients, which include the Army Corps of Engineers, Citigroup, and Wells Fargo, "not one of them brings up as an issue, 'How cheap can my power be?'" But then, their data centers aren't growing the way Google's are.
Google declines to discuss how much it spends on electricity, but its financial documents note that data center costs have been rising. In a June filing, Google attributes an increase in the cost of revenues to, among other things, increasing data center costs, which include depreciation, labor, energy, and bandwidth costs.
Yet, even more than the availability of power and local subsidies, beauty may explain Google's decision to locate in Oregon. Rob Enderle, principal analyst at the Enderle Group, suggests that locating in an affordable, attractive, relatively undeveloped community represents an established strategy in high tech to attract and retain talent. "That was the old Microsoft model," he explains, noting that Redmond, Wash., was a nice place to live without much industry at the time Microsoft got started. "So once you got there it wasn't like you were going to go anyplace else. And they were able to keep turnover to an absolute minimum as a result."
Electric power may be expensive, but it's cheap compared to brainpower. "For as long as I can remember, organizational growth has been the No. 1 issue," says Holzle. "If you need to grow quickly but don't want to fall apart as a company, you need to focus a lot more on hiring, training, and nurturing the right culture."

Profile: Google Technologist Knows Problem Solving Firsthand
Engineering VP overcame his own disabilities in determining which challenges matter most.
By Thomas Claburn, InformationWeek
Aug. 28, 2006
URL: http://www.informationweek.com/story/showArticle.jhtml?articleID=192300295
I'm a technology management type. I have pointy hair. No, I am not the pointy-haired boss." That's how Douglas Merrill describes himself in his blog, The Other End of Sunset. Google's VP of engineering, in keeping with the company culture, lives out loud.

Douglas Merrill, VP of engineering, Google

Merrill, 36, graduated from the University of Tulsa, majoring in social and political organization, then earned a master's and a doctorate from Princeton in psychology. He worked for the Rand Corp. as an information scientist; taught information security in Southeast Asia; joined Price Waterhouse, where he became leader of the West Coast security practice; then moved to Charles Schwab as senior VP of information security. In 2003, he was hired by Google.
Google doesn't have a CIO. Merrill, with responsibility for Google's internal IT systems, comes closest. He overcame several challenges to get on that track. He was deaf from 3 years old till 6, the result of an infection in his auditory nerve. He sometimes apologizes for his accent, vaguely Southern and Canadian, a consequence of being raised in Arkansas, yet having a Canadian voice coach. He's also dyslexic; reading and math remain difficult.
Merrill's hacking roots go back to Arkansas, where he disabled an online bulletin board used by white supremacists. "One of the things I found that really interested me is it turns out that it's not actually all that hard to crash those bulletin boards and make them unavailable," he says.
An interest in how technology works and how people use it followed. Merrill's premise: There are no lasting technical solutions to social problems, and most interesting problems are social problems. "The particular tools and systems we give [people] yield certain kinds of problems," he says. Merrill sees it as his job to help solve them.

加利福尼亚州山景城(Mountain View)谷歌公司(Google，下称谷歌)总部有一座43号大楼，建筑的中央大屏幕上显示着一个与谷歌地球(Google Earth)相仿的世界地图，一个转动的地球上不停地闪动着五颜六色的光点，恍如罗马宫廷的千万烛灯，每一次闪动标志着地球的这个角落一名谷歌用户发起了一次新的搜索。这同时意味着谷歌又一次满足了人们对未知信息的好奇与渴望。

　　谷歌是与众不同的。它的独特不仅仅表现于革新的思维和充满创意的应用(比如那个大堂里的地球模型)，更在于其有别常规的IT策略。从人们的常理来看，简单的硬件商品和免费软件是无法构建出一个帝国的，但是谷歌做到了。在性能调整后，谷歌把它们变成一个无可比拟的分布式计算平台，该平台能够支持大规模的搜索和不断涌现的新兴应用。我们原本认为这些应用都是个人消费级别的，但是谷歌改变了这一切。现在商业世界也在使用它们，这就令这家搜索公司显得那么与众不同。

　　谷歌Web服务背后的IT架构对无数使用搜索引擎的用户来说也许并不是非常重要，但它是谷歌几百位致力于把全球信息组织起来，实现“随处可达，随时可用”目标的工程师们的最核心工作。这就需要一个在覆盖范围和野心上都与谷歌的商业愿景完全相符的IT蓝图作为支撑。

　　谷歌的经理们一直对公司的IT策略话题保持沉默，他们厌恶谈及特定的厂商或者产品，当被问到他们的服务器和数据中心时，他们总是闭口不谈。但与几位谷歌的IT领导一起呆了一天后，我们最终得以揭示该公司的IT是如何运作的，那可不仅仅是一个运行在无数服务器集群上的、表面看来非常简单的搜索引擎。在其简单的外表下，蕴涵着许多内部研发软件、定制硬件、人工智能，以及对性能的执着追求和打破常规的人力管理模式。

　　IT理念方面，谷歌对同行有一条建议：尽量避免那些人人都在使用的系统和软件，以自己的方式做事会更有独特的竞争优势。

　　“企业文化决定了你的做事方式。”道格拉斯•美林(Douglas Merrill)，这位谷歌工程副总裁和事实上的首席信息官(CIO) 指出，“到了我们这样的发展阶段，企业观念和文化非常与众不同，这也反过来鞭策我们必须要采用与众不同的方式来运行那些他人看来很常规的系统。”

　　谷歌最大的IT优势在于它能建造出既富于性价比(并非廉价)又能承受极高负载的高性能系统。因此IT顾问史蒂芬•阿诺德(Stephen Arnold)指出，谷歌与竞争对手，如亚马逊网站(Amazon)、电子港湾公司(eBay)、微软公司(Microsoft，下称微软)和雅虎公司 (Yahoo，下称雅虎)等公司相比，具有更大的成本优势。谷歌程序员的效率比其他Web公司同行们高出50%～100%，原因是谷歌已经开发出了一整套专用于支持大规模并行系统编程的定制软件库。据他估算，其他竞争公司可能要花上四倍的时间才能获得同等的效果。

　　打造服务器

　　谷歌究竟是怎样做到这点的呢？其中一个手段，美林认为，“是因为我们自己动手打造硬件。”谷歌并不制造计算机系统，但它根据自己的参数定制硬件，然后像MTV的节目“靓车打造”(Pimp My Ride)那样自己安装和调整硬件系统。开源程序经理克里斯•迪博纳(Chris DiBona)评论道：“我们很善于购买商业服务器，并且改造他们为我们所用，最后把性能压榨和发挥到极致，以致有时候他们热得像要融化了似的。”

　　这种亲手打造的方式，来源于谷歌从车库诞生时与生俱来的节俭风格，更与谷歌那超大型的系统规模息息相关，良好的习惯一直延续至今。据说谷歌在65个数据中心拥有20万～45万台服务器—这个数目会有偏差(取决于你如何定义服务器和由谁来做这项统计)。但是，不变的是持续上升的趋势。

　　谷歌不会去讨论这些资产，因为它认为保密也是一种竞争优势。事实上，谷歌之所以喜欢开源软件也是因为它的私密性。“如果我们购买了软件许可或代码许可，人们只要对号入座，就可以猜出谷歌的IT基础架构。”迪博纳分析说，“使用开源软件，就使我们多了一条把握自己命运的途径。”

　　谷歌喜欢规模化的服务器运行方式。当有成百上千台机器时，定制服务器的优势也会成倍增加，效果也会更趋明显。谷歌正在俄勒冈州哥伦比亚河边的达勒斯市建造一个占地30亩的数据中心，在那儿它可以获得运算和降温需要的低价水力电力能源(参见边栏《谷歌数据中心自有一套》)。

　　谷歌以“单元”(Cell)的形式组织这些运行Linux操作系统的服务器，迪博纳把这种形式比喻成互联网服务的“磁盘驱动器”(但别和一直谣传的谷歌存储服务Gdrive混淆了，“并没有Gdrive这回事。”一位谷歌女发言人明确表示。)，公司的软件程序都驻扎在这些并不昂贵的电脑机箱里，由程序员决定它们的冗余工作量。这种由很多单元组成的文件系统代替了商业存储设备；迪博纳表示谷歌这些单元设备更易于建造和维护，他还暗示他们能处理更大规模的数据。

　　谷歌不会漏过对任何技术细节的关注。多年来，公司的工程师就在研究微处理器的内部工作机制，随着谷歌规模的持续壮大，必然会用到特别定制和调节过的芯片。知名工程师路易斯•巴罗索(Luiz Barroso)去年在一篇发表在工业杂志上的论文中证实，近年来谷歌的主要负荷都由单核设计的系统承担着。但许多服务器端的应用，如谷歌搜索索引服务，所需的并行计算在单核芯片的指令级别上执行得并不好。

　　曾在数据设备公司(Digital Equipment)和康柏公司(Compaq)当过芯片设计师的巴罗索认为，随着AMD公司、英特尔公司(Intel)、太阳计算机系统公司(Sun)开始制造多核芯片，必将会出现越来越多芯片级别的并行计算。

　　谷歌也曾考虑过自己制造计算机芯片，但从业界潮流来看，这个冒险的举动似乎不是很必要。“微处理器的设计非常复杂而且成本昂贵，”运营高级副总裁乌尔斯•霍尔茨勒(Urs Holzle)表示。谷歌宁愿与芯片制造商合作，让他们去理解自己的应用并设计适合的芯片。这是一种客户建议式的设计，其关注点在于总体吞吐量、效能，以及耗电比，而不是看单线程的峰值性能。霍尔茨勒表示，“这也是最近多核CPU的设计潮流与未来方向。”

　　裁缝般地定制软件

　　为了能尽量压榨硬件性能，谷歌开发了相当数量的定制软件。创新产品主要包括用于简化处理和创建大规模数据集的编程模型MapReduce；用于存储和管理大规模数据的系统BigTable；分析分布式运算环境中大规模数据集的解释编程语言Sawzall；用于数据密集型应用的分布式文件系统的“谷歌文件系统”(Google File System)；还有为处理分布式系统队列分组和任务调度的“谷歌工作队列”(Google Workqueue)。

　　正是从Sawzall这些工具里体现出谷歌对计算效率的执著关注。并不是每家公司都能从底层去解决效率问题，但是对谷歌来说，为常规关系型数据库无法容纳的大规模数据集专门设计一种编程语言是完全合理的。即使其他编程工具可以解决问题，谷歌的工程师们仍然会为了追求效率而另外开发一套定制方案。谷歌工程师认为，Sawzall能与C++中的MapReduce相媲美，而且它更容易编写一些。

　　谷歌对效率的关注使它不可能对标准Linux内核感到满意；谷歌会根据自己的需要运行修改过的内核版本。通过调整Linux的底层性能，谷歌工程师们在提高了整体系统可靠性的基础上，还一并解决了数据损坏和数据瓶颈等一系列棘手问题。对内核的修改也使谷歌的计算机集群系统因为通信效率的提高而运行得更快。

　　当然，谷歌偶尔也会出现系统故障，情况一旦发生，无数的用户就会受到影响了。三年前一次持续30分钟的系统故障使20%的搜索流量受到影响。

　　谷歌开发了自己的网站服务器却没有使用开源的Apache服务器，尽管它在网站服务器的市场占有率超过60%。迪博纳认为，谷歌的网站服务器可以运行在更多数量的主机上，对谷歌站点上内容庞大又彼此互相依赖的应用程序来说，这种服务器的负载均衡能力远比Apache的能力更高。同时，在用标准公共网关接口(CGI)访问数据库动态网页方面，谷歌服务器的编程难度要比Apache更高，但是最终运行速度却更快。“如果我们能够压榨出10%～20%的性能，我们就可以节省出更多系统资源、电量和人力了。”迪博纳在总结中指出。

　　谷歌还设计了自己的客户关系管理(CRM)系统用于支持自己基于竞价和点击的互联网广告收费业务。但对是否需要设计自己的工具，谷歌的态度也不是一成不变的。比如在财会软件上，它就使用了甲骨文公司(Oracle)的Financials软件。

　　美林拿着一只叉子举例说明现成的产品也可以带来价值。但在有些场合现成的软件产品就不一定适用了。“我们的文化在各个层面对我们的运作都有深远影响，”他表示，“所以我们不想让购买所得的工具改变我们的工作方式和文化层面。”

　　保持活力

　　谷歌没有透露它在IT上的开销。Susquehanna 金融集团(Susquehanna Financial Group)分析师玛丽安•沃尔克(Marianne Wolk)估计今年上半年谷歌在IT上的花费高达三亿美元，相当于在此期间谷歌全部收入的30%。而在过去几年间，谷歌把全部收入的50%花在了IT上。现在，随着公司的发展，IT开销比例正在下降，其他领域如房地产业的投入正在增加。

　　非正统的科研人员管理方式使谷歌没法和大多数科技公司一样把研究和开发的预算分离开来。高级工程和研发副总裁阿兰•尤斯塔斯(Alan Eustace)解释说：“在许多大公司你会发现研究和工程是分离的。”但是把这两部分预算分离开来，他认为“就等于从制度上遏制了把各项工作变得更加有趣的可能。”

　　谷歌的IT是“去中心化”(Decentralized)的管理方式。公司没有 CIO或首席技术官(CTO)，但它拥有一大群高级工程师和技术人员。这中间包括主管系统基础设施工程的副总裁比尔•库格伦(Bill Coughran)，他负责掌管和支撑谷歌在线应用的大型分布式运算程序，而尤斯塔斯则负责产品的研发。谢尔盖•布林(Sergey Brin)的身份并不仅是谷歌的创始人之一，他的日常工作是技术总裁。美林在三年前作为信息系统高级主管加入谷歌，现在负责内部工程部门的管理和全球技术支持。

　　谷歌采用的是矩阵式管理系统，每位经理都有好几位直接汇报的上司，每位工程师也可以同时向几个人汇报。多数情况下，工程师们从共同工作的项目领导人处获得指导和评价。但每隔三个月工程师们就可以自发调换项目，因此谷歌改变了传统意义上的项目运作和绩效考评模式。如同处理其他技术问题一样，谷歌用人工智能和计算机自动化的方式来处理一些日常工作。“我们的目标是尽量自主运作，以免工作变得乏味。”美林表示，“没有人会喜欢乏味的工作，对吗？”

　　此外，谷歌的招聘跟踪系统能自动聚合应聘者的各种信息，为招聘经理提供候选人的工作简历，建议的面试问题，在招聘之后还会自动发送电子邮件询问招聘方对应聘人选的意见。工作面试包括逻辑问题、编写代码、讨论软件架构，和其他任何可以使谷歌相信应聘者具有快速学习新技术能力的证明，因为谷歌是不会让它的员工长时间只干一件事情的。

　　许多小型和短期的项目意味着基于任务表的传统项目管理方式的软件不适用于谷歌。比如，他们的技术人员都不擅于总结自己的时间分配计划。他们更擅长的是写下每日工作相关的简短介绍或是代码片断。这些内容会定期和小组领导提交的项目更新信息一起汇编到数据库里，项目管理系统根据输入内容的主题，加上标签把它们转发给相关人员。美林认为，“这样的系统还算不上真正的人工智能。”但是，除了谷歌还有哪家公司会这样管理员工呢？

　　绩效考核也采用了类似的技术机制。谷歌的“Perf绩效”系统让经理们写电子邮件 (当然还是先由电脑而非人工进行分析)，邮件描述了员工在项目中表现的优劣。同事们则会收到另一封来自系统的电子邮件要求他们对比该员工与其他谷歌员工。 Perf系统自动分析所有反馈信息，对员工对比结果进行评估，并把结果公开。这正是美林想要的效果，技术让工作关系变得更加透明和人性化。可以预见，虽然打开窗户的同时也会放进苍蝇，但美林表示凡事总有缺憾。“飞速发展的同时，还必须保护我们的企业文化，”他说，“这能让我们保持清醒。”

　　对谷歌的方式，有人持不同意见。2004年因兼并加入谷歌市场部门的某工作人员，就由于缺乏资源和支持而辞职。“从IT工程师角度看，谷歌的确是个了不起的地方，”这位前雇员说，“但是，从商业或市场角度看，它就未见得如人们想象的那么出色。”

　　谷歌对市场的理解力，还远跟不上它的技术实力。除了搜索引擎和广告系统，谷歌大量的在线应用项目目前的作用还很有限。谷歌的Gmail服务面临着长期霸主雅虎和微软免费电子邮件服务强有力的挑战。它的在线金融门户Google Finance系统也存在类似的问题。谷歌地图也远远落后于MapQuest和雅虎地图，位居第三。

　　甄选企业文化

　　谷歌员工可以按照自己的需要和兴趣选择PC的操作系统，包括Linux、Mac OS和Windows。很多人使用公司自产的程序，如谷歌桌面(Google Desktop)、谷歌地图、购并的在线Writely 文字处理软件，以及最近研发的谷歌电子数据表(Google Spreadsheets)。在一般情况下，如果公司的某位员工需要某种特定软件，他不用费太多周折就可以通过公司内部网提出申请而获准使用。

　　当被问及谷歌在使用哪些商业PC软件时，美林闪烁其辞。“重要的不是我们在每台桌面电脑上安装了什么程序，而是我们对安装了什么电脑程序所持的态度，”他含糊地表示，“谷歌的原则是自由选择永远优于简单控制。严格的集中化管理会妨碍创新。”

　　接着，美林对其他公司的CIO制度略有非议，他认为这是老式公司才用的头衔。“大多数在我这个位置的人都在勉强控制局面，他们会说‘你只能购买这三样产品’。”美林解释说，“我会尽可能缩小控制范围，试着简化工作，把相关制约因素限制在我比较有把握的范围之内。

　　在告诉别人做什么和阻止别人做什么之间，美林很清楚两者的区别。例如，他注意到有些金融服务机构禁止员工使用即时通信(IM)软件，因为他们就是这样诠释自己的规章制度。美林指出“这不是正确的方法”。

　　在美林的观点中，正确的做法是：允许充分讨论，畅用数据，拒绝直觉，尽量自动化。

　　集思广益

　　去年工程副总裁亚当•伯斯沃斯(Adam Bosworth)在提到谷歌的文档排名算法(PageRank)时说，谷歌的搜索结果比别人更准确，是因为其得益于“大众智慧的力量”。[引自詹姆士• 苏罗维奇(James Surowiechi)所著《大众的智慧》，2004年由兰登书屋(Random House)出版。]公司创始人拉里•佩奇(Larry Page)和谢尔盖•布林(Sergey Brin)正是从PageRank开创了事业，PageRank通过分析网页上人工生成的链接结构来决定网页的相关重要性。对PageRank来说，越多人链接到一个特定网页，这个网页很可能越重要。

　　这样，人力和机器得到了完美分工：人类擅于内容评估，计算机则可轻松分析大规模数据集。通过嫁接集体的智慧与自动化处理，佩奇和布林建立了一家由人工智能驱动的公司。“人工智能是个了不起的工具，它可以帮助人们更好地做决定，”美林说，“但是它不适合做复杂的决定。”

　　通过机器收集和提炼的集体智慧对谷歌非常重要。就像美林指出的那样，“集体比个体更聪明”。“这个见解也许已不像当初那样令人耳目一新，因为维基百科(Wikipedia)和Digg.com已一再证明了这点。但是对于很多其他同类公司来说，对于这个观点的理解仍然有限。”美林解释说。

　　但在谷歌，相关的例子比比皆是，比如公司选择新员工的方式。“这里没有人有权聘用任何人，”美林坚称，“聘用由公开的集体决定。是我们所有人聘用了所有其他人。”

　　对集体智慧的坚持也体现在谷歌的午餐文化。谷歌为雇员提供免费午餐作为激励和提高效率的方式，但同时也是为了鼓励员工之间的互动。这是杯盘交错之间的交流。“如果你希望人们说话，希望人们投入，你会怎样做？”美林反问道，“你得给他们提供午餐。”

　　关于企业内部公开讨论的方式，谷歌有一条口号：畅所欲言。“在其他地方私下讨论的东西在这里变得公开了。”美林说。(仿佛为了说明这一点，美林在我们拍照时还脱掉T恤展示他的刺青。)“我们的决议都是公开讨论的结果。我们欢迎辩论。我们期待每个人的参与，渴望不同的观点。”美林这样说道。

　　当然，凡事都有底线。美林也同意有些事情仍然需要保密。“用户数据的私密性对我们显然非常重要，所以我们对这类数据资料会严加看管。”他表示，“但是某些常规商业领域里的秘密我们反而认为并非真的那么需要保密。”常规商业领域？那就包括了99%的其他公司。

　　谷歌目前面临的挑战就是保持其独特性—这恰恰也正是它的核心竞争力—同时它也坚持承担了组织全球信息的责任，使之随时随地可以被访问和利用，不受任何限制。可以想象，其他公司会密切关注和学习这家特立独行的企业的一举一动。