kitchen table math, the sequel: Google speaks...

Sunday, February 23, 2014

Google speaks...

...through Tom Friedman, no less:
The least important attribute they look for is “expertise.” Said Bock: “If you take somebody who has high cognitive ability, is innately curious, willing to learn and has emergent leadership skills, and you hire them as an H.R. person or finance person, and they have no content knowledge, and you compare them with someone who’s been doing just one thing and is a world expert, the expert will go: ‘I’ve seen this 100 times before; here’s what you do.’ ” Most of the time the nonexpert will come up with the same answer, added Bock, “because most of the time it’s not that hard.” Sure, once in a while they will mess it up, he said, but once in a while they’ll also come up with an answer that is totally new. And there is huge value in that.

How to Get a Job at Google
FEB. 22, 2014 | NY Times
I am praying our superintendent doesn't see this.

UPDATE: Daniel Willingham on the actual research. Thank heavens we have him.


Catherine Johnson said...

The superintendent is hosting a book club for parents. The book he's chosen is Paul Tough's "How Children Succeed," which argues, in the first pages that what matters is character, not knowledge.

This is the kind of thing that majorly gets my goat. The only reason to pay him what we're paying him (total compensation over 300K) is to ensure that Irvington kids acquire knowledge.

Character, they get from their families.

SteveH said...

"... what matters is character, not knowledge".

And Personnel gives applicants silly tests that supposedly can tell. Perhaps they can weed out the obvious ones, but not those who will become real problems. Once they get hired, it's very difficult to get rid of them.

As for skills, job postings are very specific. Rarely do companies hire people based on just potential, especially once they have been working a few years and have specialized their skills. Why pay them a salary based on different skills.

"... and they have no content knowledge, and you compare them with someone who’s been doing just one thing and is a world expert, the expert will go: ‘I’ve seen this 100 times before; here’s what you do.’ ” Most of the time the nonexpert will come up with the same answer, added Bock, “because most of the time it’s not that hard.”"

Yes, being a "world expert" is just not that hard.


SteveH said...

"Laszlo Bock, the senior vice president of people operations for Google .."

What do you expect from a person without an understanding of what it takes to master technical knowledge and skills?

"For every job, though, the No. 1 thing we look for is general cognitive ability, and it’s not I.Q. It’s learning ability. It’s the ability to process on the fly. It’s the ability to pull together disparate bits of information. We assess that using structured behavioral interviews that we validate to make sure they’re predictive.”"

Ha, ha, ha, ha, ha. "Structured behavioral interviews". It's a Catch 22. He has to sell this crap to management as if it's some sort of deep new thinking or expertise, but then again, experience and expertise don't matter. Right?

"The second, he added, “is leadership — in particular emergent leadership as opposed to traditional leadership. ... What we care about is, when faced with a problem and you’re a member of a team, do you, at the appropriate time, step in and lead. And just as critically, do you step back and stop leading, do you let someone else? "

And of course their psychobabble tests can detect this. They lay out exactly what Google looks for, and lo and behold, they will be amazed at how many more of these people apply there. It couldn't be that they now know what to tell them.

"What else? Humility and ownership. “It’s feeling the sense of responsibility, the sense of ownership, to step in,” he said, to try to solve any problem — and the humility to step back and embrace the better ideas of others."

Right. They want those who are passionate about their ideas, but are willing to let others take all of the credit. Passionate workaholics who don't want to get ahead, or at least, get their name on the patent.

"The least important attribute they look for is “expertise.” Said Bock"

Liar, liar, pants on fire.

Read "The Circle" by Dave Eggers. It's not subtle enough for my taste, but then again, Lazlo Boch is not very subtle and many people buy that crap.

Then again, this is what Lazlo said last year in the NYT.

"Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess, except for one guy who was highly predictive because he only interviewed people for a very specialized area, where he happened to be the world’s leading expert."

I'll repeat that:

"the world's leading expert".

He goes on to say that:

"Instead, what works well are structured behavioral interviews, where you have a consistent rubric for how you assess people, rather than having each interviewer just make stuff up."

Gee, you have to claim that something works right after admitting that personnel interviews have shown no predictive ability. However, instead of going with what works - using the content experts - they have to find some psycho mumbo jumbo that personnel can claim as their own area of content expertise.

I'm particularly annoyed at these gatekeepers because my son is going through just this sort of process to get into college. Rather than bring in professorial content experts to judge applicants, admissions' officers claim this expertise for themselves. Holistic. It's called job security.

Glen said...

What I don't want to hear from my airline pilot or surgeon: "I may not have any traditional content expertise, but I'm bright, innately curious, and have leadership potential."

Anonymous said...

I've interviewed at Google, and I know lots of people who have worked there. Trust me, they want expertise. WIthout deep expertise, you won't get through the phone screen.

SteveH said...

"The book he's chosen is Paul Tough's "How Children Succeed," which argues, in the first pages that what matters is character, not knowledge."

Tough said in The Washington Post in 2012:

"The skills I’m talking about include grit, curiosity, perseverance, conscientiousness, self-regulation, and optimism. I do think they can be taught in the classroom – I think most of us can think of a teacher in our past who helped us develop one or more of those skills – but I don’t think we yet have an ideal model for exactly how to teach them in the classroom."

I'm sure educators will come up with ways that are only top-down driven. Engagement, apparently, drives everything. If skills are not mastered, at least they have the grit to live up to their cognitive potential. Right? After not learning fractions and basic algebra, their grit will fix all of their very bad curricula and pedagogy?

The teachers I most remember are the ones who set high expectations in terms of mastery of content. (I remember the 40 page calculation assignments.) They are the ones who pushed and got me to realize that I can figure it out. Modern educational thought, however, is only focused on engagement. I call it eating mathematical Twinkies. Exciting, but of no lasting value. One does not magically find "grit" and are then able to master difficult content.

SteveH said...

Willingham uses the argument:

"Look, we could fill the freshman class with students who got 800,800 on the SAT. Literally. Every single freshman, 800,800. We're just not interested in doing that."

"Harvard admissions officers (at least as represented by my friend) were also quite serious in how they tried to do it, and quite humble about their ability to assess them."

I have heard that Harvard does bring in the perspective of professors, but I can't verify that. We have been told (by all of the top colleges) that once you get above a certain level academically, then many other attributes become important. Unfortunately, that level is relatively low and that these colleges don't seem to make finer academic distinctions after that point. One can have 750+ on the SAT math portion, but still not be a great prospect for a STEM career. Even the SAT II for Math 2 and Physics do not provide enough granularity. Some colleges do ask for AMC/AIME scores, but I have no idea of how that information is weighted. Once you get above a certain level academically, then they seem to ignore credible academic distinctions and turn the process over to voodoo essay psycho-guessing. It would be better if they would just base it on filling the lacrosse team or building a quality orchestra.

At least many admissions people are honest in how they have to reject so many students who they know would be perfect for their school. (At MIT, one admissions person talked about "enjoying" the random walk.) The rejection could very well ignore important academic distinctions once you are above their magic cutoff. And it could be based on the random sequencing of how the applications come in or on whose desk it falls. What is the probability of being potentially a perfect student for a college, but getting semi-randomly rejected? At the top schools, this percentage is high. The percentage is still high after taking into account many colleges. If your essays are not so great, then the process is not random at all. The admissions gatekeepers are all cut from the same cloth. They might be using their own fuzzy structured behavioral or boring essay techniques.

I dislike the idea that once one has reached some vague minimum cutoff for a job, college, etc., that it's proper to ignore finer content and academic distinctions and turn the process over to those whose only content expertise consist of fuzzy concepts like "structured behavioral interviews".

Anonymous said...


Your post reminded me of something I was wondering about just this morning. For the SAT2 math level 2, last time I checked, you could leave out 8 questions and still get an 800. And an 800 got you to the 92nd percentile, so 8% of the testers were getting that 800. My question is why? Why don't they curve it more harshly and/or raise the top score? It seems that they are throwing away data that could be used to make distinctions. Isn't that what they are there to do? If you are going to throw away that data, why not give everyone a participant trophy and move on?

I assume that I am missing something here, that there is some reason why that last chunk of data can't be used. Anyone know why?


momod4 said...

I'd guess that discriminating more at the top would produce more Asians and more males in that group. There's a significant push to pretend that males and females are equally likely to score in that furthest-right group.

About 20 years ago, a friend of my son was passed over for admission to MIT, in favor of two classmates who were less mathy but more "balanced" (the mathy kid had had a couple of Bs in English/Spanish). Even compared to his mathy classmates, many of whom had 800 on SATI/SAT II and 5s on AP calc BC, he was at an entirely different level. Everyone felt that the math dept at MIT probably would rather have had him, and his intended math major, than either of his classmates (who had no interest in a math major).

SteveH said...

" 8% of the testers were getting that 800. My question is why?"

I have and had the same question.

"It seems that they are throwing away data that could be used to make distinctions."


It's one thing to ignore the distinctions, but quite another to hide the data - not that admissions people will value it. My son and his friend got 800's on the Math 2 (and on the physics test), but nobody in their classes would think they are even close. Likewise for the AP tests. A 5 covers a lot of ground.

What this means is that colleges don't have those distinctions, but they don't seem to care.

SteveH said...

MIT might claim that it's not a problem with their attempt to balance the gender ratio, but it requires this sort of "bucket" philosophy - that all students are somewhat equal once they meet a certain academic level.

However, I think that MIT makes a finer distinction for their Early Admittance process. They want the single focus national and international high flyers. My son probably made a tactical error in applying to MIT EA (deferred) because he is balanced with a national level in piano. Although his AIME score was at the 80th percentile, he never was on a math team and never prepared for the AMC/AIME. For EA, I don't think balance works at MIT. I think that changes all around when you get to regular admission. Maybe balance works better for the EA in other top schools, however.

"...he was at an entirely different level.

Were the other two accepted during EA? My view now is that the way to show that level is with the AMC/12, AIME, etc. You really need to get to the USAMO level to stand out. If you are not at that high of a level, then all bets are off.

If I had to do it all over, I would have pushed my son's preparation for the AMC test. However, I thought it was fine for him to do his own thing and accelerate using GeoGebra and Mathematica. (He took AP Calculus as a junior.) The problem is that this is almost meaningless on an application. Yes, he likes to do math at home. Ho hum.

Admissions people claim that they are holistic, but the tangible bullets that are calibrated at a regional and national level matter much more that any sort of intangible interest.

Perhaps the biggest issue is that admissions is incredibly competitive. This means that admittance shifts more from academics to the fuzzy holistic thinking of the admissions people. I also wish my son had a music "coach" pushing to get him on the team.

I can see this with personnel groups at companies, as the demand goes up, they don't bring in the content experts to help them with the final decision. Like college admissions, they might be picking one out of five or more applicants without further technical or academic input from the experts. Rather than having an initial "bucket" for academics, they need to have an initial "bucket" for balance and intangibles. Data on academics is more credible than fuzzy tests on leadership or humility.

Anonymous said...

"For the SAT2 math level 2, last time I checked, you could leave out 8 questions and still get an 800. And an 800 got you to the 92nd percentile, so 8% of the testers were getting that 800. My question is why? Why don't they curve it more harshly and/or raise the top score? It seems that they are throwing away data that could be used to make distinctions. Isn't that what they are there to do? If you are going to throw away that data, why not give everyone a participant trophy and move on?"

It is odder than that.

When the basic SAT was re-centered in 1995, the folks *could* have moved the new average from the old desired average of 500 to 400. The assumption being that most colleges don't see much of a need to distinguish students on the left side of the SAT bell-curve. Instead, they re-centered to 500, but an 800 no longer required a perfect score. Information was thrown away here, too.

And the Math 2 isn't the only subject test that scoring 800 our of 800 doesn't put you in the high 90s percentile.

An 800 on the Math 2 in 2009 puts you in the 89th percentile (so ... top 11% of those taking the test). 800 on Chemistry is the 93rd percentile. 800 on Physics is 90th percentile. 800 on listening Korean is the 66th percentile. Etc.

So it isn't just math.

And the number of kids taking these individual tests isn't always that small, either. We "only" had 4,625 kids take listening Korean, so this suggests that about 1,500 got a "perfect" 800. Maybe not worth further discriminating?

But 155,952 kids took the Math 2 test. If an 800 only meant "top 11%" then this is over 17,000 kids with an 800.

So ... yeah, clearly the College Board is fine lumping all of those 17,000 kids together for math score purposes. What *IS* interesting is that Literature, US History and World History are much more discriminating ... and 800 maps to 99, 98 and 97th percentile respectively. And an 800 on Ecological Biology also maps to the 99th percentile. So this doesn't seem to be some principled "don't distinguish the top of the curve" thing.

But I don't know what is going on, either.

-Mark Roulo

Anonymous said...

Steve, what you are noticing is a feature, not a bug. The admissions offices at the top schools are not interested in identifying the most academically capable/accomplished students in the world. They are interested in identifying students who can meet the academic demands of the institution and who are most likely to contribute to the future of the institution. The students who will keep the famous professors happy so they don't decamp to New Jersey. The students who will become hedge fund zillionaires and then donate a large number of millions of dollars. The students who will become prime ministers or world-class business leaders. The students who will get the institution in the news for the right reasons. That's what they're looking for. Not a standardized test score. They are looking after the long-term interests of the institution, not serving as the world's prize committee for being the most smartest person with the best grades and test scores.

SteveH said...


I never questioned holism. I just question the people making those decisions. But I'm also questioning the balance they use. I doubt that admissions officers get input from their "famous professors", and I don't think, as with Laszlo Bock at Google, that they know what works. They don't know who will become big donors or stars in the future. They can't predict a Yo Yo Ma. However, Bock did say that the best results they had were made by a world class content expert.

So personnel departments use fuzzy content expertise, like Structured Behavioral Interviews, and admissions officers use essays and other holism techniques ("would I like to have a cup of coffee with that person?") to capture any sort of credibility. At least Bock made an attempt at monitoring a feedback loop. I have heard vague reports that some colleges do the same thing, but admissions people are not going to willingly turn over their control to content experts. As with Bock, they will only tweak their own variables.

The problem is that with the current level of competition, holism factors are being blown way out of proportion. Admissions officers are now deciding on one-in-five, or more, "academically equal" applications. As this thread discusses, many academic distinctions are ignored or trivialized.

"...not serving as the world's prize committee for being the most smartest person with the best grades and test scores."

This is so not what I'm talking about. Besides, there is holism in academics too, but admissions officers are not able to see or understand the distinctions. Some colleges do ask for things like AMC/AIME, etc. scores, but that benefits those who have focused on those tests.

So, personnel and admissions people claim some expertise in holism, but they can't show that it has any real benefit past a basic "bucket" fit. However, many of these (non academic content) people get to choose between one-in-five, or more, applications.

The simplistic stereotype is that higher academic scores correlate with nerdiness and poor social skills. Many love the idea that those who get really good scores are academic robots - that they are not creative or risk takers. This smug and self-serving meme pervades K-12 education.

Competition is so intense that admissions officers know that they are guessing. They should turn back to their content experts for holistic advice on an academic level. The social holistic bucket should come first, and then the final holistic decisions should be made by the content experts. I'll wager that they will also be very good at the non-academic fuzzy variables.

momof4 said...

The academic side of college admissions, where the academic guys have no input, is an interesting contrast to the athletic side, where coaches (the content guys/women) make the decisions. One of my kids was a top academic recruit at a particular school, and was accordingly invited for the first on-campus weekend. My DH took him there, and watched as all of the athletic recruits were met by their prospective coaches and their weekend host/prospective teammate. Academic recruits were hosted by random students, not those in the departments in which the recruits were interested. My future-engineering kid's host was a sophomore English major, taking much less demanding classes than my kid was taking in HS (first month of senior year) - very obvious when accompanying his host to classes. His host was very welcoming but had no point of connection with the things my kid was interested in exploring and my kid came home having wiped that school off his list.

cranberry said...

At Harvard, apprarently professors have input. They look to math contests for math talent, according to this article:

In addition to his staff of 35, Fitzsimmons enlists Harvard's coaches and professors to look for talent. The math department, for instance, starts to identify budding math geniuses by keeping a close eye on kids doing well in math contests. That vigilance is a key reason why last December, Harvard won the prestigious Putnam Math Competition—beating out over 300 other colleges—for the 25th time. Harvard's closest rival, Cal Tech, has only won nine times.

It may seem as if the professors have no input, because they have fewer students alloted to the math department than, say, the football team.

Anonymous said...

You rely too much on the opinions of the math department and you're going to risk having another Unabomber among your alumni.

SteveH said...

At colleges like Harvard, what has equivalent pull, a regionally ranked lacrosse player and a student who is one of twelve national USAMO math winners? I would expect that colleges bring in the academic and athletic experts in the Early Admissions stage, but the levels seem to be quite different and I doubt the number of slots offered is equal for academics and athletics.

We got a brief view of this process for music at Brown. My son was told that he could set up a sample lesson with a professor there (This is in addition to his music supplement. This process is common for conservatories, but not well known for other schools), and she could (she had already seen him in a master class at Brown), push for his acceptance. We were told that playing this card was best used for those who were very likely to attend and might need a boost because of non-stellar grades and scores. Clearly, the piano professor did not have many slots to advocate, and didn't want to waste any pull on students not likely to attend. My son passed on the offer.

This seemed to be worse when we visited a music professor at Yale. He was lamenting the lack of pull he had even for a YoungArts finalist he wanted. He could give that person a great music supplement rating, but once it went to admissions, he had no say.

My view is that admissions offices vary quite differently in how they deal with external academic and athletic forces. It appears that athletics wins the battle in many cases. However, I suspect there are great battles in admissions offices about balancing these different forces. What helps the argument for many is the idea of holism and how it comes after a basic bucket fit for academics. While potential future Putnam math winners might have an advantage, I think it's hard to fight the pull and push of athletic coaches. Somehow, a "scholar-athlete" seems more impressive than a "scholar-musician" or just a "scholar", unless that scholar is one of a very elite group.

Anonymous said...

I'm pretty sure the Coop sells more t-shirts that say "Harvard Lacrosse" than say "Harvard Putnam Team."

Jeremy Lin got way more press than Gabe Carroll.

The alumni are more likely to donate when the sports teams are winning.

Anonymous said...

"Somehow, a 'scholar-athlete' seems more impressive than a 'scholar-musician' or just a 'scholar', unless that scholar is one of a very elite group."

I'd say that there are a few reasons for this, and I don't see them changing any time soon.

1) Scholar-athlete sounds broader (and thus more interesting) than scholar-musician or scholar-scholar because athletes use their bodies as well as their minds. Yes, I know, musicians also need to use their bodies, but I don't think it comes across the same way.

2) People who don't play a given sport can still watch it and see who is better (usually). And enjoy it. They can also appreciate the skill, even if they don't have it. This is also true for music, but is *not* true for things like the Math Olympiad or the Chess Team.

3) Sports tend to have a natural win/loss ranking to them, so folks can tell how "their" team is doing. And it is often very clear -- just count the number of points. You don't need judges to tell you who won (which is why I'm uncomfortable treating ice dancing as a sport ...). Music doesn't have this ... for the non-experts, it can be very difficult (or impossible) to tell who is "winning."

So ... my take on why scholar-athlete is prized:

    1) It shows excellence along two separate axes (unlike chess, say, or math).
    2) You can cheer on your team (unlike, say, the drama department).
    3) You can figure out for yourself if your team is winning (unlike music or math).

And, possibly, in the west this also goes back to the ancient Greeks: "A sound mind in a healthy body."

I don't see this changing any time soon. But I don't think it is a huge mystery *why* things are the way they are. If Michigan could fill their football stadium 6 times a year with people who wanted to watch the chess team or the math team, then Michigan would put more emphasis on recruiting these folks. But the stadium wouldn't be full and Michigan isn't going to recruit (much) for these activities ... certainly not compared to football.

-Mark Roulo

Anonymous said...

By ancient Greeks, I meant Romans :-)

-Mark Roulo

SteveH said...

I understand the reasoning. I'm just trying to get a feel for the level of imbalance.

My son is perfectly athletic, but not in an organized sports fashion, so it's not a holistic evaluation of two-axis characteristics. It's more about giving students and alumni some team to cheer. However, I just noticed that they had a big "PIANO BATTLE" between Harvard and Princeton. Maybe they could expand on the "battle" part to make it more athletic.