Every year an article or two comes out in the mainstream media about how SAT coaching is ‘ineffective,’ ‘educationally unsound,’ and/or wasteful, often prompted by a report on this issue by a supposed ‘expert’ often associated with a professional organization from the college admissions field. A new report has just been released by NACAC (National Association of College Admission Counselors), and media coverage has subsequently been covering two main issues raised in the report:
Claim #1: Small changes in SAT scores translate to big improvements in college admission chances.
But is this really true? Here are a couple of quotations from an article in USA TODAY (May 20, 2009) on this issue by Mary Beth Marklein:
“A 30-point boost in math and critical-reading scores on the SAT reasoning test is statistically meaningless yet could make or break a student’s chances of admission at ‘a substantial minority’ of colleges, a research paper says. And the more selective the college, the more that bump pays off, it finds.”
She goes on to report that:
“…in the new study, 20% to 40% of officials at 130 colleges that consider the SAT in admissions said a 20-point math increase or a 10-point reading increase would ’significantly improve a student’s chances of admissions’ if all other factors in a student’s application were the same. ‘If marginal college admission decisions are made on the basis of very small differences in test scores, a small coaching effect might be practically significant,’ says author Derek Briggs of the University of Colorado-Boulder. Briggs says he was ‘pretty stunned’ by the findings…”
My commentary on this particular point is fairly brief - I think the case is overstated, and it is likely very rare that a major admission decision is made on the basis of a 10 or 20 point spread in SAT scores. First of all, I hope I have some credibility on this issue because you would think that if I have any bias (perhaps both role-based and financial) it would be to encourage parents to be overly concerned about the SAT and too focused on helping their child raise their score. So when I tell you that based on nearly 20 years of working with college admission professionals and helping create and then run our College Admission Prep Camp programs, I just don’t see much evidence of such extreme sensitivity by colleges to small variances in SAT scores, I think you can trust me given that my apparent ‘biases’ go in a different direction. Second, I surveyed some of these folks myself a few years ago at the annual NACAC conference on issues related to extra-curricular activities and college admissions, and I came away with the impression that they don’t take such surveys very seriously. In my survey my results indicated that the student essay is often decisive in admissions decisions; well, I find that very hard to believe, as most any professional admission expert must know that many students are coached on such essays, and that in fact is one of the main reasons the SAT now includes a ‘writing sample’ section (so schools can check for writing sample consistency - so beware having an essay written for an application that isn’t really the student’s own work!). Finally, admission offices are often a couple of dozen people or more. When such a survey is filled out it is often by a very new / young addition to the office, and rarely by the ‘top dog’ admission director who truly makes many or even most decisions. Newer admission officials tend to oversimplify, in my experience (and I’ve spoken with hundreds of these folks over the years), and that lends itself to substantial exaggeration. In the end, for me to really have much faith in this fundamental claim, I would need to see hard evidence based on real admission data from real applications. Survey material is interesting and food for thought, but hardly reliable or cause for major concern.
In case you are interested, here is the address of the original USA TODAY piece:
http://www.usatoday.com/news/education/2009-05-20-SAT-prep_N.htm
Claim #2: SAT prep has little effect on student scores, never has, never will.
It is very interesting to me that an article making this claim was just published by the Wall Street Journal, with a very different take on the same research report discussed in the USA TODAY piece. In my view, it reveals many of the problems with the discussion as it is presented in the mainstream media. Here are a few thoughts I’d share in general about these sort of reports and the conclusions they often make about the ineffectiveness of SAT prep:
1. These articles almost always seem to feature a research paper by someone affiliated with College Board, the organization that creates and markets the SAT.
Many, myself included, feel that the College Board is very fundamentally biased toward the proposition that the SAT is not ‘coachable.’ The problem for them is actually very easy to recognize; if the SAT is coachable, then those with better resources be it financial, familial, school-based or other, have an unfair advantage over others, and then how can colleges and universities in good conscience use the SAT for admission purposes? There has in fact been a movement in the last several years among some schools to drop the SAT as an admission factor, and some small traction has been gained in that movement. The more that the SAT can be seen as a barometer of real inherent student ‘intelligence’ and ‘ability,’ rather than ‘rote learning’ or ‘coaching,’ the better the SAT makers are able to defend the proposition that the test is fair and reliable as a predictor of college level success, and thus should be retained. Financial bias is for many a strong motivator, and of course, they have it. Do so-called ‘test prep’ organizations have that bias as well? Of course, but what is so interesting to me in watching 20 years of media and dialogue on this issue is how the very blatant motivations of the test makers themselves is so ignored by the press and by pundits. In fact, from my perspective, I actually favor reducing or eliminating reliance on the SAT (meaning the SAT Reasoning Test, but not necessarily the SAT Subject Tests), but I don’t think that would seriously impact what we do with our College Admission Prep Camp programs, as there would still be SAT test preparation, and other admission factors, such as the student essay / writing samples, and the need for strong college counseling. Those with the most motivation to preserve the test are in fact the SAT makers themselves, the College Board.
2. I’m always amazed how the colleges and universities themselves decry the money being spent on SAT preparation. First, the cost of SAT prep, writing instruction and professional college counseling is often / usually the equivalent of just a few percent of even one year’s cost at a top school. If the college or university is worth that much, isn’t spending a small fraction of that amount getting counseling to make a good-fit choice a wise idea? Isn’t a little of that amount spent on learning to write more effectively a worthwhile investment if the student can not only get into their college of choice partially as a result, but can also benefit, perhaps for a lifetime, from just being a better writer? And finally, if the SAT prep works, even a little, given the high cost of a top college education, for many students might that expenditure not be worthwhile? Colleges would have us think that attending the strongest and ‘best’ school gets a lifetime of benefits – the more that is true, it seems to me, the more a modest relative investment in preparing for the process and making that decision wisely seems prudent.
As far as the article itself, here is the link:
Article title: SAT Coaching Found to Boost Scores – Barely
Wall Street Journal, MAY 20, 2009
http://online.wsj.com/article/SB124278685697537839.html
Here are some comments I have as regards this article, some of which were posted in the related comment section:
I’m afraid the WSJ has been taken by some poorly constructed ’studies’ here, and let some very specious data pass without serious questioning.
Derek Briggs conducted a “study” in 2002 for Chance magazine, and I pointed out some major flaws to him at the time that he did not deny in direct conversation. First and foremost in that paper which he seemed to like to refer to as a study, but which did not collect original data and instead only parsed and repackaged existing data (and which lacked controls on variables, as most any worthwhile study must have) students with as little as a few hours of SAT prep were lumped together for the purposes of analyzing the data with those students who had 40 hours of SAT prep or more! This is a deadly flaw, as saying that one day of SAT prep only increases one’s scores by 30 to 40 points, if that is the case, actually makes the point that MORE extensive SAT prep likely can actually have very substantial gains (for if one day of test prep yields 30 to 40 points of gain, how many points does one gain from 10 days of such prep? 100 points? 200 points? More?).
I contacted Mr. Briggs at the time and invited him to come and study Education Unlimited’s 11- and 18- day residential College Admission Prep Camps, and he said he was interested, but I never heard from him again. This was disappointing given that we resided in the same city, and I offered open access to our camps for him to conduct a true study, one with two test groups, controls and variables. I also offered to help recruit students to do a double-blind study of those receiving the test prep we offered, while also having another group fully controlled during that time to receive no test prep at all. In addition, we discussed testing a group of students who just took multiple College Board practice tests, in order to evaluate and determine how much gain a student could make studying on their own by taking practice tests. I believe to this day that testing three such groups would provide valuable information not only about the effectiveness of test preparation, but also about what techniques translate into the most effective education for students.
I pointed out to Mr. Briggs that Education Unlimited at our College Admission Prep Camp regular program uses College Board materials for the opening test and the final test, as well as much as possible for practice problems, and he agreed when we spoke on the phone that such data would likely be reasonably reliable and was different from how the SAT programs he included were run, at least for the most part and as far as he knew (in fact, he didn’t know any specifics of how the courses were conducted, or for how many hours, only the outer parameters of least and most total hours, but not how many throughout the distribution!).
At Education Unlimited’s original 11-day College Admission Prep Camp program, our students achieve about 200 points of average improvement over the course of the camp using the above described method. Be aware that in Mr. Briggs’ original article, most of the students surveyed were likely those taking the shortest test prep. At the time, he acknowledged to me that he did not know and that the data wasn’t even collected to indicate how much actual SAT prep was done, or how many students had done one day only, and how many had done 5 or 10 times that amount of preparation (referring the NELS data / study). It is quite obvious that most students in such an analysis would be those taking the shortest program, as the less effort and commitment required, and less financial cost presumably, the more students will participate. So what his data really indicts is very short, inadequate test prep. In a bit of intellectual sleight of hand, however, Mr. Briggs passes that off as an indictment of far more substantial test prep as well, because he believed at least a few of the kids reviewed in the data had likely taken longer test prep courses! I recall at the time asking him if it were possible that 90% or more of the students had taken only a few hours total of test prep, and he said he didn’t know! Obviously, if most of the students only took a few hours of test prep and that actually yielded 30-40 points of gain, that seems to establish that a little bit of SAT prep can yield quantifiable improvement, and would certainly leave open the possibility that significantly more preparation could yield better results. Honestly, it is actually fairly outrageous to me that such obvious interpretative flaws would not be laid out by authors themselves and indicates that one should be seriously concerned about ‘bias’ in how the data is selected, packaged and processed in Mr. Briggs’ reports.
I also want to point out that the data in this new paper/report actually appears that it might just be a rehash of the exact same material from several years ago, which was itself really a rehash of the original NELS data from the 1980s! Since that time the SAT has gone through multiple revisions and changes in scoring and content, and SAT test preparation itself has certainly improved at least some. So how can such old, only remotely related data indict modern era SAT prep? No one in the press seems to ask Mr. Briggs any of these sorts of questions, and as someone who cares about this subject, that is disappointing.
At other times the College Board has claimed that students can gain something like 50 to 100 points just by taking multiple practice exams. While I agree with that, and feel that many students could perhaps achieve somewhere near half of the gains of even fairly expensive test prep by just taking all available practice type exams and studying where and how they went wrong with incorrect answers (another topic I suggested to Mr. Briggs we study together, and he indicated initial interest but never followed up), it highlights that the College Board does know and admit that ‘preparation’ can make for tangible gains. They simply can’t have it both ways; either preparation makes a difference, or it doesn’t.
I am disappointed that research which is so woefully flawed from a methodological perspective is given such a pass by the WSJ, as admittedly it has been in prior incarnations of ‘research reports’ as well by the media which have reported on such ‘research’.
My point is just that I think it is worth noting how the College Board folks consistently try to spin all data to show that coaching the SAT fails, even when the data actually show the opposite. Thinking on this subject, and depth of research, is from what I have seen very poor, but if you look into the NELS data yourself, you will see that what I’m saying is quite accurate. As I’ve recounted, I mentioned this fairly dramatic analytical and interpretive flaw to Briggs at the time his paper was published, and he had nothing of substance to say to deny the claim. Now, for what appears to be the same material to be tossed out again as new research, with the same flaw repeated and not acknowledged or discussed yet again, well, I’ll be honest that in my mind that begins to call into question the objectivity of those producing this ‘research.’
You can learn more about our various College Admission Prep Camp programs (11-day Original, 18-day Intensive, Day Camp, and Resort) on our website and in our brochures. Personally, I am very proud of our ground-breaking College Admission Prep Camp programs, the first comprehensive programs of their type ever offered, and if you have any questions about these or others of our programs, please feel free to contact me or my worthy and able staff, as we would be delighted to discuss our programs with you!
Tags: SAT Prep