Training and Certification

Rubel Khan's Blog

Ten Things Microsoft Learning Can’t Tell You (with Q&A)!

Posted by Rubel Khan on July 29, 2009

Source (Born to Learn):

I first posted a few weeks ago to solicit questions about certification exam development. At the time I neglected to introduce myself. I’m Krista Wall, one of seven Content Development Managers (CDMs) on the certification development team. The CDM role is responsible for managing the design and development of Microsoft Certification exams. We facilitate the OD sessions that Shon Hong was talking about in his post earlier today, as well as item writing and technical review (alpha) sessions. We review items to ensure that they meet content specifications, assemble the forms for the exams, and hand them off for publishing. Along with a lot of other assorted duties, which you’ll hear about from other team members in later posts.

In that first post, I mentioned the fact that some things are just plain secret. So here’s the part where we tell you what we can’t tell you. Please note that this is not necessarily an exhaustive list of everything we can’t tell you—I’m leaving it open for a “Some more things we can’t tell you” post at some later date. But I’d like to get these out in the open earlier rather than later. Drum roll, please….

In no particular order, we can’t tell you:

1. Which items you got right.

2. Which items you got wrong.

3. Which answer is correct.

4. Which answers are incorrect.

5. How many items you need to answer correctly to pass the exam.

6. What percentage of people have passed a particular exam.

7. The text of an item. (And yes, I have had requests from candidates to “just send me a copy of the items I got wrong so I can study.”)

8. Whether or not a particular item was scored or unscored.

9. Which testing technologies are going to be on the exam you’ll be taking.

10. When we’re going to add new items to the exam.

We can’t tell you any of these things, not even if you’ve already passed the exam. The primary reason for these “we-can’t-tell-you’s” is that we have to protect the integrity of the exam content. In some cases, the “we-can’t-tell-you” could jeopardize the integrity of the exam by messing up the statistics that we use to judge the validity of each item. In other cases, it would provide an unfair advantage to you or someone else taking the exam.

One other thing I can’t tell you is when the Windows 7 exams will be released. That’s because we don’t know for sure yet. I can tell you, however, that we are working on them right now. More on Windows 7 exams in a later post. Until then, keep the great questions coming!

Posted by Krista


Friday, March 06, 2009 4:07 PM by becn

# re: Ten Things We Can’t Tell You

Interesting list. Most make sense – but I don’t see the need, or sense, in items 5,6, and 9.

What’s wrong with publishing the success rates? In fact, what’s wrong with publishing the success rates per region/nation?

Why keep secret the pass/fail mark?

I’m sure you USED to publicize whether something was adaptive or not (I might be confusing with novell), but it seems a little nasty to surprise test-takers with different technology. What’s the harm/risk in being up-front about that? Word gets out anyway, and I don’t believe that it’s breaking nda to mention the technologies involved.

The “protect the validity” argument really doesn’t hold for those items. And there’s no “unfair advantage” if information is public.

I suspect the real reason for those secrets is that there really isn’t a great deal of consistency in things like pass-rate between exams such as XP-Pro vs Vista.

Saturday, March 07, 2009 7:01 PM by Peter Read

# re: Ten Things We Can’t Tell You

I’d say 9’s actually valid to withhold (who cares the format, you know the technology right? ) – it’s 6 that strikes me as odd.  Either a flat “taken x times in all, passed y times” or even something more interesting like a histogram of %age of tests taken vs score (omitting pass mark or whatever).  I think it’d be interesting as an MCP and exam taker myself, but even more interesting as a hiring manager/interviewer to know that this person with an obscure cert must be great as only 3 people passed it out of 300 takers.  if only 3 people passed but only 3 people bothered to take it that’s perhaps less impressive (although statistically dodgy at that sample level I’ll admit)

Sunday, March 08, 2009 10:35 AM by Cathy P.

# re: Ten Things We Can’t Tell You

Instead of giving me this bullsh*t, why don’t you do something about the BRAINDUMPS that anyone can buy or download for free on eMule?

Howcome TestKing ( is kept on-line all these years???

Why don’t you do something about it, in order to REALLY protect the exam integrity?

Sunday, March 08, 2009 7:59 PM by Claudia Woods

# re: Ten Things We Can’t Tell You

Hi Krista,

Thanks for the list.  Is it possible to become a volunteer member of the exam writing team?  If so, please advise how to apply for such membership.

Monday, March 09, 2009 10:28 AM by Kevin

# re: Ten Things We Can’t Tell You

Well, #5 makes perfect sense to me, because there’s probably no hard number for how many items you need to get correct to pass.  Think about it for a minute.  If you have a 43 question exam and need 700/1000 to pass, does that mean that you need to get 70% of the questions correct (31, rounded up to the nearest whole number)?  If it did then you should be able to do that math on your own.

On the other hand, if different types of questions are worth more or fewer points based on their complexity, or their difficulty (as demonstrated by the percentage of people who got it correct in beta or in subsequent tests), then you can’t really give a hard number of “you must get X number correct”.  Also, if you throw into the mix the possibility that there are non-scored questions then that also affects the count.

Number 6 is a fairly useless stat (for us who aren’t creating the exams, anyway).  You could say that 95% of the people who attempt an exam pass…eventually.  Of course if it takes 3 tries per person to pass, you might have a 33% pass rate.  One stat makes the exam look easy, the other makes it look much harder.  But I think that the “difficulty” of the tests should all be roughly equal for someone who meets the recommended experience level with the subject matter.  But since the exams (at least as I understand it) are not identical for every tester, it’s hard to get particularly useful numbers without knowing exactly the makeup of the exam taken.

Monday, March 09, 2009 1:30 PM by libertymunson

# re: Ten Things We Can’t Tell You

Great comments (and of course, great resource for ideas for future blogs)! Let’s see if I can shed some light on the whys and why nots to the bullets that seem to be of most concern so far.

•Q: Why don’t we tell you how many items you need to answer correctly to pass an exam?

A: Because the number of items that you need to answer correctly to pass a particular exam is based on two key pieces of information: 1) the level of competency that SMEs tell us candidates must have to be considered minimally qualified in that content area and 2) the difficulty of the items on the exam. Both of these vary from exam to exam and can vary from one version to another for the same exam. As a result, this number will be different from exam to exam and possibly version to version within the same exam.  Providing this information will lead to more questions than it answers. Additionally, from a business perspective, we consider this bit of information to be proprietary; this is less important to me from a psychometric perspective but understandable why our business partners take this position. By the way, scoring, including the misperception that 700=70%, is clearly a complicated issue that will be discussed in more detail in a future blog.

•Q: Why don’t we tell you the passing rates for exams?

A: I think the comments that have been posted on this issue shed some light on the answer that question. Passing rate is a very poor indicator of the quality of the exam and is very easily misinterpreted. For example, a high passing rate could mean that nearly everyone who takes the exam is at least minimally qualified (which is good), the cut score is too low (which is usually more bad than good but it depends on factors that aren’t immediately obvious), or that exam content is too easy, meaning that unqualified candidates are passing (which is bad); a low passing rate could mean that those taking the exam are not minimally qualified, the cut score is too high, or that the exam content is too difficult. The passing rate in and of itself is more of a source of disinformation than it is of actual information. People draw the wrong conclusions from passing rates, and we don’t want to perpetuate that problem.

•Q: Why don’t we tell you testing technologies that will appear on an exam?

A: As Peter pointed out, if you’re at least minimally qualified, you should be able to pass the exam regardless of the testing technology is used. More practically, we don’t publish this information because the technologies used can and do change at any time.

•Regarding exam piracy, we have someone in our organization that works on this full time. She’s a bit shy, but I promise we will have a future blog on this topic.

Tuesday, March 10, 2009 3:25 PM by Michael Dragone

# re: Ten Things We Can’t Tell You

I have a question that you might not be able to answer.

Is Service Pack content introduced into exams if the SP changes the product’s functionality? Example: Exchange 2007 came out, exams and TKs were published. SP1 is released and adds/changes functionality (take the additional of Public Folder management to the GUI). The TK doesn’t get republished. Do “SP1 questions” get added to the exam? What about exam questions/answers that would change based on the application of a Service Pack?

Wednesday, March 11, 2009 5:28 PM by Krista

# re: Ten Things We Can’t Tell You

Hi Michael,

When a product update is released that changes functionality, we review the changes against the items in the pool, and determine if any items become technically inaccurate because of the changes. If items become technically inaccurate, we’ll remove them. If the change adds functionality that maps to an existing objective, then that functionality would be fair game for any new items written for that exam. Service packs rarely, if ever, have changes significant enough to require even the most minor of changes to the exam’s preparation guide. However, if/when service packs do require major changes to the content area, we create a new exam/certification rather than incorporating those changes into the current version of the exam. This ensures that people who hold the credential have demonstrated competency in the same content area.

The training kit may or may not get revised, depending on the scope of changes. I’m not an expert on how those decisions get made, but I will pass on your question to someone else who is.

For items where the answer would be different depending on the service pack, we would specify the service pack in the stem of the item.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: