Search
  • Stephen Biss

Manipulating Control Test Data to Enhance Precision


Purpose:

To neutralize the CFS scientist's assertion that anomalous data should be excluded from calculation of precision.

Sample cross-examination of CFS expert on indication of drift in precision over time (see Hodgson definition of reliability) following CFS expert trying to rehabilitate his earlier evidence on the need to throw out control test data outside 90 - 100 mg/100mls in any calculation of precision

This cross-examination shows the need for defence counsel to retain experts in statistics AND metrology.

This cross-examination shows the need for defence counsel to carefully review the study that the Crown expert is relying upon.

A. Just one thing before we get started. Let me

get organized here. On the previous day I spoke about a

paper. This was with respect to the data you referred to me

from South Simcoe Police Service.

Q. Yes.

A. Where the first calibration check was 88 and

then the subsequent ones were acceptable, and that it was a

phenomenon where we know that to happen with respect to

sometimes the first calibration check can be low, and that

wouldn't be improper to exclude that data point from the

calculation of the accuracy and precision of the instrument.

And I referred to a paper by Dubowski, and, in fact, it

turns out it's actually a paper by a gentleman of the name of

Hwang, H-W-A-N-G, et al, co-authors. And it's the Journal of

Analytical Toxicology, published in 2016, volume 40, page 338

to 344. And in that paper they refer to the fact that they

do a series of accuracy and precision testing that we talked

about, and that was shown in that case with respect to the

South Simcoe Police, and that they exclude the first point

because of they're aware of the limitations of the first

sample often being unacceptable due to

non-equaliberlation [sic] -- equilibration of the head space,

so they rely on nine data points, as opposed to ten.

Q. But, of course, they would never exclude

number two and number three and number four.

A. Correct.

Q. In other words, a significant proportion of

the data in any calculation and precision.

A. Yes.

Q. So if you're dealing with 50 points, it makes

no sense in doing a calculation of precision to exclude the

first 11, or 11 out of those 50?

A. Well, in this case... if I can find the data.

It wasn't the first 11 data points, but it was all the 11

data points that were below the low end of the acceptable

range of 90 milligrams of alcohol in 100 millilitres of

blood.

Q. Right. But on the last day you talked about

the phenomenon that during training you inform the officers,

the new qualified technicians or the recertifying --

requalified technicians, that sometimes you have a problem

with the first cal check being low because...

A. Yes.

Q. ...the system needs to warm up. Maybe they

haven't waited long enough for the whole system to -- is

"equilibrate" the right word?

A. Yes.

Q. To stabilize. To reach an equilibrium of

temperature so that Henry's law applies?

A. Correct.

Q. So that's with respect to one test, not eight,

nine, ten, eleven of them?

A. Well, what we were doing here with respect to

this data that you are referring to is different than what is

done during an accuracy and precision test with respect to

the Intoxilyzer 8000C.1. Here we took field data that had been

obtained at various times, various calibration checks, and so

as I explained earlier, that had any of these been part of an

actual breath test, breath testing wouldn't have been able to

proceed, and that the results are outside of the range that's

considered acceptable of 90 to 110 milligrams of alcohol in

100 millilitres of blood, and that's why they would be

excluded, because they're not representative of the range

that is accepted to be in.

Q. I see. But there's still control tests, and

we've got no indication, without looking at the accompanying

documentation, there was anything not functioning in the

simulator. There are control tests. We assume, without

digging further and actually seeing the officer's notes,

there are control tests that would appear to be with a proper

alcohol standard at the temperature shown in the data. The

temperature is there of 34 degrees plus or minus 0.2 degrees

Celsius. So why shouldn't there be at least an indication of

the precision of the instrument, to look at the last 50 cal

checks prior to my client's tests?

A. Because they are outside that acceptable

range. There's obviously some other issue that's going on

during those particular standalone calibration checks that we

don't know anything about, and because it's aberrant data,

you don't include it in the actual calculation.

Q. But the issue may be that the instrument is

producing consistently low calibration checks.

A. Well...

Q. And this is an indication that --

A. We know that -- sorry. Sorry. Go ahead.

Q. This is, this is an indication. That change

in precision is an indication to us. I mean, subject to

review of the documentation by the officer who was dealing

with it on that day, this is an indication of consistently

low cal checks.

A. And that's what the data that we went through

last time appears to show. But the instrument can't proceed

with testing. So in this case -- again, I don't remember how

many of these are associated with one particular case, but,

again, it speaks to a time other than when the tests in this

case were done. It speaks to another time in question that

you can't relate to the tests done in this case. And

normally, as I said, probably 99 percent of the time you get

a calibration check standalone and everything is fine,

diagnostics are fine and a self-test that's fine, and breath

testing proceeds to get two tests that are in good agreement

and game over. But occasionally there will be difficulties

associated with the breath tech trying to get the instrument

ready for the purposes of conducting breath tests. And as

we've talked about consistently, that sometimes that first

calibration check is below the acceptable range, in which

case something has to be done to try to get it back into the

acceptable range, whatever that might be. Sometimes it might

take more than one calibration check to get the instrument

into proper working order. It may take some additional

effort on the part of the qualified breath tech to get that

instrument ready and get the calibration check into the

acceptable range to proceed with testing, because if they

can't, then they're supposed to find another instrument.

Q. They're supposed to find another instrument

because...

A. Yes.

Q. ...it would then, at that point in time,

appear that the accuracy and precision of the instrument may

be jeopardized?

A. Well, there could be any number of

explanations why the data is low. That's one possible

reason, yes.

Q. And so that's why we should be looking at the

contemporaneous documentation2. that went with those control

tests?

A. I would disagree with that. Again, that

material speaks to times other than when the test in question

here was done, and you can't relate that information to the

tests that were done in this case in any way -

mathematically, statistically or otherwise.

Q. Well, we've just talked about a method

statistically of relating it to the time of the subject test

in a calculation of precision. I mean, either those control

tests are relevant to a calculation of precision or they are

not, right?

A. That is correct. In this case those 11 points

are outside the acceptable range, and, therefore, would be

excluded from any kind of accuracy and precision calculation.

Q. But if we're trying to determine the why of

why it happened, especially when they aren't all that far off

from the low 90s that we're seeing in some of the other

calibration checks, the only way to make that determination,

to give a proper scientific opinion, is to look at the

contemporaneous documentation.2.

A. You need to look at that information for the

purposes of the breath test that was done in that particular

case. It's irrelevant to the test that was done in this

case.

Commentary:

The cross-examiner should have gone further and once again connected reliability with drift in accuracy and precision over time. An indication of drift in precision is therefore relevant to an assessment of reliability. A review of the contemporaneous documentation is therefore useful in studying whether or not there is a real drift in precision or rather some other cause for the anomalies.

1. Maybe the bigger problem is that no one is doing careful accuracy and precision tests (in an ISO 17025 certified lab) of approved instruments out in the field, including this particular instrument. Here we have at least an indication of a problem of precision. The instrument results are wandering. That problem requires disclosure of all accuracy and precision testing and contemporaneous documentation of anomalies.

2. See Motherisk Inquiry Report on Contemporaneous Documentation and ISO 17025.

#crossex

0 views

© 2019 Allbiss Lawdata Ltd. All rights reserved. This is not a government web site.

 

 

For more information respecting this database or to report misuse contact: Allbiss Lawdata Ltd., 303-470 Hensall Circle, Mississauga, Ontario, Canada, L5A 3V4, 905-273-3322. The author and the participants make no representation or warranty  whatsoever as to the authenticity and reliability of the information contained herein.  WARNING: All information contained herein is provided  for the purpose of discussion and peer review only and should not be construed as formal legal advice. The authors disclaim any and all liability resulting from reliance upon such information. You are strongly encouraged to seek professional legal advice before relying upon any of the information contained herein. Legal advice should be sought directly from a properly retained lawyer or attorney. 

WARNING: Please do not attempt to use any text, image, or video that you see on this site in Court. These comments, images, and videos are NOT EVIDENCE. The Courts will need to hear evidence from a properly qualified expert. The author is not a scientist. The author is not an expert. These pages exist to promote discussion among defence lawyers.

 

Intoxilyzer®  is a registered trademark of CMI, Inc. The Intoxilyzer® 5000C is an "approved instrument" in Canada.
Breathalyzer® is a registered trademark of Draeger Safety, Inc., Breathalyzer Division. The owner of the trademark is Robert F. Borkenstein and Draeger Safety, Inc. has leased the exclusive rights of use from him. The Breathalyzer® 900 and Breathalyzer® 900A were "approved instruments" in Canada.
DrugTest® 5000 is also a registered trademark of Draeger Safety, Inc.. DrugTest® 5000 is "approved drug screening equipment" in Canada.
Alcotest® is a registered trademark of Draeger Safety, Inc. The Alcotest® 7410 GLC and 6810 are each an "approved screening device" in Canada.
Datamaster®  is a registered trademark of National Patent Analytical Systems, Inc.  The BAC Datamaster® C  is an "approved instrument" in Canada.