Search
  • Stephen Biss

Distorted Concept of "Accuracy", "Precision", and "Reliability"


Purpose:

To remind the Court of the concept of "reliability" used by the SCC in St-Onge.

To educate the Court as to the definitions of "accuracy", "precision", and "reliability" used by Brian Hodgson in his paper relied on by the SCC in St-Onge.

To focus on drift in accuracy and precision over time from:

the manufacturer's specifications,

and/or the accuracy and precision of the 8000 that was evaluated,

and/or this specific instrument when first placed into service,

as compared with accuracy and precision close to time of use.

Q. …But I want to suggest

to you that’s a different question from accuracy and precision

of the instrument and I think there’s the major point of

difference in your perspective from Mr. Kupferschmidt’s

perspective.

A. So, the acceptable range for a calibration

check is 90 to 110...

Q. Yes.

A. ...milligrams of alcohol in 100 millilitres

of blood so a result of 93 or 95 is considered to be within

that acceptable range, and when the instrument detects that

concentration during that cal-check, it allows testing to

proceed. If the cal-check is outside that acceptable range,

it will halt testing, and say ‘control outside tolerance.’

Q. So....

A. And the test will be terminated.

Q. So, you and the Centre determine accuracy

and precision based on policy as opposed to the definition

that Hodgson uses in the paper that was referred to in the

Supreme Court of Canada. You can go back and check that

definition if you like.

A. I might not have a copy of the Hodgson

paper.

Q. All right, I’m not sure if I have an extra

copy, but I can show you mine from Exhibit Number 2. It’s

Exhibit 2, Tab 2.

A. Ah, here we go. Okay. I’ve got a lot of

stuff here.

Q. Exhibit 2, Tab 2, page 85.

A. Eighty-five.

Q. Eighty-five has a definition of accuracy.

“Is the ability to measure a substance with a result that is

as close to the true quantity of that substance as possible.

For breath alcohol testing, this means measuring the breath

alcohol concentration as closely as possible to the true or

actual concentration of alcohol that is calculated to be in

the breath at the time of sampling.”

A. Correct.

Q. But if one wants to make a determination of

the accuracy of the instrument – I’m not talking about

accuracy of the procedure, but accuracy of the instrument, one

needs to take known samples, a number of them, using known –

known references, if I can use that, known alcohol standards,

run them through the instrument, a number of them, and take an

average. Compare that average with the value that one assumes

to be the true value. That’s how you determine accuracy of an

instrument. At least accuracy of the instrument for

instrument purposes, as opposed to accuracy of the subject

breath tests.

A. That sounds correct.

Q. Right. And with respect to precision,

“Precision is the ability to measure that same...” – and I’m

on page 86. “...is the ability to measure that same substance

repeatedly to produce the same result. For breath alcohol

testing, this means recorded measurement of the same

concentration of alcohol to produce results that are close to

each other as possible.” Well, that sounds a lot like

consistency.

A. Yes.

Q. But I want to suggest to you that the

determination – I mean, I think – I think I’m reading you as

saying the consistency from the Centre of Forensic Sciences’

perspective, the consistency is only with respect to the two

numbers of 93 and 95. It’s not with respect to consistency of

the instrument itself as determined by repeated – by a group

of control checks.

A. So, the accuracy or the ability of the

instrument to accurately record and measure the blood alcohol

concentration of a person who provides a sample is determined

by the calibration checks that are performed at the time of

testing. And so, those are deemed to be within the acceptable

range, then the accuracy of the breath tests based on that

standard are implied from that.

Q. I understand that that deeming....

A. And – sorry, and then there’s the two

samples that are taken in good agreement. All right? So,

that’s the second criteria is that the samples have to be

within 20 milligrams of alcohol in 100 millilitres of blood,

truncated. And they are.

Q. I understand your deeming.

A. Yes.

Q. I understand that policy, and it may in

fact, be a good policy for training of police officers, but

when it comes time in Court to access – assess accuracy and

precision I want to suggest you cannot assess instrumental

accuracy and instrumental precision based on just two values.

It’s impossible.

A. By that – yes, that’s correct. You would

need to do more analyses than that. And the accuracy and

precision of the instrument was determined at the time of the

evaluation by the Alcohol Test Committee.

Q. But not with this particular instrument.

A. Correct.

Q. And that was a long time ago.

A. That's correct. But, as the analogy I gave

before, is that two random instruments were tested back in

2007.

Q. Yes.

A. And the data from that applies to all the

instruments that come out from that, in the same way that the

alcohol standard solutions, when they come from the

manufacturer, are certified by us. We get a random lot.

Q. Yes.

A. And scientifically, it’s valid to take a

random sample of that, and that is applied – the data from

that is applied to the whole lot as an entity.

Q. They were random new instruments.

A. Yes.

Q. Recently calibrated by the manufacturer.

A. Yes.

[The cross-examiner should have clarified - But the issue is not accuracy, precision, and reliability at time of evaluation but rather:

1.this instrument

2. this instrument never so evaluated

3. long after 2007.]

Q. Right. Now, Hodgson says, at page 86,

third paragraph, “When evaluating instruments for accuracy and

precision, known concentrations of alcohol in water are used.

Accuracy refers to the ability of the instrument to measure

the breath alcohol concentration, to give readings that are

within plus or minus 5 percent of the true alcohol

concentration of known standard.” Now, I think there, Hodgson

is referring to the A-T-C evaluation procedures of plus or

minus 5 percent.

A. And it could be that that’s exactly what

he’s talking about with respect to all of these parameters,

accuracy, precision, reliability and specificity. That all

has to do with the evaluation parameters, not the operational

parameters.

Q. All right.

A. That’d be my interpretation.

Q. Then how is....

A. Without reading it entirely from the

beginning again, I can’t be more...

Q. Then how do....

A. ...conclusive than that.

Q. How does that help us? Using the Centre of

Forensic Sciences’ perspective on how you determine, deeming

accuracy and precision to be acceptable, how does that work

with this concept of reliability in Hodgson, which refers to

the ability of the instrument to perform over time, without

any significant drift in accuracy and precision. Because

obviously, the over time involves a comparison.

A. Correct.

Q. And you said that over time could mean

months, could mean years.

A. Yes. And the reliability of the instrument

is determined by the calibration check at the time of testing.

If there’s any gross drift over time, that would be detected

by the calibration check. Right?

Q. Even – even with respect to precision?

A. With respect to accuracy.

Q. What about precision? I mean, we have a

practical problem in Court now. We have to analyse the

question. I’m ultimately going to make submissions to the

Court to say there’s a problem with the reliability of this

instrument. So, reliability refers to the ability of the

instrument to perform over time without any significant drift

in accuracy and precision. Well, if we use as a reference

point the over time being at the time of the evaluation, we’ve

got lots of data on accuracy and precision of the evaluated

instruments.

A. Yes.

Q. Right? We have that from Terry Martin’s

paper.

A. Correct.

Q. We don’t have – I don’t know what data

we’ve got with respect to this Intoxilyzer 8000C when it was

first started being used, but we’ve got the practical problem

of what data do we use in Court to see if there has been or

has not been a drift in accuracy and precision, and what

you’re saying is, that there’s a policy, there’s a deeming....

A. A recommendation.

Q. A recommendation from the Centre of

Forensic Sciences that’s based wholly on just looking at two

pieces of data to make the determine [sic] of precision.

A. All right. Um, with respect to accuracy

and precision, the accuracy and precision is determined,

again, at the time of the instrument evaluation. The proper

operating of the instrument is determined at the time of

testing. With respect to the accuracy of that, it’s

determined from the calibration check. All right? So, the

acceptable variability associated with that calibration check

in the field is plus or minus 10 percent. We recognise that

there are external parameters that can cause the result not to

be 100 milligrams of alcohol in 100 millilitres of blood but

that could cause it to be 93 or 95. And so, given all that

variability we recognise that with that, that there’s going to

be some kind of change. If the calibration check is outside

the acceptable range, you’re not going to be able to proceed

with testing, and we know that the variability associated in

the field is always going to be greater – as you can see from

these numbers, if you were to take the last 10 values, and

then put those into a spreadsheet, you would see that the

accuracy would appear to be lower, but we don’t know that,

right? And the precision would be much greater because

there’s greater variability in these numbers compared to being

done in a laboratory and there always will be.

Comment:

It seems that the CFS "deem" accuracy and precision without drift over time simply as a result of the operational procedures at time of use. Where is the empirical proof for that hypothesis? It may be a technical norm or model that works for police, but is not good science. There is no empirical evidence that this deeming works. It is faith-based science. It does not foit the international literature.

#crossex #manufacturersspecifications #precision

0 views

© 2019 Allbiss Lawdata Ltd. All rights reserved. This is not a government web site.

 

 

For more information respecting this database or to report misuse contact: Allbiss Lawdata Ltd., 303-470 Hensall Circle, Mississauga, Ontario, Canada, L5A 3V4, 905-273-3322. The author and the participants make no representation or warranty  whatsoever as to the authenticity and reliability of the information contained herein.  WARNING: All information contained herein is provided  for the purpose of discussion and peer review only and should not be construed as formal legal advice. The authors disclaim any and all liability resulting from reliance upon such information. You are strongly encouraged to seek professional legal advice before relying upon any of the information contained herein. Legal advice should be sought directly from a properly retained lawyer or attorney. 

WARNING: Please do not attempt to use any text, image, or video that you see on this site in Court. These comments, images, and videos are NOT EVIDENCE. The Courts will need to hear evidence from a properly qualified expert. The author is not a scientist. The author is not an expert. These pages exist to promote discussion among defence lawyers.

 

Intoxilyzer®  is a registered trademark of CMI, Inc. The Intoxilyzer® 5000C is an "approved instrument" in Canada.
Breathalyzer® is a registered trademark of Draeger Safety, Inc., Breathalyzer Division. The owner of the trademark is Robert F. Borkenstein and Draeger Safety, Inc. has leased the exclusive rights of use from him. The Breathalyzer® 900 and Breathalyzer® 900A were "approved instruments" in Canada.
DrugTest® 5000 is also a registered trademark of Draeger Safety, Inc.. DrugTest® 5000 is "approved drug screening equipment" in Canada.
Alcotest® is a registered trademark of Draeger Safety, Inc. The Alcotest® 7410 GLC and 6810 are each an "approved screening device" in Canada.
Datamaster®  is a registered trademark of National Patent Analytical Systems, Inc.  The BAC Datamaster® C  is an "approved instrument" in Canada.