Donate
In this episode of K9 Conservationists, Kayla Fratt speaks with Dr. Simon Gadbois to talk about signal detection theory and his conservation work.
Discussed in this Podcast:
- Dr. Gadbois and his work in Nova Scotia
- What is signal detection theory?
- Signal detection theory is a means to measure the ability to differentiate between information-bearing patterns and random patterns that distract from the information.
- It has been around for a long time
- Mostly used for physical stimuli vs olfactory stimuli
- It wouldn’t be useful for clear distinctions or errorless data
- However, signal detection theory needs a clear yes or a clear no
- What’s the potential problem with proportion or percentage correct data as performance?
- What’s the problem with our basic lineup for training and testing detection dogs as it relates to SDT?
- How can trainers at home put this to use with their dogs?
Links Mentioned in the Episode:
Memorial University d’ Prime Calculator
Further reading on signal detection theory:
Where to find Dr. Gadbois:
Website | Lab Facebook Page | Lab Facebook Group
You can support the K9 Conservationists Podcast by joining our Patreon at patreon.com/k9conservationists.
Kayla Fratt (KF): Hello, and welcome to the K9 Conservationists podcast, where we’re positively obsessed with conservation detection dogs. Join us every other week as we discuss ecology, odor dynamics, dog behavior, and everything in between. I’m your host, Kayla Fratt, and I run K9 Conservationists, where I train dogs to detect data.
Today, I have the absolute joy of talking to Dr. Simon Gadbois about signal detection theory and how it relates to our conservation detection dogs. This conversation is dense and fascinating; if you’re someone who like me tends to listen to podcasts at an accelerated speed, I would recommend slowing down for this one. Potentially listen to it while you’re cooking instead of driving so you can take some notes and pause to look things up. It is a fascinating discussion, I think you’re gonna really enjoy it, but it is it is dense.
Dr. Gadbois studies olfactory processing in canids and the application of canine scent processing in wildlife conservation dogs. He works with Nova Scotian reptiles like northern ribbon snakes, and wood turtles. He also works with dogs used for biomedical detection, diagnosis, and assistance for diabetes, anxiety, and PTSD.
Before we get started, I’m going to remind you that our field vehicle repair fundraiser is still going. As we record, the van is finally ready to go and Barley, Niffler, and I are ready to start our field season. Our repairs were about $4,000 over budget; you can read all about that saga over on our Instagram or on K9Conservationists.org, where I’ve kept everyone updated on the repairs and the minor nightmare that’s been. Any support you can give to the fundraiser is super appreciated. I know that money is tight, so even if all you do is share the link, that’s helpful. You can find the link in the show notes and at K9Conservationists.org. You can also help the podcast by leaving us a review on the Apple Podcast store. Your reviews make my day, and they help other people find the podcast.
Welcome to the podcast! Simon, can you start out by telling us a little bit about your current work in Nova Scotia at Dalhousie University?
Simon Gadbois (SG): Yes, around 2006, we started working with wildlife conservation canines. It’s something I had been doing since the early to mid 90s, but not in a formal way. Around 2012, we also started working on biomedical stuff, including hypoglycemia detection in humans, and more recently, anxiety attacks in humans, in people with PTSD. I’m kind of moving away from the biomedical stuff, mostly because I think electronic noses are going to take over eventually. I don’t think this will happen with the conservation work just because it’s field-based, and I think dogs will keep an advantage there. No doubt.
KF: Yeah, I agree. I could see how it would be much easier to make an electronic nose for a medical application versus trying to find something that’s self-propelled and waterproof for the field.
SG: There was a documentary released recently, and I was asked to comment on a few things they had done. They were showing how humans are a little bit stupid the way they do things. The person was looking for something with one of those devices in an arena, and it was taking forever. They released a dog halfway into a five minute time span, and the dog found the source of the smell within, something like 10 seconds.
If you’re talking about search patterns, about the motor component to it, the looking for something – what we call trailing or air scenting – I think electronic noses may eventually get there, but it’ll probably be decades from now. They don’t work terribly reliably even if they’re right next to the order source. They also happen to be incredibly specialized, which means that, if you need to look for something different, you need to reprogram the whole thing. I think with field work, the dogs will have the upper paw for a long time.
KF: That’s good to hear for job security on my end, and I assume on yours as well.
Our main topic for today is signal detection theory and how we can use that for training our dogs, testing our dogs, and fielding our dogs. Can we start out by explaining what signal detection theory is? You’re the only person I’ve heard in this field who talks about it, and I don’t think a lot of our scentwork people know enough about it.
SG: Yeah. I know Nathan Hall did a talk about it at some point. We don’t hear much about it because a lot of people that come into this have a background in learning theory. I have some of that, but I also have a background in psychophysics and sensory ecology, and I think I have come more from the sensory processing perspective than from the learning theory side.
Signal detection theory has been around for a long time in psychophysics; Green and Swets (1996) is the seminal paper that started this. It was something that initially was of huge appeal for engineers. It’s an interesting continuation of information theory, because it’s about what kind of decisions you make when you’re uncertain. And that’s really the core of it. Because signal detection theory doesn’t work if the conditions are ideal. If it’s really easy to detect and discriminate between odors then you’re wasting your time trying to use it. But when you work with either a challenging stimulus, you know, a stimulus of low saliency, or a very challenging environment where there’s a lot of turbulence, a lot of wind, harsh conditions, or it’s really dry, then it becomes useful, because it tells you what kind of errors the animal makes. That’s really the crucial part of signal detection theory.
The paper we wrote on this, which actually was a proceeding for a conference, is way too short. But another thing that I think explains why we don’t hear much about signal detection theory in olfactory work is that it has been mostly used for physical stimuli, so auditory, visual, and tactile stimuli. But chemical stimuli, like taste and olfaction? Not that much. But there’s really no reason why it wouldn’t apply. But it’s not very common in the olfactory world. There’s no doubt about that.
KF: Yeah. So, it wouldn’t be very useful for something that’s really clear. For example, if you were trying to tell the difference between purple and yellow, you wouldn’t really need it because there’s not a lot of overlap. But if you were trying to decide at what point blue becomes aquamarine becomes green. Is that where signal detection theory starts being useful?
SG: Yep. That’s one of the examples I give my students. If there are no errors in the data produced by the subject, the dog in our case, then it’s useless because you need those errors to do the math. It assumes that there’s some level of difficulty to the task, but if not, then percentage correct is perfectly fine. But your percentage correct should be then astronomically good, right? So 98%, 100%, in which case, why would you need signal detection theory? That’s where it really loses its purpose. There are cases where, with my students, we start using it but then we eventually abandon it, because they say, “Well, I can do the calculations. I don’t have enough error terms here to get the D’, and certainly not the bias.” And then I say, “Stop, just use percentage correct, then you’re fine.” If the dog starts at 97% and above, typically, mathematically, you’re kind of losing the capacity to get something meaningful from signal detection theory data.
KF: Gotcha. So you have to have some amount of error. And then we’re looking at whether the dog is more likely to say that something is something, when maybe it’s more like errors of commission versus omission. Basically, that’s kind of what we’re looking at.
SG: In signal detection theory, we call it false alarms and misses. You need some misses, and you need false alarms. So, false positive, false negative. There’s an overlap here with diagnostic theory, because a lot of this, in fact, is kind of repackaged in a different terminology in diagnostic theory, which is a way of extracting the kind of mistakes. So again, false negative false positives, and how they come together in judging the subject.
What I like about signal detection theory is that it gives you at least two parameters that are really important. One is the D’, which is a measure of either detectability if it’s a true detection task, or a measure of discriminability, if it’s a discrimination task. Now, this is where in the olfactory world, we’re mixing up all these words, and they don’t mean anything anymore, because in psychophysics, it’s very specific what they mean. In psychophysics, detectability is when the subject – human, rat, dog, whatever – just says, “Yes, it’s there,” or “No, it’s not.” That’s it: presence or absence. That’s detection.
But as you know, in our industry and our business, and in the science of it, we say detection for almost anything, including the search work that dogs do in the field. But it’s so much more than that, at least based on the way it’s conceptualized in psychophysics.
The other way to do it is if you have stimulus one and stimulus two, what’s the difference between those two; that’s discriminability. And signal detection theory works here if you have an S1, or I should say, maybe an S+ and an S-. When you start having more stimuli, like in a lineup or a carousel, that’s where signal detection theory starts to fall apart, and then it doesn’t make very good predictions. You have to do some mathematical adjustments, and the reason to use it starts to get lost in the complexity of the math. The math is not actually complex, but it’s that then you’re really doing an approximation of an approximation of an approximation. And then you really lose the purpose of it.
This is why in our tasks, typically, we use a lot of scintillation vials. In the field, we collect our samples with this, typically from our reptiles. And the idea would be that you put this in your device, whatever it is, we use funnels on the ground, but it can be any kind of other arrangement. The target scent is there, and the dogs sniffs it and then needs to make a decision. In a real detection task, you would have either an S-; in this case, there’s nothing in it. And I’ll just use the one with a cap to indicate the S+, which has the target scent. In this case, the dog would simply keep the nose on, it’s actually not directly on it, but it’s on the front –
KF: Kind of a sustained nose touch.
SG: Yeah, for five seconds, we count it 1000, 2000. That’s the dog committing to the answer. It’s better in detection tasks, especially for signal detection theory, if you train the dog for a clear “No” as well. That’s what we call the yes/no procedure. In this case, the S- the dog would sniff and then say, “No, it’s not it.” We have them sit back. So that’s “No, that’s not it.”
Now, there’s another way to do it, which is the one that we hint at, and I regret this now, but in that little paper from 2016, I think, is to do what we call a go/no go. The “Go” is, again, commitment. So, the five second hold, and “No Go” is simply walking away. The problem with the “No Go” when they’re just walking away is that it’s a little bit ambiguous. So again, signal detection theory typically likes a clear “Yes” and a clear “No.” If you don’t have that, you can always throw some questions about what was that really a no? Right?
KF: Yeah, did the dog perceive it? Was the wind just the wrong direction at that moment? They didn’t actually say no.
SG: Yeah, now, I have to say this is mostly lab conditions. In lab conditions, my students don’t like that some dogs don’t have a no response, and you need to train it. And obviously, in the field, you don’t want a dog that will always sit back. We need to make sure that the dog understands that this applies to the testing, either during the initial training or in maintenance training later. Because sometimes, you work the field and you’ve got to figure out if the dog is off. Has the scent changed, which happens a lot with our reptiles. Or is it the field conditions today? Although it’s a weak scent, this dog seems to be a little bit off. For instance, we find more turtles than the dog does. Then you go like, what’s going on? That’s when you go back to the lab and retest the dog to see, are they starting to look for something else? Have they forgotten what the odor is? What’s going on?
Technically, you could do the go/no go as we say in that in that paper, and it’s fine. If you’re satisfied with the dog just walking away from the odor as a no, that’s fine. But in strict signal detection theory, the argument I make now is that it’s better if the dog can commit to a no, except if it interferes in the field. If a dog constantly sits back to say, “There’s no turtle here.” Well, okay, then I understand. It’s problematic, but most dogs will learn the difference between the lab and the field.
KF: Yeah, it seems like it’s been pretty obvious to most of the dogs that I’ve asked to do both. I’m trying to imagine doing a plant survey where the dog is like, “Not this one, not this one.” You’re never gonna get anywhere.
SG: It would take forever.
KF: Yeah, it would take days to clear a 10 meter plot.
So, basically, signal detection theory is a way for us to get an idea of whether our dogs are giving false positives or false negatives, or, misses versus false alarms. And this is something that we would want to do prior to fielding to make sure that we understand where our dog’s at, and potentially to troubleshoot afterwards. Am I following?
SG: The concept we need to add here is the concept of bias, because that’s where signal detection theory shines. It will tell you what kind of dog you have. In other words, is your dog a liberal decision maker, or a conservative decision maker? That’s a lot of different things. It can be temperament as a dimension of personality, it can be the training, it can be the reinforcement history, it can be all kinds of things that make a dog one way or the other.
That’s very important, because, as you know, in some areas, you want extremely conservative dogs as opposed to liberal. The typical example I give my students is mine detection. In mine detection, you’d rather have a dog that does a lot of false alarms, because the cost of a miss is that the dog and potentially yourself just blow up on the mine. At the same time, there’s other examples. We had a contract, for instance, with the Canadian Forest Service. In that case, they had a problem with larvae from a specific insect that was damaging trees. And the method they were using at the time was producing a lot of false alarms, which meant that they were cutting and destroying more trees than needed. They would cut down trees and then realize, darn, there was actually nothing here. For the lumber industry, that’s actually a potential loss that accumulates with time and becomes a problem. What they actually wanted in this case were more conservative dogs. They didn’t want dogs that were doing a lot of false alarms. Tell us only when you’re really, really sure there’s something. A dog that, when unsure, is more likely to say no than yes; that’s a conservative dog. And the liberal dog says more yes than no when they’re not sure.
But the beauty of this is that, knowing that it’s partly the dog, most dogs will have a bias. But one of my best dogs, Zillah, for instance, was what we call the ideal observer; she was right smack in the middle. She did about an equal amount of misses and false alarms. In some cases, that’s exactly what you want. But many dogs will fall on one side or the other of this bias. They’re either liberal or conservative, and in some cases, extremely liberal and extremely conservative. Sometimes that works for you, sometimes it doesn’t.
The beauty of this is that technically, via training, you can change this. So if you have a dog that you find a bit too conservative, you want that dog to be a little bit more liberal, it’s easy. Typically, what you would do is, when a dog sniffs and seems to be hesitating, and especially if you know that the target happens to be there, you would immediately say, “Good dog,” click, or, you know, here’s the kibble or whatever you do to reinforce your dog. Now, that’s a dog that learns that, “Okay, if I think it’s it, maybe it’s it, and I should say it is it,” as opposed to the other pattern where you may want to withhold any kind of reinforcement if the dog is showing uncertainty.
In wildlife conservation canines, we have a bit of an issue when a dog gets to a scent, and the target is not there, right? This is a typical problem that happens to us. We have to make decisions about what’s good for us or right for us in that case. I have a tendency to give the dog the benefit of the doubt, especially if in the context, it makes sense. And usually with wood turtle spawn, since that’s one of our primary research subjects, they often leave some kind of mark or sign on the ground. That helps me decide, yeah, you’re right, there was a turtle there this morning. I tend to be a little bit more generous with my dogs, because I don’t mind them being a little bit on the side of false alarms, meaning being liberal. And we gather this data as a data point, there was a turtle here this morning or yesterday or whatever. So that becomes a GPS point. But I could see why some people would like to say, “If there’s nothing there, I’m not interested, find me the target,” in which case, obviously, then you want a dog that will just give a strong answer or response only when the target is actually present or extremely close. Or recent in time.
KF: Yeah, I’m thinking about some of the scat projects I’ve worked on where, we actually need enough scat present that we’re able to collect data to send to the lab. So generally, if the dog is alerting to something that you know was there, but has now been largely eaten, that’s not generally something we want the dog to be focusing on.
I was out once with a dog that found a site where a puma had cached a dead sloth and a dead armadillo in an area, but there was no scat, but it reeked of cat. And I was just shadowing a handler at that point, so I can’t remember whether they decided to reward or not. But you know, it’s one of those things where it’s like, the dog was absolutely correct. We could smell it, it was clearly a cache, but there was no scat to collect. Depending on who your funders are, and what the goal of your project is, you may or may not decide to reward.
SG: This is one of those cases where I think a dog that is tolerant of being on intermittent reinforcement is important, right? And this is where a lot of dog trainers these days don’t understand intermittent reinforcement. But that’s a such a great example of a situation where if the dog has not experienced intermittent reinforcement before, they will indeed get frustrated and stop working.
It’s a great case of, I was going to say negotiation, but that’s not really the word. But there’s a kind of like implicit understanding that you need to have between you and a dog that, “Yeah, I know, you’re not too sure. But yeah, we agree, something is happening.” And then you reinforce or you don’t, but you don’t leave the dog hanging too much, either. I’ve seen this in the field, and this is exactly where a dog will fall to one side or the other of what we call the ideal observer in signal detection theory. It’s where you technically control it in the field; in those moments where the dog’s not sure, you’re not sure, but Wow, okay, I do smell the cougar here. So yeah, you’re right, but what do I do? What’s the goal of this study? What do the stakeholders actually want? They want scat, but it’s not here, so what do I do? But the cost of this depends a lot on the cost of your false alarms. If there’s really no cost in the dog giving you that false alarm, I say reinforce. After all, you want the dog to find, you know, cougar scats in this case. So go for it. Just reinforce it. Worst case scenario, you hit a few spots where it does nothing, but it’s better than missing it.
KF: Yeah, and as kind of a nerdy naturalist, I was pretty excited to find a dead sloth and armadillo, so I don’t mind if my dog tells me about that, as long as it’s not something that’s so frequent that it’s going to slow down your searches.
I think this is where signal detection theory and understanding what errors your dog is likely to make is probably at its most important. The example we use a lot in North America is, say you’re trying to find Bobcat scat, and red fox scat is pretty difficult to distinguish visually from Bobcat scat. As a handler, it can be challenging if your dog is starting to alert to stuff where you’re just like, “Well, I’m not really sure if that’s Bobcat or red fox,” and there are enough red foxes where it would be problematic as far as lab fees and survey time for your dog to be finding every red fox when you’re trying to find Bobcat. So I’m guessing that is where really understanding the sort of errors your dog is likely to be making would be incredibly important.
SG: Yes, and to me, it shows the importance of a dog that does not always expect a reinforcement when it finds the target. Because then you’re creating an extremely conservative dog, which may not be what you want. The other dimension of this that becomes important is, I think you are allowed to grade the way you respond to a good or potentially good response. When we have a target in the field, if the dog gets right to the turtle, we give a huge party, “Yay, good dog!,” Click, kibble – although, honestly, our dogs don’t care about kibble. At that point, they’re so excited and so self-reinforced in the field that they often don’t pay attention to that at all.
Contrast that response to “Yeah, good dog.” In other words, they get the point that you get really excited when we find the actual turtle, and it’s right there, yay! Versus Yeah, you’re still supporting me, A for Effort. But it’s crazy, the number of trainers that don’t seem to understand that concept that it’s okay to grade the response. It’s not always the same intensity, the same response, the same saliency? No. Why not be able to grade this in terms of you know, yeah, it’s a B+ versus an A+, great comparison.
KF: Yeah. My dogs have different marker signals. I don’t usually use a clicker in the field because I’m not gonna walk around with my finger on the button.
SG: People don’t understand we don’t have free hands in the field!
KF: No, I’m usually using them to keep myself upright! Or pull ticks out of my hair, or something.
So we use “Catch” and “Tug,” where one’s for a thrown toy, one’s for tugging. My dogs have different motivations; one dog’s tug is better than throw, and the other is the other way around. And then both of them also have a “Yes,” or a “Find it,” which are either food delivered to their mouth, or food delivered on the ground. Food is less exciting than toys for both of them, so if I’m not totally sure, but I think there’s a good chance that they found the target, then I can grade my response to them. And then if I’m really not sure, I don’t think this is it, then I might just get some like kind of casual verbal praise, we’ll take a water break, and then we’ll keep going.
So we can really grade our responses. It is really surprising to me how resistant people are to that in this field. And maybe this is something that at some point, someone – you – need to do a paper that shows that we can give a less exciting reward without creating false alarms. I think most people are more concerned about false alerts or false alarms than they are about misses. But maybe that’s not quite accurate.
SG: Kayce Cover has this interesting idea of informing the dog. She calls it hot or cold. It’s a control condition, most likely in the lab or set up in the field. And I think that’s very useful, too. But you would be surprised, again, how many trainers are totally against this idea. I’m thinking, why not tell them? You’re close, but not exactly.
She does it with this sound, and I’m not sure I would agree with that, but she grades that faster, or louder, I think something like that. And I think that’s a great idea. I do it with my voice; canids are intrinsically very attuned to vocalizations and what we what we call the prosody in human language. And I think it’s very important to use, and you can modulate it. So if you’re uncertain, you let the dog know that you are uncertain. Using an uncertain pitch, “Yeah, I think so. Good dog. Good dog.” Eventually they go, “Okay, we understand each other. We’re not sure about this.”
KF: Yeah, Patricia McConnell’s entire PhD work was all about how responsive many of our domestic animals are to consistent changes in our vocalization.
I grew up with this beautiful Labrador Retriever that I so wish I could own again today, because she would have just been the perfect working dog. And when we threw her tennis ball, we had a big Chuck-It, and we lived on 40 acres, so we’d lose those tennis balls out in the 40 acre fields all the time. And my dad and I taught her that, if she was at least orienting towards where we thought the ball had last gone, we would kind of go “Yeah, yeah, yeah!” and get louder and then go quiet again if she if she circled away from it. And this is just a lab playing fetch in the backyard, but it worked really well. I haven’t done it with my search dogs as much but that would be an interesting thing to play around with.
SG: Yeah, and especially in early training, or when transferring the dog to a new scent, I think it’s totally justified to guide them a little bit, and do the coaching the same way we would in any kind of other kind of training or learning with our children. Like, “Yep, you’re getting there!” right?
KF: My puppy is about seven months old now, and he’s getting ready to deploy on his first project. And in the last month, I’ve noticed he’s really good at going out and finding the target. Right now, we’re working on dead bats. I’m trying to build in an actual alert for him, because he was going up, finding the bat, and then running back to me for reward, which could be an alert. That would be just fine. I know that’s what a lot of search & rescue people do, but I would prefer to have a dog who stays at target and alerts to it. So in that case, I was reinforcing him a little bit for coming back to me and then guiding him back to it and coaching him into that alert a little bit in a way that I would imagine make some detection trainers cringe.
I think the industry has swung a lot. You see things like traditional narcotics vehicle searches, where you’re presenting everything to the dog, and you’re right up the dog’s nose. I’ve also seen a swing in the opposite direction where we want the dog to pretend we don’t exist, and we want to be so thoroughly disengaged from the search, which is certainly valuable. And it’s a cool skill for your dog to be able to search independently. But that doesn’t mean that we have to do 100% of either one of them all the time.
SG: I think it depends on the dog. It depends on the situation. I think sometimes we’re just a little bit afraid to be versatile, right? Versatility is important so a dog can adapt to new situations. For example, I’ve had dogs that are fantastic at training in the lab, but as soon as we get into the field, they’re not connecting the dots. In fact, I’ve had two that never did. And I don’t know if it’s because they don’t like to work in the field, they just like the lab part and think that’s fun. And at the other extreme of this is a dog, Flynn, that I brought to the fields, and it was his first time to the field. He watched Zillah work, and an hour later he was finding more turtles than Zillah and he had no training whatsoever. And he became our turtle dog until he died in February. He was our best dog by far.
So yeah, dogs are amazing, but you need flexibility. And I think you need to see it. And what I don’t like about this industry sometimes is, it seems we’re not flexible. We’re so obsessed with protocols. This is how you do it, and every dog must go through the same procedure. Look, we’re not working with rats and Skinner boxes here. We’re working with a much more socially engaged species with humans. And let’s just take advantage of it, right?
KF: Yeah, absolutely.
Hey, everyone, just popping into this episode with an update on our Patreon. We still have the $3 a month Doggie Detector level, which allows you to ask questions for me and the guests to answer each episode. But that also lets you join our monthly training video analysis calls. These calls are going to be recorded, of course, and we’ll also publish the video afterwards for everyone to view and ask questions about prior to the call to ensure that all time zones can have participants fully. So we’ll basically publish the video we’re going to analyze so that you can ask questions and view it and prepare ahead of time, then we’ll have the call where we talk about it, we can have beverages, it’ll be a good time. And then all of that is going to be shared later. So you can participate before, during and after again, just for $3 a month.
Now, at the $10 a month Scensational Scientist level, you get everything that we got before at the $3 level, plus you get to submit videos of your training sessions for those calls. So this is perfect for the aspiring K9 Conservationist. And your target odor doesn’t really matter here as long as you do communicate what it is so we can think intelligently about your goals. This is going to be great for nosework competitors and other canine handlers as well. We’re really striving to make these video calls super kind and supportive and helpful. It’s going to be a nice safe place on the internet to get good feedback on your training sessions, because I know how much of a struggle that can be especially in the scentwork world.
Then finally, the Canine Conservationist patrons get everything from those other two tiers, plus a private 30 minute training call with me to go over whatever you’re running into with your dog. That tier is just $25 a month, and that’s cheaper than booking my time at journeydogtraining.com for behavior modification, and that’s just because I love you and I love my patrons. That’s definitely something to check out. You can join that Patreon over at patreon.com/K9conservationists or at the link at K9conservationists.org. It’s a tiny link up in the top bar, and we also drop that link into our show notes. If you’re listening on your podcast app, you should be able to find it right from there. Thank you guys so much, and let’s get back to the episode!
Let’s circle back a little bit more to signal detection theory. I wanted to talk about the problems with proportion or percentage correct as our performance metric, because I know we alluded to that earlier on. Can you explain why we don’t like the idea of just saying, “All right, we did a bunch of lineups, that’s our lab pre-training. Now we’re going to see what proportion that gets correct.” And then we know the dog is ready to field or not. What is the problem with that setup? And we can talk from the proportion correct side and the lineup side, because I know we’ve got things to say about both.
SG: Technically, there’s no problem with proportion correct. Even if you do the single presentation, or what we call Two AFC, a choice between two or three, or four or six, and that becomes the lineup procedure. So there’s nothing wrong with it, and you can figure out if they’re above chance or not.
But in signal detection theory, if you go with that thinking, you’ll get more; you get a lot more data. You get the bias, or this criterion, as we call it in signal detection theory, and you get that D’ value, which is not terribly intuitive, you’d have to spend a little bit of time looking at what it means. A lot of people think it’s a linear measure, and it’s not. Often, textbooks will tell you it’s between zero and four, which is not exactly true, it can be above four, so two is much better than 50%. Anyway, you need to get familiar with the measure, essentially. So, you get the D’, you get the different types of criteria and different ways of calculating it, and all that. There’s even a nonparametric version, by the way of signal detection theory. It’s called the A’ instead of D’. So there’s the whole mathematics of it. Signal detection theory, from that perspective, is just a quantitative tool that gives you more information about where your dog is at, assuming you’re interested in that. Some people may not be.
For diagnostic purposes, it’s very important, in my opinion. But most biomedical research will just report specificity and sensitivity, which is fine. But it’s related to all of this. Specificity measures how conservative the dog is; is the dog able to differentiate this stimulus from others? Sensitivity measures how detectable a stimulus is. These are complementary measures. And in most of the calculations that we have, you know, accuracy is actually the average of specificity and sensitivity. So if you have 90%, accuracy, you may actually have 95% sensitivity and 85% specificity, or the reverse. There’s a reason why I’m going there, by the way, mentioning specificity here.
In psychophysics, the lineup is intrinsically a specificity task, because it’s the one S+, meaning it’s measuring how discriminable – not detectable, it is from others. So you’re biasing a dog towards being very good at specificity. And this is why most of those studies have a higher specificity than sensitivity. There are exceptions, but a lot of dogs in diagnostic work have very high specificity and lower sensitivity. And that’s because you set them up for this.
There have also been studies done with a percentage. Let’s say that you have 10 trials in a session, you would have probably one out of 10 trial where the lineup is blank. So the dogs has the opportunity to say there’s nothing here, right? Because otherwise the dog will think, “Anytime I walk into this room in front of the lineup, there is a response and answer.” It’s not true. And there you really bias the dog to always alert to something when actually it’s possible there’s nothing. Typically about 10% of your trials would be a blank lineup. The other 10% are lineups where there’s actually two targets. The argument Schoon and Haak (2009) make here is that, often dogs will run the lineup, and as soon as they find a target, they stop. And what you actually want the dog to do is be able to complete the lineup just in case this is not really the target, or there’s a better target. In other words, there’s a scent that smells more like the target scent than this one. That’s something we discussed in Gadbois & Reeve 2014, this idea that you have 10% blank lineup, 10% two targets, and then the rest is one target. But you’re still setting up the dog entering the room to think that most of the time there will be a target in there. When the reality of the field? Nope. Especially in the kind of work we do with endangered species or species at risk, you can spend a full day looking and finding nothing. By definition, they’re not out there much. So that is potentially a problem.
Again, if your dog is used to reinforcement, it may be not just important, but essential, because in the field, they may deal with what we call rare events, right? And even in humans we know this, they’re looking for rare events. The example I often give here is the NSA example. Jeremy Wolf, a former postdoc here at Dalhousie University in cognitive psychology, figured out that one way to keep NSA agents alert and motivated during their work, when they look at all the luggage going in front of them, is to have the computer engage them in a kind of video game. In other words, the system plants, weapons and explosives in there, and the person does a strict detection task. Yes, nope. Yes, no. Now, if you say yes, and the system just planted a grenade. Well, good, you say yay. But oh, by the way, this was just part of the game. But if the person says yes, and the system goes, wait a minute, there was no grenade in there that was planted, you better check that luggage. And that’s simply keeping the person interested.
What’s interesting about the kind of work we do is that we are asking our dogs to perform in rare event situations, it’s the same. It’s the same with dogs that do hypoglycemia detection. Hypoglycemic events in kids and nocturnal hypoglycemia may occur once a month. People realize that dogs have stopped working after a while because they never get the event, they never get to be reinforced for it, so that’s why it’s kind of problematic. What we’re basically saying is, the lineup sets the dog up for knowing that there’s a target somewhere in the room when they get in there, when the real world is not like that. This is why detection tasks are different. When a dog gets in the room, there is a container with or without the S+ in there, and says yes or no. And you can adjust the occurrence of a yes to something that closely matches what’s actually happening in the field to make sure that your dog is not too sensitive to that expectation that 50% of the time there’ll be something there versus only 10% of the time. But in a lineup intrinsically, the dog knows. It’s not a detection task, it’s a discrimination task. It’s more about comparing this to this.
Occasionally the real world can be like this, like the work we did with snakes. We were looking for ribbon snakes, but sometime you would find garter snakes which are from the same genus and most likely have the same motor. But the real world of a search dog is rarely like that. A lot of the time it’s literally, is it there? Can you find this when you don’t have a comparison? The way I explain this to my students, I say it’s the same as a difference between a multiple choice exam versus a true/false. In a multiple choice exam, you can compare and contrast A to B, A to C, A to D, etc. Some people say, well, it can be harder, because then you confuse the animal more with possibilities that are very close to the target. True. But again, it depends on the multiple choice test, on the question. So it depends on your lineup, essentially. I experimented with true/false with my students would final exams, and I can tell you, it’s not necessarily easier to give a real good, true/false answer to a true/false exam is not as straightforward as students actually predict if you were just to ask them is it easier. The score on the true/false exams I have done with my students’ final exams are about the same as with a multiple choice, despite the fact that they say, Oh, this is going to be easy. It’s a true false exam. Yeah, well, it depends on the statements you make, right. So again, it depends on the stimulus that is given to you.
So now, remember that true/false is better if you want to use signal detection theory, than if you have a lineup or a multiple choice. Now, like I said, the books that I have will tell you that you can actually have a two way FC, so a two alternative forced choice or three alternative forced choice, etc. when applying signal detection theory. But like I said earlier, the problem is, let’s say you have a regular lineup of six items, that’s a six AFC. You lose the predictive predictability of the model, the model becomes a lot more speculative; as we would say in stats, it’s not as robust. So, it’s doable, but it’s not advisable. And you’re really kind of guessing what’s going on if you try to apply signal detection theory. If it’s a two-way, you have to do the square root of D’, basically. So if it’s six AFC, you can see how it gets a bit ridiculous. At some point, you get really strange, low numbers that are kind of meaningless. So that’s why it’s not advisable to do it with a lineup. Again, theoretically, you can, but most people avoid doing it.
KF: Yeah. So for people at home who are interested in trying this out with their own dogs, whether they’re competitive scentwork trainers or more on the conservation dog side of things, are there good places to go as far as finding these formulas and learning how to set up these choices for their dogs so that people can actually get a good idea on where their dogs fall?
SG: Well, yes, there are a few resources. One you can find online is from Memorial University, here in Canada, that’s a little Javascript app that I’ve been using with my students when they want to calculate their D’. The books I just showed, although they get technical, it doesn’t have to be that technical. I know there’s spreadsheets out there that are used by people to do quick calculations. But here’s the thing; the basic thing that you need is this quadrant. You need to be able to fill a matrix based on four scenarios that will give you the hit, the false alarm, the miss, and a correct rejection. So in other terms, true negative, true positive, false, negative, false positive.
KF: Yeah.
SG: That means that in your setup, you need the situation where the stimulus is there and the dog can say it’s there, that’s a hit or true positive. You also need a situation where the stimulus is there, but the dog says it’s not, that’s a miss. You get the point. You just need to be able to put numbers in all those four in that matrix of four cells. And to calculate a D’, really all you need is hits and false alarms. And there’s a little formula. You need to do the Z scores, so you need a little bit of stats to be able to do this. But the app that I was talking about earlier, you can just plug in a proportion, the numbers, and they always work for you. I can send you the link.
KF: Yeah. And we’ll share the links to that. And you know, Schoen and Haak (2009) and Gadbois and Reeve (2014), and the two books that you showed, we’ll have all those links in the show notes for people so they don’t have to panic about trying to track it down themselves.
SG: Just to go back to the lineup thing, to complete your question, outside of signal detection theory, there may be a working memory issue. One of the arguments that we made in Gadbois and Reeve (2014), and Lazarowski and others have published more on this recently, was that, especially if you do a matching to sample task, potentially there is a working memory issue in a lineup, and we demonstrated that with two of our dogs at the time. In other words, what we’re saying is that the dog has a very high accuracy early in the lineup, with the first items that it samples. And that accuracy goes down, as you go down the lineup, basically. We show that after the third item in a lineup or carousel, whatever, then really, the dog doesn’t know. Now, this is in matching to sample, meaning that, every time the dog enters the room, the target sample may be different, right? So you’re working in different odors to start with.
What we didn’t explain in that paper, and the other one that you were talking about, is that sensory interference is still an issue in lineups. So the first one is working memory. But even if it’s only one odor, or the contrast between let’s say, two odors, or yes or no, or a urine sample that may or may not contain a marker for cancer, let’s say, the problem is that as the dog samples from one station to the other, there is actually sensory interference, first of all sensory memory. What did I just sample there, you know, previous one, next one, etc. Not to mention all the molecules from the previous sample that are now mixed in the nasal cavity of the dog.
So, there’s still an issue here in terms of how you space this out, like physically and temporally. And that’s why again, the detection task is better, because in the detection task, the dog enters the room, there’s one sample, they sniff it, yes or no, that’s it. Just, Is it cancer? Is it not? No comparison, no possibility of mixing up all of these odors and, and then giving the wrong answer. And that’s the other dimension of it, there’s literally mnemonic, meaning memory interference, but also sensory interference, when you do lineups, especially the way they tend to be done. I often see these dogs doing lineups or carousels, and they’re literally running around it. And I’m thinking well, in terms of sensory sampling, this is good if they can do it, and if they can be accurate, but imagine if the odors were really close, you know, between an S+ and an S-, or have really low saliency. There, suddenly you get this threshold of error that’s totally different.
KF: Yeah, I mean, I’m imagining, if you blindfolded me and set me loose in the fruit aisle of a grocery store and had me find the citruses, I could do that pretty easily, I would imagine. But if you started having the time to tell the difference between lemons and limes, I’m not sure I could do that by scent. Especially if I was running through the grocery story trying to do that.
SG: There’s another analogy I often give my students, especially for the working memory part. When you do the matching to sample, you give the dog an order, you say okay, this is lavender, and now there’s a lineup of other essential oils or whatever smells – go find a lavender one. We’ve seen dogs sometimes that go in like, “Yeah, it’s lavender. It’s very strong. This is obvious.” But halfway through the lineup, they look at you, and you can tell they go like, “What was it again?”
Here’s the comparison. It especially if your potential number of target odors is large, like 8, 10, or more. A student of mine that once said, “Okay, it’s a little bit like when I’m at home, and I change the water in my aquarium. And I know that when I get new water in my aquarium, I need to match the temperature, otherwise, my tropical fish will actually die. So if I put my hand in the initial aquarium, and then in a second one, I can tell easily, it’s the same temperature or not the same. And then I go to the third aquarium, I may remember what, you know, the first aquarium was like, then the fourth, then…” But you know, as you go down after a while, you go like, “Wait a minute, what was the first one again?” And then you have to go and resample the aquarium to match and do that accurate comparison.
So, in other words, I think with sensory information, when the target smell can be different every session, I think it’s important to remember that they have a limited capacity at the sensory and mnemonic level to process stuff. Now, if the dog is allowed to resample, great. And again, I think in the kind of work we do, it’s not that important. Usually, I make that point moreso with the biomedical detection dogs that may actually do different kinds of diagnostics in the same day, where it’s really important to remind them, what they’re working on in this session, or this trial.
KF: Okay, that makes sense. I think we need to wrap up here. Did you have anything else that you wanted to circle back to or revisit as we’re wrapping up? I think I think this is gonna send a lot of people’s head spinning, but hopefully in a good way. I know I’ve enjoyed this. I normally listen to my podcasts at 1.5 speed, and I think I’d have to listen to this one at normal speed. That’s a good thing. We’re helping people learn, and I really appreciate your time. So if people wanted to follow your lab or continue learning more about you, and the work that you do, where can people find you online?
SG: On Facebook, there’s a page and a group that have roughly the same name. One is the Canid and Reptile Behavior and Olfaction Team, and the other one is Canid and Reptile Behavior and Olfaction Group, I think. One you just have to “Like,” and the other one you have to ask to join. I’m on Facebook myself, you can join although I think I’m at the limit or very close. We are on Instagram, but it’s not very interesting what’s going on there. Same for Twitter. There’s obviously email, and my personal website that you can find by typing my name and Dalhousie University. I think it will come up. So people are welcome to follow us. And hopefully we can get out of this pandemic, because honestly, this has been limiting a lot of our field work in the last year.
KF: Yeah, I think we’re all looking forward to that. I’m also keeping an eye on the border. I’ve been trying to get up to Banff for years now. We’ll see if that happens before I leave Montana. So again, thank you so much for your time. And we’ll probably be in touch soon.
SG: No problem. Thank you for inviting me.
KF: Thank you so much for listening. I hope you enjoyed that conversation with Dr. Gadbois as much as I did. I was taking notes, and my eyes were wide, that whole conversation. He’s just so, so intelligent. And again, I know that some of the stuff on like the D’ and the S+ and that sort of thing can be a little confusing. I’ll drop notes into the show notes to make sure that everyone can look those up and stay present with the conversation.
I hope that you learned a lot, and that you’re feeling inspired to get outside and be a K9 Conservationist in whatever way suits your passions and skillset; maybe play around with some signal detection theory. Again, you can always find those show notes and extra information on this episode at K9conservationists.org