S5E22: Survey Fallacies

Leen: Sorry, I'm trying to get the outline out. It's kind of a situation.

Harjit: You're just trying to figure out what's going on in life.

Leen: Nobody should ever have me trying to figure out their life because it will not work out. How do we start this thing? It's been a hot minute, right?

Harjit: Yeah. "Welcome to 'Bundle of Hers.' This is Leen."

Leen: Hi, everyone. This is Harjit.

Harjit: We're doing a role reversal.

Leen: We are in our virtual studio. We're here to talk about something that I think we've all noticed, we've all had issues with, and we've all tried to at one point or another see if anything results from these things, but that never works out, right? And so I'm here to talk about survey . . . I guess I'm calling this episode "Survey Fallacies," but who knows.

So, throughout medical school, they'll send out surveys, and we'll fill them out. And the surveys will be anywhere from administration trying to get feedback on whether it's a course, or the class culture, or some type of rotation, or their feelings on how the program is going.

And it's kind of interesting, especially when it comes to the surveys that are about diversity initiatives, or education styles, or some conflict that happened in the class. And so we all kind of put our opinion on there, and then we're like, "Did something change or not?"

Harjit: Or did it just go in the dark hole of information?

Leen: Right.

Harjit: Yeah.

Leen: And it's kind of funny because oftentimes we get told, "You guys need to be professional people." I was sitting there thinking about this the other day, and I was like, "How many times have these surveys gone through and we're forced to take them?" And I say forced as in it's required to take them. Sometimes I get emails saying, "You should take these surveys." And every time I'm like, "Yeah, but does it really matter?"

And so I end up taking them and nothing happens from them. I'm like, "Where's the professionalism in this?" And I'm not blaming any certain institution or anything like that.

Harjit: No, not at all.

Leen: I've been talking to friends from all institutions. It's not just medicine. I think this is a big issue.

I remember at least one time there was a survey about the diversity and how we felt about the diversity. We all put, obviously, "No, we're not happy with the diversity representation," but then they came and presented the survey results. And they're like, "Most of the people who took the survey are pretty satisfied with the diversity." I was like, "What the heck?"

Harjit: Yeah. It didn't feel accurate to you.

Leen, you bring up several good points. One is where does the information go? How is it reviewed? How is it used? Is it even used? And I think about those things often as well.

It just made me laugh really hard when you . . . there was this statement that you just said. I need to remember it. L-O-L. It's okay. It's coming to me. It's coming to me. It's coming to me. Oh, yes. So there was a statement where you said that there's kind of this professionalism involved, where if you don't take a survey, sometimes you're "called out" for your being unprofessional. But then that same level of professionalism isn't expected or required of those in power.

And this goes back to our topics of professionalism, how it's usually the person who is in power that decides what the measures of professionalism are, and it's not an equal back and forth relationship.

I particularly remember, when I was a fourth-year medical student, we had collected data on mistreatment and how we should deal with it. And I remember that we collected all the data, analyzed it, and put it into a PowerPoint.

There was a situation where we had reached out to those in power, or who had the ability to impact the way that mistreatment would be reported. And there was no, "Yeah, let's meet," or, "Thanks for putting together this data. Let's see what the next steps are."

So pin that in your mind, because I was like, "Okay, whatever. They must have been busy." This was during pandemic times as well. I think that had just started. So I was like, "Maybe they're working on that." Give people the benefit of the doubt, which a lot of people in oppressed situations do interestingly, is give people the benefit of the doubt, even though no one thinks that we do that, right?

So, anyhow, then what happened was I hear in the next year, they're compiling the same data of "How should we handle mistreatment?" And then I was like, "Man, this is just a cyclical circle of getting information in surveys."

You know what it really made me think about? I think it's a psychological trick. It's that, "We're listening to your needs. We're listening to your thoughts. We're listening to what you want." And the truth is surveys do change things, but I wonder if they change the things that people want changed.

People will just pick one or two things that they really want to work on or focus on and be like, "Oh, the survey says we should do this, and that's why we're improving this for this thing." But it's not generally in line with upholding the intent of what the survey was in the first place.

Leen: Yeah, absolutely. And it's funny because I think regardless of whether they uphold it, whether they actually change it . . . I guess one thing to think about is those surveys are given does not mean surveys are an actual quantitative form of data that is truly objective, right?

You're surveying a population that might not see a problem. But any time you have a population that might not see a problem, the minority population will always not be included in that survey. You know what I mean? They won't be included as they need represented.

We have to go back to think, "Are these surveys really the efficient way, the most appropriate way of getting us close to quantifying issues in order to say that this is what we have to change?"

How many times have we been presented with survey results from our class, and there'll be like, "Ninety percent of your class says this is not an issue," and you're like, "How is this not an issue?" And something would be so blatantly obvious.

I still remember one survey where they said . . . it was, "About 90% of your class does not think diversity is an issue in our class," and I was looking around our class and I was like, "We're all one big majority here."

Harjit: Right. It kind of comes back to one of our episodes where we talked about decolonizing research. Where are we getting those survey results from? If a class is made up of the majority, they obviously don't have an issue of minority issues. So it's being careful to really understand and be critical of the survey results that you're getting as well.

Leen: And I think when you now further extrapolate it into the next stage of medical training, you get into residency. And now surveys are a lot more critical, especially for the ACGME, the GME office. These are the accreditation offices that report if a residency is having any violations, and they can put residencies under probation, which ultimately turns into this whole mess.

We prepare almost months to a year in advance saying, "Okay, we're ready to take this survey." And I'm talking about these are the big surveys. Obviously there are many surveys within residencies that I think are also part of this.

But ultimately, I was reading an article. I guess the title of the article was, "Why would residents lie about violation occurrences on ACGME surveys?" And this was taken back in 2010. But interestingly enough, it starts out in medical school training and then extrapolates to residency training. And it might extrapolate further to that, right?

At this point, you don't want to say anything false because that's going to turn into consequences on your residency being under probation, which then can cancel your residency, which then puts you in a bad place, right?

So, again, it falls back into that whole "this is a problematic issue," and we continue to go through the cycle, right?

Harjit: Leen, I think you bring up a really good point. I think surveys, like you said, are very useful, especially when it comes to residency, to make sure everything is being conducted in a proper way. And I think the biggest thing we think about is, like you said, work hour violations. But then there's always a worry that this survey may impact our training and if we will be accredited or not, right? So there's a direct negative consequence for true answers.

Okay, fine. Let's say a residency is violating something. So what are they going to do? Shut down the residency? Yeah, I guess they get some time to try to fix the problem. But shut down the residency? And then what's going to happen to those people who are going to be the residents there? What kind of consequence is that?

So I think that's another thing we should be wary about when we think about surveys, is can people truthfully tell their opinion, even if it's anonymous, because it impacts them directly?

Leen: I don't have data on this, but I almost wonder for residencies where there are a lot of IMG students, which are international medical graduates . . . As an international medical graduate and the people I know who are, they will grasp at any residency that will take them just because it's such a tough process to get into the United States for residency.

So this article is saying that back in 2011, like 60% of residents indicated that they falsely report duty hours to appear in compliance. Is that even more if you go to residencies with high IMGs? Especially because they're worried about visa status. They're worried about completing their residency to be able to start their medical careers in the United States.

In a sense, you're almost teaching them that medical professionalism includes lying on these surveys here and throughout, because that's the only way you're going to keep a job in the United States. What kind of work environment is that fostering?

I wonder about this log. Again, I don't have data on it. But I just wonder . . . They keep throwing surveys at us. We all know we have to say this right answer. And it extends all the way back to medical school for many various things, group basis to larger basis. And I just wonder how much of that is truly for our benefit versus the numbers to make us look viable and without violation.

Harjit: I think that's a really good way to think about it, Leen, because I think about the same things from surveys. I think the intent initially of surveys is to cause improvement, right? Find something, improve, and have it be a cyclical thing.

But often what happens is when we have so many surveys given to us rather than even a sit-down conversation . . . I know people want to be anonymous and stuff. It's like what impact is that really having, right? All this information that we're giving is still being utilized by a system that is oppressive, which we've talked about.

And so it's like, yes, there are data and figures, but is it being, in a sense, manipulated? Manipulated is a strong word, but really, is that data then being used for the powers that be for their benefit?

Leen: And I think they get good information. If they truly just make it sound like there's no going to be such a detrimental consequence, if we take this information, and I want you to be truly honest about it, this is how you fix problems, right? You identify the problem, and you fix it. But I don't know how it's gotten so skewed into this detrimental career-ending kind of business.

Harjit: Yeah.

Leen: I'm like, "Yeah, you guys get such good data. Why do we have to . . ." Or, "You have the opportunity to get such good data." I mean, every resident takes these surveys, right?

Harjit: Yeah, but why can the consequences of them be detrimental to us?

Leen: That's just like, for instance, those big surveys we take in residency. There are obviously little surveys we do among our classmates. "How do you think this person is performing?" Feedback was a huge thing, right? Feedback was a huge thing in medical school. They were teaching us how to accept feedback, how to give feedback. And I'm just sitting here and I'm like, "Why? What was the purpose of that?" That's just more individual growth rather than truly collective growth of a facility or institution or whatever we are in.

This is a topic that really, in a sense, enrages me because it's kind of sad. We kind of just say, "Hey, we're just going to deal with it while we're in residency, live under this really stressful environment, get through it," and then be like, "Oh, okay, we're on the other side. Now, everybody will get through it." What are we changing?

Harjit: But I think that's the whole thing. The separation between different levels of training is so intentional, because it really is there to not build a collective and have a collective voice and a collective thought. We just end up doing the same work over and over and over again, and it just takes that much longer to do.

Like I said in my initial example of the survey about mistreatment, we had given and we had collected. Nothing was done with it, and it was then done again. And it might be done again, and then again. That's kind of the cycle that it goes through because it takes a lot of effort to put things into place versus collecting people's thoughts, right?

And I think that's where surveys can be utilized in a very useful way. But in reality, a lot of surveys are then used to change an action or they have consequences that sometimes aren't as helpful as they should have been.

Leen: So what do you think? I've also talked to some residents, and they say, "You know what? No, I'm going to be truly honest on these surveys, and I don't care what people tell me I should do."

Harjit: I kind of get that, because isn't that the purpose of a survey, is to be truthfully honest? That's the way it's presented to the world, right? And that's what surveys . . . especially if they're anonymous surveys.

But then it's also, like us, we're always thinking about things further. And this is really, again, a topic about the system that is very capitalistic.

It just reminds me of a moment when I was an intern in May, and my mental health wasn't doing great and I needed a mental health day, right? I needed a day off. But I was like, "If I take this day off, then someone else will have to work in my spot. And these people are already working so much. Another intern will have to do the work that I'm not doing."

So it just reminds me of the surveys because I want to be truthful, but then a part of me is like, "Ugh, if I'm truthful, then it could hurt someone more than it would even hurt me." I think that's where the dilemma really comes of surveys.

Surveys are so interesting because there's also this whole concept of being anonymous, which can be really scary, right? I'm just thinking about the time where people say really nasty, mean stuff on surveys because it's anonymous, and it makes people feel that they can just say things in a way that is just so hurtful and so wrong. That's another thing that it reminds me of.

Leen: Yeah, it's like the windshield effect. It's kind of like when you're behind a windshield when you're driving and you're just like, "I can do all the road rage I want," or the Facebook effect.

Harjit: I think I'm a fan of the windshield effect.

Leen: I always think that. I'm like, "When you put someone behind the wheel and all of a sudden, they think they're the coolest thing on the road and they start cutting off everybody in road rage. Yeah, you take that person outside of the car and then you'll see how well they . . . It's a whole different person."

Harjit: Yeah, truly. I think that it provides some sort of security for people to just be completely honest with their thoughts, which is scary at times because you're like, "Wow, there are people that actually think about these things."

Leen: If you want to think of a pro for having construed and/or falsified information on surveys, in a sense, is if you have program leadership that truly wants to represent your residency and/or medical school or whatever institution of healthcare you're in, one thing I was thinking is . . . Let's say you know that there's not good amount of representation within our class, right? But because of us wanting to maintain and stay out of violations, maybe the students are putting that there is great representation.

I feel like if leadership has that insight into these findings, they can say, "Well, we're submitting that because obviously that's kind of out of our hands. We've got to do that." But let's go back and actually make that true.

It's kind of hard because that takes a lot of insight from administration, a lot of insight and down-to-earthness and communication and connection between our program leaderships.

I don't know. I haven't experienced every program out there, but I think that that's something to keep in mind. If I saw that my residents were putting information on there that I'm like, "Oh, is that really true?" then I would try to work so hard to make that actually true.

Harjit: That's something we could wish for. I don't know if it's always accurate, but I understand where you're coming from, Leen.

Leen: And I think that's the point that I want to reiterate. I guess, at the end of all this, I think surveys will continue to be sent out obviously. And whatever profession you go into, you're going to encounter that falsified, hyped-up sense of whatever field you're in. And I think it's just important to recognize that these are not truly quantitative measures and/or accurate measures of what's going on around you. It really takes everybody to step up and really help each other out with these things.

Harjit: I think that's true for everyone. And I would still argue that being truthful on surveys is the point and is important. If not anything, it allows us to vent and let out our feelings.

But yeah, I do think about this often and how much effort I put into surveys because of, I guess, the feedback I've gotten, which is things don't really change. But that's also something that I don't want to continue to think. I think it takes all of us to kind of give our thoughts hopefully loud and clear so people know what's up.

Leen: I agree. I think that is the right message. Continue to be truthful, continue to push for these things, and change will happen.

And with that, thank you for listening in to this episode on "Bundle of Hers." We appreciate our listeners. We hope you guys are staying well out there in this pandemic/era of madness. Please continue to listen to us and follow us on wherever you get your podcasts. And we are on Instagram @bundleofhers.

Thanks, everyone. Bye. That was a really lame bye. Bye.

Harjit: Your "bye-bye" used to be more interesting.