Damien:
Welcome to the Data Science Conversations podcast. My name is Damian Deahan, and I’m here with my co-host, Philip Deesinger. Hi, Philip, how’s it going?
Philipp:
Hi Damien, it’s going well and looking forward to the conversation with Julia.
Damien:
Great. So today, a very interesting topic. We are talking AI ethics, responsible AI. So we’ll be touching regulation, education, and the need to be critical about the current AI hype cycle. Our expert guest from NYU is Julia Stojanovic. Julia, welcome back to the show.
Julia Stoyanovich:
Great to be here.
Damien:
Excellent. So just to remind everyone, Julia is an in, sorry, say that again, just to remind everyone, Julia is an Institute associate professor at NYU in both the Tandon School of Engineering and the Center for Data Science. In addition, she is director for the Center for Responsible AI also at NYU. Her research focuses on responsible data management diversity, transparency, and data protection in all stages of the data science lifecycle. Julia has been teaching and developing courses on responsible data science at NYU and is the co-creator of an award-winning comic book series on the topic. She holds a master’s and PhD in computer science from Columbia and a bachelor’s in computer science and maths and statistics from the University of Massachusetts at Amherst. So I think the best place for us to start Julia is perhaps you could give us your definition of what responsible AI is.
Julia Stoyanovich:
That’s a great question to ask and a very difficult question to answer, so I will take it sort of step by step and try to unpack different components of what we may mean by AI and responsibility. So as Damian, as you mentioned, I direct the Center for Responsible AI at New York University and our goal is to make responsible AI synonymous with AI in a not too distant future. And so what do I mean by that? Just colloquially, I would say that responsible AI is the socially sustainable design, development, and use of technology. Meaning that we want to build and use AI systems in ways that make things better for all or for most of us, that help us reach our societal goals of being able to cure diseases that we haven’t been able to cure before, being able to distribute resources. more equitably, more justly in society, and also of course making money for people, but making it so that economic opportunity once again is distributed in ways that are equitable, and where it’s not the case that we exacerbate the disparities in access to opportunity that already exist and are rampant in our society. So when we talk about responsible AI. One component of this term is AI ethics. So what is AI ethics? Ethics, again, perhaps informally, is the study of what is morally good and bad and morally right and wrong. And then AI ethics is usually used to refer to the embedding of these moral values and principles of AI systems. And much of this conversation usually centers around the unintended consequences of AI, of some mistakes that an AI system may make or will suggest something to a human and then a human makes that mistake. And it also concerns things like bias in AI that many of us have heard about. We also want to think about arbitrariness in decisions as a kind of a mistake, perhaps. that an AI system may make. And this all falls under this AI ethics agenda. So as a data scientist and a computer scientist myself, I prefer to think about things not in the abstract, but rather using some concrete examples to make things a little bit more tangible, a little bit clearer. So let’s look at one specific example that I think exhibits some of the hallmarks of responsible AI. And this example comes from the domain of medical imaging, where like in many other scientific domains and commercial domains, AI is all the rage today and will likely continue to be. We are starting to use cutting edge AI tools in clinical practice to improve diagnosis and prognosis capabilities. So one specific example that I like to use is a recent study that was done by researchers from NYU together with folks from Facebook AI, where they developed a technology called Fast MRI. So what is Fast MRI? It’s a way to generate magnetic resonance imaging MRI scans that allows us to use a lot more data. a lot less data, sorry, and that works much, much faster because this data that is generated is semi-synthetic. So we start with some MRI scan of an individual, a quick one, and then we fill in the gaps, so to say, with the help of the SAI technology. And then it has been shown that these semi-synthetic MRI scans are diagnostically interchangeable with full regular MRI. And this MRI machines are in short supply in many locations, and so this allows you to improve throughput to just get more people through, get more people diagnosed. And it also can make a huge difference for somebody who is claustrophobic, right? And they don’t want to stay inside an MRI machine for longer than is absolutely necessary. And so here, what’s wonderful really is that we’re able to use the hardware, the software. the prediction capabilities to generate this data and to assist clinicians then in diagnosing things like cancers, different types of cancers, based on this technology where there is a clear need for improvement. We do need more access to MRI machines. We need better ways to diagnose and do medical prognosis where we can also validate the predictions that are being made with the help of these fast MRI scans, you can get a bunch of clinicians together, they can look and then they can determine whether the person in fact does or does not have the kind of a disease that they are being diagnosed with. And we are able to do this given the current state of the art in hardware and in software. And also very importantly here, what we have is an environment in which machines cooperate productively with clinicians who are well-trained. both in the domain in which they work, they understand what it means for a person to have cancer, and where they understand that the responsibility for the decisions, and also for any mistakes that they may make, even if the decisions are AI assisted, are still with them. Right? So all of these clinicians, they have been trained in medical ethics, they understand that the decision is theirs, and that the responsibility for the decision rests with them. It’s not with the machine. And so this… together essentially gives us an environment in which AI is being used responsibly. It’s helping us solve a problem, an actual problem. We can check that it works and we have a responsible human decision maker in the mix. And this I like to contrast with some other examples where the use of AI is less than responsible. And here there are lots of things that can go wrong. from us trying to essentially build machines that are creating a horoscope, a kind of a self-fulfilling prophecy rather than addressing an actual need and where we can validate the predictions and the solutions that the AI suggests to environments where we’re just asking machines to make predictions that are morally questionable, like predict whether somebody will commit a crime in the future based on how others like them have behaved. or where we put AI in an environment where they interact with humans who have not been taught how to interact with these machines, and then they just take the suggestions on face and they don’t take responsibility for any mistakes. So let me stop here. I know I’ve been speaking for a while and maybe have this as more of a dialogue.
Philipp:
Thanks, those are great examples. You gave this example of the MRI machine, basically, which is in the healthcare sector. That’s already a very regulated space because you’re dealing with patients and vulnerable people and so on. What role will or can global regulations play to move to a space where we really have no difference anymore between responsible AI and AI, like the vision that you pointed out in the beginning?
Julia Stoyanovich:
Yeah, this is again a very, very difficult question. I don’t know whether we are prepared to regulate the use of AI globally. And we are and have been trying to do this in a number of very concrete domains. So for example, little autonomous weapons is a very, very scary domain where AI is used, right? These weapons are autonomous. They decide what is a target. and what is a civilian. And even in this domain, it has been very, very difficult. The United Nations has of course been playing a tremendous part in this. It has been very difficult to come to a global worldwide agreement about how we can control these tools. And these days AI is being used as we all know in a variety of sectors, right? With a variety of impacts to health and to economic opportunity and to… people surviving or dying in a battlefield. And so because of this variety, I think that it’s gonna be tough to come up with globally accepted ways to regulate. In part, this is because of what I said about ethics early on and that is that it’s about values and beliefs. And we don’t really agree on a universal set of ethics or moral norms or values. But this is not to say that we shouldn’t try. I think that there are some high level insights that we all share and some high level goals and these are that we should keep our humanity in our interactions with AI, that we should make sure that it’s people who are deciding what the future will look like and not machines somehow. Yeah.
Philipp:
But in your expertise or from your perspective, is this a problem that can be solved through regulations?
Julia Stoyanovich:
Regulation is a very valuable tool in our responsible AI toolkit. It’s not the only thing I think that we will rely on, but regulation globally as well as locally, as well as you know oversight within companies and oversight by vendors themselves, as well as awareness of the people being impacted by algorithmic decisions that are made with the help of AI. These are all very, very important tools in the way in which we control these systems. And you also, you mentioned, Philippe, that we already know how to regulate in the medical domain. AI, of course,
Philipp:
Mm-hmm.
Julia Stoyanovich:
presents
Philipp:
Yep.
Julia Stoyanovich:
new challenges even there, right? So even if we have regulation in place, we need to be careful about how we think about the impact that AI is having. So there’s also kind of a negative example in the medical domain that is well known. and that was surfaced by Obermeier and co-authors in 2019, where they showed that predictive analytics that are used in many, many hospitals throughout the United States, and that estimate how likely somebody is to be very ill, that they show racial bias. And this is because of the
Philipp:
Hmm.
Julia Stoyanovich:
way that the predictive problem has been set up essentially, is that we don’t really… for the bulk of the population up front. We don’t know how ill they are. We want to predict this. One of the proxies that was being used to predict how ill somebody is, is how much money have we been spending on healthcare for them up to now, right? And because we have a biased system in the US, where people from lower income communities, and these are very often people who are African-American or Hispanic have less access to medical care, then healthcare spending is going to be less for them than for somebody from a more affluent social group, but who is equally as ill, right? And so by using these biased proxies, we end up making predictions and kind of propelling the past into the future in the way that healthcare… access suffers from these disparities. So even in this domain, I think we need to be very careful about how we use data, how we collect it, what it encodes, and what are some of the harms that
Philipp:
Mm-hmm.
Julia Stoyanovich:
irresponsible use of data, inadvertently irresponsible use of data, may bring to this domain.
Philipp:
Yeah, that makes a lot of sense. From your perspective, what’s causing these bias boxes or biased features? Is it limitations of data? And also, what role would data models play for AI?
Julia Stoyanovich:
Um, so like with everything else in this space, things are very complicated, right? Certainly, uh, the fact that data may or may not represent the world faithfully. And we call this bias in the data or one way in which data can be biased. This certainly contributes,
Philipp:
Mm-hmm.
Julia Stoyanovich:
uh, very strongly to predictions of, of these machines that are trained on data being
Philipp:
Mm-hmm.
Julia Stoyanovich:
biased, but it’s not the only signal. Right? So I like to s- Think about bias in the data by invoking this metaphor that data is a mirror reflection of the world. And this
Philipp:
Hehehe
Julia Stoyanovich:
mirror may be a really good, precise mirror that reflects the world correctly. And I’m using air quotes here. You can see this on the recording. But even if we reflect the world perfectly correctly in the data, it’s still a reflection of the world such as it is today. Right? So if our world is not right. some way, maybe it’s gender biased or racially biased or it has some other distortions built in, then the data will reflect this and it will legitimize it essentially, right? Because it’s just correct,
Philipp:
I
Julia Stoyanovich:
it’s
Philipp:
sense
Julia Stoyanovich:
objective.
Philipp:
it.
Julia Stoyanovich:
And so we need to also think about whether given the current state of the world and given the current state of our data collection and data processing capabilities, we in fact can do things better than… replaying the past into the future with the help of these predictions.
Philipp:
Makes a lot of sense. And when we’re talking about AI, one question that’s on my mind is, do you differentiate between different types of AI, like a simple regression model versus GPT, LLM, or so, from the perspective of responsible AI, or would it all fall under the same umbrella? In terms of the
Julia Stoyanovich:
So here as an academic I choose to take kind of an extreme point of view and of course in the real world things may be a bit more nuanced, but just to make a point, right? I actually think that it doesn’t matter what sort of a
Philipp:
Mm-hmm.
Julia Stoyanovich:
technology lives inside that box, that we
Philipp:
Alright.
Julia Stoyanovich:
sometimes call black boxes or opaque boxes. It could be a very complex model or it could be a very simple one. So I have spent the bulk of my career studying these very simple gadgets called score-based rankers, where you have a data set of items, let’s say these are people applying for jobs, and you compute a score for each item using a formula that you know up front. Some combination
Philipp:
Mm-hmm.
Julia Stoyanovich:
of standardized test scores, for example, and some number of years of experience. And then you sort everybody on that score. And even in that case, right, by taking, let’s say, the top 10% of the people from that sorted or ranked list to then invite for in-person job interviews, you’re introducing a lot of opacity. You as a decision-maker are not going to immediately understand what is the impact of the scoring formula on whom you choose and whom you forego. So for example, let’s say that we’re talking about college admissions and you are using some combination, let’s say, half of your score is made up of the grade point average of the student from high school, and half of the score is made up of their standardized score, like the SAT in the US that we use. But if this is a very selective college, then applicants self-select, and only those with the very top SAT scores will apply. And so, although your SAT score component has an equal weight in your formula, it’s gonna have far less importance. because everybody’s tied on that component of the score, right? So this already shows you that even seemingly simple devices can have side effects or direct effects that are hard for people to predict. And so rather than worrying about what lives inside that black box, whether it’s a generative AI model, or it’s a simple, you know, rule-based AI or a scoring formula, we should worry about the impacts that these devices have. So what is the domain in which we use them? Can we tell what they do rather than how they work? And we have, of course, the scientific methods at our disposal to help us deal with and unpack how black boxes work, right? So we can feed it some inputs and see what happens at the output. Are there any changes, for example, if I change nothing except an applicant’s gender or ethnicity? If the output changes, then I can have a hunch that there is something going on here that is perhaps something that I should be looking into more closely. So I wouldn’t worry about whether we are dealing with a very complex machine or a seemingly simple machine. I would worry more about what these machines do, whether they work, how they work, how we measure their performance and what are the stakes of a mistake and how can we correct the mistakes.
Philipp:
Yeah. Did you see a boost or an increased interest in regulatory questions and responsible AI with the rising now of generative AI?
Julia Stoyanovich:
Yes, absolutely. So it’s a blessing and a curse, right, that there’s now this hype
Philipp:
Hmm.
Julia Stoyanovich:
around generative AI. The blessing is, of course, that almost everybody is paying attention. So worldwide, right, in the European Union and the United States, where I live, we have politicians speaking about the needs to control the adverse impacts or the risks of harm that the use of generative AI can bring. But together with that, everybody’s just paying attention to AI more generally and to how we might oversee, regulate,
Philipp:
Mm-hmm.
Julia Stoyanovich:
bring more responsibility into our deployment of these systems. So it’s a good thing in that sense. But, but of course, hype is also very tiring and it’s also harmful in that we are
Philipp:
Yeah.
Julia Stoyanovich:
paying a lot of attention to things that may or may not matter immediately. And so we shouldn’t forget that we already. are and have been using really for decades, AI tools in very impactful domains. And these are not going
Philipp:
Mm.
Julia Stoyanovich:
to be for the most part fancy tools like large language models. They’re going to be much simpler tools like rule-based systems, score-based rankers, linear regression models. And these are being used in, you know, in hiring and employment and then credit and lending and then access to housing. And we shouldn’t forget that if the tool or the AI technology is simpler, that there can still be and have been documented tremendous harms that the use of these systems can bring. And we should definitely regulate in a way that looks at who is impacted and what are the impacts rather than by regulating a particular kind of technology that sits inside the box.
Philipp:
Makes a lot of sense. Coming back to the topic a little bit of regulations. So what’s happening in that space at the moment? So I think everybody is looking at the EU working on the AI Act. They have been publishing small pieces of information about the ideas, how to approach it, and how to categorize risks and so on. As you said, already, the US is also working. on something, China is working on something. What would be your best guess? How regulations will come to life? And who’s working on that, or going in the right direction from your perspective at the moment?
Julia Stoyanovich:
I think that we just need to try to regulate in this space. We shouldn’t wait until we come to a moment
Philipp:
Mm-hmm.
Julia Stoyanovich:
where we’re absolutely sure that this is the right perfect way to put regulation into place
Philipp:
Mm-hmm.
Julia Stoyanovich:
and then everybody is going to agree that this is the way that we should govern the use of these systems. That will never happen. It’s very hard to reach
Philipp:
Mm-hmm.
Julia Stoyanovich:
consensus. So I think that we should try. We should talk less and do more. And I’m really glad that the European Union has been leading the way in this, starting with the general data protection regulation that has been extremely impactful, and for which we in the United States still don’t have an analogue, and this is really problematic for us in the US. And I’m really glad that the AI Act in the European Union is moving forward. So again, in the US we have been… hearing lots of people speak about this and there are roadmaps and blueprints etc. and these are very valuable. But we are yet to see regulation at the federal level in the United States. And so we are lagging behind. And in the US, of course, we have to some extent a decentralized system. So there is also a lot of an opportunity in the United States to regulate at the level of cities and states. And there there’s quite a bit of opportunity. And this is something that I have a more immediate experience with than sort of federal level or national level regulatory approaches.
Philipp:
Yeah, I think here in the EU, it’s a similar discussion. There’s also a perception that the EU is falling behind when it comes to AI technologies, and that seems to feed a little bit into the discussions about regulations. So they want to be practical, they want to obviously catch up with AI technologies and so on, but at the same time, regulate it and make it safe. Do you, from your research, see any evidence between the effort in regulations or the strictness in regulations maybe and suppression of technological development or innovation in countries or geographies?
Julia Stoyanovich:
So I’ve not actually done any research specifically to look at the impact of regulation
Philipp:
Mm-hmm.
Julia Stoyanovich:
on innovation. And I think it’s hard to do this research really, because we don’t have examples of two places that are comparable in every way, except that one has stronger regulatory regimes than another. I don’t believe that regulation stifles innovation in any way at all. So when I spoke about what responsible AI is, to me, it’s… socially sustainable development of AI. And for us to reach social sustainability, of course we need to make it so that when we deploy a tool that it doesn’t break society further, right? Because then you have to recover from the ill impacts of that. So to me, really deploying something and then seeing how it plays out is not at all a sustainable way to operate a society. And it only
Philipp:
Mm-hmm.
Julia Stoyanovich:
advantages a very select few. And it’s the people who… are releasing the technology and stands to benefit from it financially immediately now. But in the long run, this is going to hurt us and it is already hurting us. So I personally see no alternative here. We do need to, considering the success that this technology has had, we do need to think about regulation at the same time as we think about large scale adoption of things like large
Philipp:
Mm-hmm.
Julia Stoyanovich:
language models. Excuse me.
Philipp:
Yeah, that makes a lot of sense. Maybe one more question in that regard. So on the one hand side, we are working on responsible concepts, AI ethics, measurement frameworks, understanding what’s happening inside the black box and so on. There’s a lot of research going into that, a lot of smart people working in that field like yourself. On the other side, it seems also that the technology itself is vastly. also rapidly developing, gaining new capabilities, gaining more access to reasoning and so on. So the risks might also be increasing over time. So there is kind of like a fight between those two or a balance between those two that needs to be hit. In which direction is it going from your perspective at the moment? Are we catching up? And we will be able to regulate it. a long-term kind of positive interaction with AI? Or is there a certain risk that AI is just developing so rapidly that we will not catch up and there is a real problem about sustainability?
Julia Stoyanovich:
So I am an engineer, right? I’m not a philosopher
Philipp:
Mm-hmm.
Julia Stoyanovich:
or somebody whose job it is to predict the future. So engineers predict the future by making it. And I think that’s actually the only way somehow that
Philipp:
I
Julia Stoyanovich:
you
Philipp:
like
Julia Stoyanovich:
can
Philipp:
that.
Julia Stoyanovich:
predict it. So my prediction is that more and more engineers are going to understand that it’s our responsibility to make sure…
Philipp:
Mm-hmm.
Julia Stoyanovich:
that we build systems that we are proud of and that we can stand behind. And that we engineers, but also others in society, of course, should take control and participate in making decisions about what we think we should be building and using and what we should not be building and using. And so… When we talk about responsible AI, that term itself is also a bit misleading, right? Like everything with AI, frankly, right? Artificial intelligence that’s misleading. There’s no
Philipp:
Yeah.
Julia Stoyanovich:
such thing. I’m sorry. But so, so responsible AI doesn’t mean that the AI is responsible. On the contrary, right? It’s the people who are responsible for the development and use of AI. And one of the things that’s… particularly dangerous with this AI hype that we’re experiencing right now with generative AI, is that there is this push from some folks who are very vocal to say that AI is about to take over, that it has a mind of its own, that whatever
Philipp:
Mm-hmm.
Julia Stoyanovich:
harms before us socially as a result of this AI now having accelerated, it’s the AI’s responsibility. And this is a really problematic narrative, because it absolves those who stand to benefit, financially and otherwise, from the deployment of these systems of any responsibility for the mistakes. And we cannot allow that
Philipp:
Hmm.
Julia Stoyanovich:
to pass. And this is a very simple, naive kind
Philipp:
Mm-hmm.
Julia Stoyanovich:
of an engineering point of view, right, on what’s going on. But, you know, to invoke William of Ockham, Ockham’s razor, if something is simple and captures the phenomenon, it’s likely correct. So I think that this is really a point in history where we’re witnessing folks fueling this AI hype for personal benefits, so that they absolve themselves of the responsibility and yet reap all the benefits. Generally, to me, responsible AI is about human agency. It’s about people at every level taking responsibility
Philipp:
Mm-hmm.
Julia Stoyanovich:
for what we do professionally, for how we’re impacted personally, for how we use AI in our professional… lives, like a doctor using an AI to help them diagnose whether somebody has cancer, right? We all need to step up and say we are the people, we are in control here. Agency, the agency is ours, the responsibility is ours. And this is again one area in which generative AI is presenting us challenges, because a lot of the kind of impetus for these tools to exist is to seem indistinguishable from what a human would have produced. create art that seems like a human could have made, or to respond to your question in natural language in a way that sounds like a human could have uttered that, right? So this anthropomorphization of AI, this is very problematic, because it
Philipp:
Mm.
Julia Stoyanovich:
takes us away from this goal of staying in control and into somehow giving up agency, giving up control to machines. And this we should resist as much
Philipp:
Now.
Julia Stoyanovich:
as possible.
Philipp:
I think that makes a lot of sense. You’re talking about who has the agency and of course, it’s the humans building the systems in the first place. So the responsibility would also lie with them. But what do you say when people counter that genuine isosomes can start writing their own code now and potentially maybe start self-improving at some point in the future?
Julia Stoyanovich:
I don’t believe that that’s the case. And furthermore, I mean, we should decide whether we are okay with this. If generative AI writing
Philipp:
Mm-hmm.
Julia Stoyanovich:
code is something that we think we can benefit from, where they can be used to automate some tasks, let’s say of software testing, something that is maybe mundane to some extent and humans get bored and are not very thorough, then certainly we can allow this. this use, right? But
Philipp:
Mm-hmm.
Julia Stoyanovich:
whenever we ask an AI to do something, we need to be able to measure whether whatever it has done is correct and good and adheres to the requirements that we have set out. And if we can’t do that, then we cannot take an AI’s word on face that it worked, right? One example
Philipp:
Mm-hmm.
Julia Stoyanovich:
here that I like to use is in a domain that I’ve been very interested in, and this is the use of AI in hiring and employment. So
Philipp:
Mm-hmm.
Julia Stoyanovich:
there are these tools that have been developed, several of them exist, that claim to construct a personality profile of a job applicant based on their resume. So you give a resume to an AI and it says you are 80% conscientious, 20% neurotic, and 30% dominant. Right? And you just step back and as a human you think, is there any way to validate this? Like if I as a person made such a prediction, could I actually check if I was correct? And if the answer is no, then we should not be using machines to make these predictions because they’re engineering artifacts. And if we can’t tell that they work, then they don’t work.
Philipp:
Mm-hmm. Yeah, it makes a lot of sense. Maybe one last question. So what’s your opinion on the release of JetGPT to a mass audience? Is that something that happened too early, or was it good because it created a lot of awareness? I mean, it obviously kind of triggered this development race between Microsoft and Google now. There were attempts to kind of pause. GPT development or GNI development for a while, and so on. But ultimately, we are in this race now. Was this too early in terms of the maturity of the technology? Or do we not have an opinion on that at all? Or what’s
Julia Stoyanovich:
I have
Philipp:
your stand
Julia Stoyanovich:
lots of
Philipp:
on that?
Julia Stoyanovich:
opinions on this. I definitely think that
Philipp:
Yeah.
Julia Stoyanovich:
it’s too risky and
Philipp:
Yeah.
Julia Stoyanovich:
that it’s extremely irresponsible to have unleashed this technology
Philipp:
Mm-hmm.
Julia Stoyanovich:
without giving us any meaningful way to control how the data travels, where the data goes, to give us any meaningful way to understand where this technology can be safely used. There are also tremendous labor issues that go along with the release of the technology, that essentially artificial intelligence of course is based on everyday intelligence of people and in this case these were people who were, who and continue to receive very low pay for their work with just you know the labor aspects of this technology are tremendous. Sustainability. environmental sustainability is very problematic of these tools. So I personally think that the harms, the actual harms to individuals and to society and the risks of further harm due to data protection violations, due to bias, due to again anthropomorphization of these tools, right, where they respond to you as if it’s a human on the other side and this is deliberate, they far outweigh the benefits. But then the question is benefits to whom, right? So if we talk about financial benefits of the company having released this technology then to them, that’s what matters. And this is why we need regulation, right? So that it’s not just the select few who benefit. So I personally don’t do any work that involves because I just don’t think that we should be feeding into this hype and we should be giving our data and I don’t think we should be studying these devices. Those who produce them need to spend resources, time, money on figuring out how to control them before they can go into even broader use.
Damien:
Hmm. Good. So, yeah, I think what I’m hearing, Julia, and I agree with you, is that the whole talk about the existential risk of AI is definitely overemphasized, overhyped. I’m with Jan LeCun on that. I think it’s overstated. But what are the bigger risks? We’ve talked about some of them, bias. hiring, et cetera. What are the other more pressing risks that you see with the way that the current AI systems are being deployed right across our technology infrastructure?
Julia Stoyanovich:
They are the same ones that you listed, right? I mean, so one of them is that, of course, decisions will be made with the help of these tools by people who do not question whether the predictions of the tools are correct in any sense. So many of the decisions being made will be arbitrary, and this is even beyond bias, or an orthogonal dimension rather. Being able to control the data that goes into these systems is something that we… cannot yet do in the European Union because there is the GDPR, people are much more in control of the data that goes into these systems. We in the United States and elsewhere in the world are not as fortunate, right? So how our data is used, whether we’re comfortable with our data being used in this way is also problematic. And one of the angles on this is that, you know, there may be harms due to privacy violations, but another is just that even without a harms-based conversation. People have rights, right? We have rights to privacy. We have rights to agency, to being in charge, both of our own data and existence and also of the world that society functions. So at the high level, it’s really just that we’re inserting a technology that we don’t yet really know how to control, but to be more concrete, and we were at a very high level in our conversation today, higher than I usually like. we need to think domain by domain, example by example, who are the people who benefit, who are the people who are harmed, and who is in a position to mitigate the harms. And it’s the same story with every technology that we’ve been experiencing throughout humanity, right? The industrial revolution also left out some and benefited some others, and so we need to make sure that we are, you know, acting and using technology in ways that are more equitable this time. Hopefully we’re more mature than we were 100 years ago.
Philipp:
And Julia, we talked about OpenAI already. Do you have a perspective on their super alignment initiative?
Julia Stoyanovich:
No, I don’t have a perspective on their super alignment initiative, and I’m not a fan of the term alignment in general, because it
Philipp:
Mm-hmm.
Julia Stoyanovich:
essentially is… Usually the message there is that somehow we’re able to just automate moral and value-based reasoning in machines, and I don’t believe that is possible, nor should it be the goal.
Philipp:
Hmm.
Julia Stoyanovich:
So it’s contrary to the agency conversation that… Yeah.
Philipp:
So in your view, it’s more an engineering responsibility or engineering problem in the first place.
Julia Stoyanovich:
what is.
Philipp:
the making AI systems like GEN.AI systems responsible and following or not causing any harm and so on. So
Julia Stoyanovich:
I don’t
Philipp:
if
Julia Stoyanovich:
think
Philipp:
I understand
Julia Stoyanovich:
we
Philipp:
you
Julia Stoyanovich:
can
Philipp:
correctly,
Julia Stoyanovich:
automate
Philipp:
yeah.
Julia Stoyanovich:
ethics. I don’t think we can automate responsibility. I don’t think alignment in the way that it’s being discussed right now is a productive way forward
Philipp:
Hmm.
Julia Stoyanovich:
because it essentially borders on this conversation about algorithmic morality that folks like Elon Musk and others
Philipp:
Mm-hmm.
Julia Stoyanovich:
have been pushing where essentially it’s just the simplest, least nuanced version of utilitarianism that we end up. trying to embed, like how many people die, how many people are safer. We add these numbers up, we subtract some, and then based on that, we decide whether or not it’s safe to deploy self-driving cars, for example. And I think that the use of AI is way too complex, way too context dependent
Philipp:
Mm-hmm.
Julia Stoyanovich:
for us to pretend that we can automate ethics and responsibility and morality. So I think that that’s a
Philipp:
Mm-hmm.
Julia Stoyanovich:
dead end. There are… For technologists like myself, I think the main task is to figure out where technology can be helpful and where it has its limits. Technology cannot solve all problems that our society has presented itself with over
Philipp:
Mm-hmm.
Julia Stoyanovich:
millennia. There’s no way for you to debias a dataset and then proclaim that now you are hiring with no bias or lending with no bias. This is hubris. So we need people in the mix, people making decisions, taking responsibility for decisions throughout. There’s no way that we will align technology to our values, push a button and then say, okay, the world is just.
Philipp:
Makes sense, yeah, I understand.
Damien:
So what you said earlier, responsible AI is about human agency. So what can practicing data science leaders and data scientists who are developing AI systems for internal use in their company or for their customers, what should they be thinking about to make sure that it’s being done responsibly?
Julia Stoyanovich:
So in my very simple worldview, there are essentially four things, at least four things, right, that you need to think about and convince yourself that these conditions are met for you to claim that you’re using AI responsibly. The first is that you are using AI to meet some clear need for improvement. You’re not just using it because your competitors are using AI, but there’s some actual problem that you can clearly articulate. and that you want AI to help you solve. The second related point is that you can actually check whether the AI that you’ve developed or are about to develop is going to meet the requirements for that need for improvement. Can you validate that the predictions of the AI are good and correct? If you can’t validate it, then again, it’s not the right setup. Also, of course, we need to convince ourselves that the problem that we have set out to solve can be solved, given current capabilities in terms of hardware and data and software. And if that is not the case, if data doesn’t exist that would allow you to predict the kind of thing that you want to predict, then it’s hopeless. Because AI is not magic, right? If you expect it to be magic, you’ll be disappointed. And the final thing is that AI very rarely operates autonomously. Usually it’s in a collaboration with a human, right? So the question is then, do you actually have these decision-makers who are well-equipped to work with your AI and to not take its recommendations on face, but rather to challenge them when they need to be challenged? So here again, this example of a clinician working with AI to diagnose a disease. shouldn’t just say, yeah, this person has this type of cancer and we are going to subject them to chemotherapy because the AI said so. They understand that it’s up to them to make the decision. So this decision-maker readiness is a crucial component. And then of course other things like legal compliance. Are you going to be legally compliant in your data collection and AI use? Is this socially sustainable? There are lots of other very important criteria here. But these four are absolutely crucial. Is there a problem to solve? Can we solve that problem? Can we check that we solved it? And can we use this AI, this solution, safely together with the human?
Damien:
So why are you saying that AI should not be used for any, virtually any automated decision making? It should always be done in conjunction with the human?
Julia Stoyanovich:
This depends on what are the harms, actual or
Damien:
Hmm.
Julia Stoyanovich:
potential harms. Right? I mean, and here there is a lot of conversation in policy and legal circles. The European Union AI Act speaks about this. And of course, before them, there were other legal documents that also speak about risk levels, right? That then compel you to institute a particular level of oversight. So for example, the Canadian directive. on automated decision making that the Canadian government has been abiding by for several years now. It also has risk levels and they incorporate both, you know, the extent of the risk of harm, the severity, is it reversible or not, how many people are impacted, is it just one individual, or is it a large population group. So these are nuanced conversations to have. They’re complex. But if it’s a toy, maybe it’s less important to regulate it quite as closely. But if it’s something that’s deciding who gets medical care and who doesn’t, or who gets a job and who doesn’t, or who gets shot in a battlefield because they are considered a combatant versus a civilian and who doesn’t, then we need to think, depending on the domain, how closely we look at these systems.
Damien:
Can we let it drive a car ever, do you think?
Julia Stoyanovich:
So I’m not an expert specifically in self-driving cars, but I think that we can let it drive a car if all other cars are also driven by AI and if there is some level of coordination between them. But if you have a mix of, you know, humans and AI driving and the roads such as they are today, I don’t know. I mean maybe in Germany on the Autobahn we could do this sooner than elsewhere, but… I’m in rural Pennsylvania right now and here I just don’t see how this is gonna happen anytime soon.
Damien:
Great, good, good. Philip, any further questions from yourself?
Philipp:
No, I think we have what we wanted to discuss. Julia, I could speak hours with
Julia Stoyanovich:
Yeah, sorry,
Philipp:
you
Julia Stoyanovich:
this was a very
Philipp:
and
Julia Stoyanovich:
high
Philipp:
go deeper.
Julia Stoyanovich:
level conversation.
Damien:
Mm.
Julia Stoyanovich:
I’m usually less comfortable with this than with concrete examples, but yeah.
Philipp:
I can see that. But to be honest, I’m surprised about a lot of the things that you say, but they make a lot of sense to me
Damien:
Mm-hmm.
Philipp:
also at the same time. So it would be really interesting to maybe we can even do a follow up at some point again and just to a deep dive into
Julia Stoyanovich:
Sure.
Philipp:
one of the topics
Julia Stoyanovich:
But I mean,
Philipp:
at
Julia Stoyanovich:
there’s
Philipp:
some point.
Julia Stoyanovich:
also this, you know, when you speak, do you hold an extreme point of view, or do you hold a point of view that is very nuanced and reasonable, right? There is kind of a balancing act there. And very often it’s useful to articulate extreme points of view, because they make people think more clearly.
Philipp:
That’s
Julia Stoyanovich:
Yeah.
Philipp:
true, yeah. That’s true. But I think it’s also, I mean, I’m working a lot with GNI, right? I mean, as much as you can at this point, let’s say this. And I’m also following the regulations discussion and so on and so forth. But your point of view is obviously different from what I read in my daily work, or it differs to some extent. What was very interesting for me to hear from you is this idea that the responsibility lies much more with the engineers who build the systems and it’s much less of a chip that you put into the machine or a component of the machine that resolve the problem. Of course, that’s the way we think because that’s all we do. If something’s missing, we build it into the system. So there is a very strongly biased way of us approaching that. as data scientists or data engineers. Yeah, I don’t know what the final answer will be in the end, but I found it very interesting to
Julia Stoyanovich:
Yeah,
Philipp:
hear
Julia Stoyanovich:
so
Philipp:
for
Julia Stoyanovich:
maybe
Philipp:
sure.
Julia Stoyanovich:
I’ll just add to that since we’re still recording. So in engineering, right, we have a point of view that is reductionist by necessity, right? What does that mean? Uh,
Philipp:
Yeah.
Julia Stoyanovich:
we need to compartmentalize problems, right? We need to come up with models for subsets of the problem. So if you think about building, you know, a car, you don’t want everything to be connected to everything. You can never make that run, right? You want to make sure that there are boxes that are clean. Maybe you don’t look inside the box. And then. there are very few wires connecting these boxes. And this is a really good clean design, right? But the world is really complicated, unfortunately. It’s very hard to have a reductionist point of view on things like bias in society when it comes to lending or hiring, right? Everything is connected with everything else. So then how do we figure out what we reduce into these boxes? And does it help to add more boxes or more wires? Or is it just like a problem with just the car not running at all? Or you not being able to use the car that you built for the road for driving now to fly suddenly, right? It’s a very different. It doesn’t take another carburetor or just attaching wings. It takes a complete redesign. So it’s really the extension between like the engineering reductionist point of view and the more holistic. social science kind of a point of view. And we need to land somewhere in between, but we don’t yet have the methodology for this, even for the thinking. I mean, the way that we’re taught is either reductionist or holistic. There are very few people who are able to
Philipp:
Mm-hmm.
Julia Stoyanovich:
navigate between these, so that’s the challenge for engineers. And like, we are used as engineers to saying, here’s a problem, there’s a solution to that problem. And the solution is from within the toolkit that we got. But that’s not the case here, sadly. I wish.
Damien:
Yeah, I think that’s beautifully put. You’re right. There’s very few people that can straddle both the holistic point of view and the, um, the more reductionist. I think that’s a perfect sentiment on which to conclude.
Julia Stoyanovich:
Great. And you know, I’m not an expert in this either, obviously. I’m an engineer myself. I’ve taken a couple of philosophy classes throughout my studies, but yeah, we all have our biases, I guess, from the background that we come from.
Damien:
Cool. So I’ll just close
Julia Stoyanovich:
Okay.
Damien:
off this, the summary and then, um, we’re done. We can pause the recording. So, so that concludes today’s episode folks, before we leave you, I just want to quickly mention our magazine, the data scientist issue four is out in early September and we’ll be focusing some of the words leading companies such as Merck and continental and what they’re doing in relation to generative AI in particular. It’s packed full of insights. and not adverts. And there are future articles from Julia and of course, Philip is a regular contributor. You can subscribe for the magazine for free at data science talent.co.uk forward slash media. But Julia, thank you so much for joining us today. Extremely thought provoking, some amazing insights there. It was an absolute pleasure having you on the show again.
Julia Stoyanovich:
Thank you very much, Damian, and thank you, Philip. It was a pleasure, and we’ll be in touch. Thank you.
Damien:
Great,
Philipp:
Thank you.
Damien:
and thank you also to Philip and of course to you for listening. So do check out our other episodes at data science conversations.com and we look forward to having you on the next show. Good. All right. Excellent.
