Hadooponomics: Debugging the Gender Gap & Finding Big Data Success (Podcast Transcript)

Hadooponomics11Listen to the original podcast episode.

James Haight: Welcome back to the Hadooponomics podcast, everyone. I’m your host, James Haight, and today we have Cornelia Davis, she is the Senior Director of Technology at Pivotal. And Cornelia has a really interesting job. Her job is to work with giant companies and help them figure out how to actually make sense of their investments in Big Data, and she helps lead them to success. So from this we actually get a really interesting, first hand view of what makes some companies fail, what makes them succeed, and how to overcome a lot of common stumbling blocks.

But also, Cornelia, on the side, is an activist for gender equality, especially in the world of technology, and we use this as an opportunity to build on our previous discussion. If you remember back to our episode with Emer Coleman, we talked a lot about the importance of gender equality. And I think the most interesting thing that we touch on is really that it’s not any sort of very specific action that people are taking, or any sort of malicious behavior. But rather, it’s about understanding the series of implicit biases that effect our everyday life. And the opportunity to recognize this and to change them is really where we get the chance to improve, go forward, and make things better.

So a lot of interesting stuff in here, I know you guys are gonna enjoy it. We link to a lot of the resources that we mention, like the documentary, so if you want to find that, bluehillresearch.com/hadooponomics. We’ll have links to all the resources that you need as well as a transcript up on the website if that’s more of your thing. And of course, if you wanna find us, iTunes, Stitcher Radio, find me on Twitter, @James_Haight. You guys know the drill, we’re there, anywhere you wanna find podcasts. Find us, interact with us, we’d love to hear your feedback.

And with that, that’s all I have for you today, so I’m gonna step out of the way, and without any further ado, here’s our interview with Cornelia. Okay everyone, I’m here with Cornelia Davis, she is the Senior Director of Technology at Pivotal. Cornelia, welcome to the show.

Cornelia Davis: James, thank you so much for having me, delighted to be here.

James: So Cornelia, one of the things we love to do when we start off is just get to know you a little bit. Can you give our audience a sense of the background of who you are and what you do?

Cornelia: Sure, thanks so much. Yeah, so as I said, I work for Pivotal, and I’ve been with Pivotal since the spinoff. Your listeners probably know that Pivotal is a spinoff from VMWare and EMC a little more than three years ago, and I’ve been there since the spinoff. I came from EMC where I worked in the corporate CTO office, and I was working on emerging technology. And right before the Pivotal spinoff, actually several month before the Pivotal spinoff, I was working on the emerging technology space of Platform as a Service. And being EMC, and given that Cloud Foundry was being incubated at VMWare, I was naturally looking at Cloud Foundry. So I’ve actually been working with Cloud Foundry for almost four years now, so that was pretty early days in Cloud Foundry. And when the Pivotal spinoff happened I joined the Cloud Foundry product organization which is where I am now, I’ll say more about that in just a moment. And I’ve spent the last three years in a role where I was working with customers and partners to help them understand this emerging technology space. Because, in fact, what was emerging technology four years ago when I was at EMC is still somewhat emerging technology for our customers today, for large enterprises today. And more recently, because, actually, the applications platform side of it is becoming better understood by enterprises today, there is another area, another element to the platform that is perhaps still a little bit more emerging, which is the data component of this platform. And so as of late, I’m focusing exclusively on how the data elements effect and come in to the platform that we offer through Cloud Foundry.

James: Oh, perfect. And what I love about your background, and part of the reason why I’m really excited to have you on the show is that whole preamble of what you’ve worked on, and sort of what you’re doing now, as I understand it, what you’re doing is, people who make these big bets on Big Data, and the folks who have made these investments and are trying to bring them into their organization, your job is to help them actually realize the promise, right?

Cornelia: Yeah, absolutely. Gosh, I have so many stories already, just in the few months. I’ll maybe start with the first one, the one that really planted the seed for this kind of shift in focus. And that is that I would say it was about three months ago that I was asked to come out and speak with one of our customers. It was a group of individuals who didn’t know very much about one of the topics that I’ve been talking about for years now, which is DevOps. And they said, oh, we need to bring Cornelia in, you need to talk to her, she knows a lot about DevOps, she’s worked with lots of customers on their DevOps strategy. And so I came in to help this group, and it was supposed to be a couple of hours kind of brainstorming, meetingwith this group. And I came in and I started talking to them about all of the things that I’ve been talking about for years. Continuous delivery, and why continuous delivery is actually the goal, it’s really, ultimately, the goal of DevOps, and that, in fact, continuous delivery, the goal of continuous delivery is fast feedback loops, risk reduction, and so on. And then some of the techniques that support that, like having your developers and your application operators have a self-service capability where they can just get the things that they need, the resources that they need to do their development, and to do their deployments into production. So automatic provisioning, deep provisioning, environment parity, so that we don’t have that finger pointing of, “it works on my machine, it doesn’t work on the production machines”. Things like immutability, not allowing your environment to experience drift. So doing things like not allowing SSH access and not allowing your support teams to create snowflakes, cuz that’s really bad for DevOps. Having the right platform with the right abstractions. And I was talking about all these concepts and it was a fabulous meeting. And the team, they were cringing in all the parts where I was talking about things that are anti-patterns, they were recognizing that they had many of those anti-patterns. Nervous laughter in places, and looking at each other and saying, oh, we absolutely need to do that, or, that’s a great idea. And as we were stretching into the fourth hour of what was supposed to be a two hour meeting, it was great. Everybody was having a great time.

Cornelia Davis:
I’ve been having these conversations for three years with teams responsible for application development. And all of these things that we’ve been talking about from an application development perspective, DevOps, agility, and all of that stuff, apply equally well to data.

But here’s the punchline, James. I had been having these conversations with three years with teams that were responsible for application development, this team was the data team. It was a very large enterprise, a large organization, and this is the team that was in the business of providing data solutions to their businesses, to the application teams and the application operations team. And I have this major a-ha moment where I realized, all of these things that we’ve been talking about from an application perspective, application development perspective, DevOps, agility, and all of that stuff, applied equally well to data. And that the time was ripe for us to bring data into that platform that is really providing the right support for a new way of building software, that we needed to bring data into that platform in a really meaningful way as well.

So that was kind of a big a-ha moment that I had where it was like, oh, okay, I wanna focus on this.

James: Sure, and it strikes me that there’s a few parallels there that are really important. I want to dig into a couple to tease them out and sort of expose what that actually means for our listeners who are trying to optimize their own Big Data investments. But I know that at the Strata-Hadoop Summit this year, the one in San Jose, we actually released a blog talking about how Big Data’s fundamentally an application problem, right? And this is something that I think has been a recent transition in the fact that people are talking about it in that way. It’s transcending a lot of the old way that we were thinking about bringing data into our organizations, and it really invites a new way of thinking about it. I’m curious, to you, what’s that mean? What’s it mean to our audience, how should we be thinking about actually making value out of our Big Data investments? Because it strikes me that the approach has changed and will continue to change.

Cornelia: Yeah, so there’s actually two comments I’d like to make with respect to that. I find it a little bit ironic that having been on the applications side now for the last three or four years, that there’s something that we say on the applications side: applications are really a data problem. So you have to build these applications but there’s no such thing as an application that doesn’t have data that’s serving it, that there’s some kind of data that’s supporting the application at the back end. And so in order to be able to bring a new application, a new digital experience, value to the customer via those digital experiences, we really need to have Big Data be a part of that as well. So I think it’s ironic that, coming from the Big Data side, you see it as an application problem, and coming from the application side we see it as a data problem. The crux of it is is that it’s a data and application problem.

James: Yeah, or maybe it’s a business problem, right? Where we’re really just trying to further the interest of the business and it takes this intermingling of the two sides, right? So maybe the divide is artificial in how we think about it.

Cornelia: Yeah, I think that’s exactly right, I think you’re right. The divide is artificial, I love how you put that.

James: One of the things that I love about sort of what I’ve been able to understand from your work, and this goes with a lot of our people that I get a chance to talk to, and we brought up before on the podcast is, people say okay, great, Hadoop is free because it’s open source. Big Data doesn’t cost that much, right? But they find out that it’s free like a puppy dog, right? There’s a whole lot of things that you have to do to make it work. And I’m curious if you can either, one, talk to what you see, what hidden costs are that people don’t understand. Or what are the stumbling blocks that people overlook until they actually roll up their sleeves, and dig in, and try and bring it to their company?

Cornelia: Yeah, so there’s a couple of instances, couple of stories spring to mind. One is that, in fact, that customer story that I told earlier, one of the things that they shared with me was that they had rolled out Hadoop, and they had this Hadoop system available and ready to go in case somebody wanted to use it. And even within their own organization, and I mentioned that they were the data team, and they’re the data team for a very, very large organization, large, Fortune 50 organization. And there’s 600 people in that data organization that are providing these data services to other teams within the company. And they estimated that of those 600 people, about 30 of them had enough proficiency with Hadoop to be able to use it meaningfully. So I found that number to be staggering. So they have established this capability, but they really aren’t using it because they don’t have the right skill sets, and I, in the last few months, have been out talking with customers about their requirements and it’s not just a skillset problem. It is really just even more fundamentally a matter of unfamiliarity. And what I mean by that is, again, not lack of skill, but it’s this is the way we’ve always done things.

Cornelia Davis:
Of the 600 people [on the data team at a Fortune 50 organization], about 30 of them had enough proficiency with Hadoop to be able to use it meaningfully … Even though Hadoop has been around for some time, there is still a little bit of a ramp up to understand – to map particular use cases to these technology stacks.

So we’ve learned over the course of many, many years, many decades, we understand relational models really well. We understand SQL extremely well. And so when we’re faced with a problem we tend to fall back to what we’re familiar with. And so there’s still, even though Hadoop has been around for some time and there’s been a lot of hype around it, especially in large enterprises, there is still a little bit of a ramp up that is needed to be able to gain that proficiency. And to understand, to be able to map particular use cases to those new technology stacks, I think that’s where the gap is. Sometimes we can’t map the requirements to the new technology stacks because we just are not used to that relationship.

James: Sure, and I think we see, quite often, when people are faced with a challenge, and maybe they’re time crunched and they don’t have the mind space to be creative and try something new, they’re like, hey, well, this is the way I’ve always done it. I know it might not be the fastest thing, but I know that if I do this it’s gonna get done, so maybe I’ll try out that new thing later. I wonder, is that a piece of the puzzle too?

Cornelia: Oh that’s definitely a big piece of the problem. And that’s really what I was trying to get at, which is, it’s that familiarity thing. And you’re right, I think you actually touched on something that I wasn’t talking about, which is risk. I think that there’s a concern of okay, if I go down this new technology track, is it proven technology? Is it something that our developers understand, or that we could even do operationally? Because operations of these new platforms are very different from the old. That’s something that I’ve learned very, very concretely on the Cloud Foundry side. Is that the old model of plan, build, run in an IT organization, in terms of there’s a group that plans out what the application is that is needed by customers, then there’s a group that builds them, and then there’s a completely different group that runs them, the Cloud Foundry platform has allowed us to reorganize those groups into one group that has the platform, ownership of the platform, which they treat like a product. And then another group, which is the application team, and they own the application through the entire lifecycle. We’ve understood that transition pretty well on the application side, what are those organizational changes on the data side? How do you remap the roles to become more efficient, even on that?

James: Yeah, that’s amazing, and I think the way that you verbalize it and have sort of consolidated all of that, it takes a lot of the experiences that I’ve run into and talked with people, and brings it into a really nice way of thinking about it.

So what I wanna do also with this is, it’s great to get your perspective on how people can maximize their investments in Big Data and so forth, but another piece to this is bringing more people into the equation. And what I love about your background, and our audience, I’m sure, will as well, is you’re doing a lot of work on the human element of this as well, around gender equality. And I’m just curious if you could touch on that a little bit, and there’s a number of directions I want to take it from there.

Cornelia: Sounds great, I’m delighted to talk about this topic any chance I get. And your timing is good. You might know that next week is the Hadoop Summit in San Jose, and I was invited to sit on a panel of the Women of Big Data. And I thought it was rather funny, because a couple of months ago when this invitation came to me, and I kinda chuckled a little bit, and I told the organizers, well, you know that I’ve spent the last three or four years on the applications side and I’m just starting on the data side. And they were like, you know what, that’s a great part of the story. So I focus on the Big Data now even though I am relatively new to Big Data. And I guess that’s part of the whole story and part of the message that I want to maybe leave some of these listeners with. Is that I’ve been doing this a long time, I’ve been in the industry for 25 years, and I’ve probably reinvented myself a dozen times. There’s always great problems to be solved, and with enthusiasm you can go out and do cool things. So yes, we have this event next week, the Women of Big Data and I’m so honored to be part of that.

Cornelia Davis:
I’ve been in the industry for 25 years, and I’ve probably reinvented myself a dozen times. There’s always great problems to be solved, and with enthusiasm you can go out and do cool things.

But maybe I’ll back up and tell you a little bit about how I came to be involved in this, because I just mentioned a moment ago that I’ve been doing this a long time. I’ve been in the industry for over 25 years, and my background is in computer science, I have a Bachelor’s and a Master’s degree in computer science. And I’ve always been on the technical side. A couple of times I’ve dabbled with going over and doing a little bit more management and I never care for it. I always end up going back to hardcore technology. And over those 25 years I often noticed that I would be in a room, and I would look around, and there’d be 25 of us in the room, and I’d look around and think, oh, there it is again, I’m the only woman. Or oop, look, there’s two of us today. And it was really just something I kinda noticed, and I didn’t do anything about it other than notice it occasionally. And about two years ago I had a just really, really great, lucky moment in that we were very fortunate at Pivotal to get to know two filmmakers, Robin Hauser Reynolds and Staci Hartman, who are the filmmakers behind a film that debuted last year in April at Tribeca Film Festival, and the film was called Code: Debugging the Gender Gap. And it’s a documentary that really goes out and explores, first of all, the statistics, so the raw numbers of women versus men that we have in these computer science, these software positions in the industry today. And I’ll tell you, it’s not great. It’s roughly in the teens, so somewhere between 15 and 20%, and that’s across everything. That’s in college, it’s in the workforce, it actually gets worse in the workforce the longer people are in there. But in any case it looks at these statistics, and then it tries to dig in a little bit on why we’re seeing these numbers. Oh, in the late 80s or in the mid 80s when I was in college, it was the peak. At that time roughly 38% of the students doing degrees in computer science were women. And it had been increasing for some time, and so everybody thought, wow, this is great, we’re gonna reach gender parity here pretty soon. And now, unfortunately, that number, that same number, the percentage of computer science students who are women now stands at roughly 17%, less than half of it what it was 25 years ago, which is quite alarming. And so the film is really digging into what are some of the factors that might be contributing to that.

James: Sorry to jump in, but, I mean, I’m honestly shocked. So do they offer an explanation as to why the numbers have gone down? I can’t think of any scenario where that makes sense to me. Really curious to know if they found anything or if there’s sort of this untold narrative that we should be paying attention to?

Cornelia: Yeah, it’s really, I’ll tell you, it’s not one factor. It’s a number of things. And they do make some suggestions in the film and do look at some very specific topics. So one of those topics, and I’m gonna take it a little chronologically, and to some extent they do this in the film as well. So it really begins with young girls. And that’s where the film kicks off, is it kicks off with interviews of young girls asking them what their perception of computing is, and what the perception of the individuals that are computing professionals is. And at a very young age, girls already have this perception that a computer scientist is a man, a little nerdy, sits in the basement drinking soda pop and just coding until 3:00 in the morning every morning. That’s the perception they have. So we have to ask ourselves, where are they getting that perception? And to a large extent, I personally believe, and a lot of good people believe that it comes from the media.

James: What we’ll do up on bluehillresearch.com/hadooponomics, in our show notes we’ll put a link to the documentary, Code: Debugging the Gender Gap. We’ll put a link up to that so people can check that out and learn a little bit more.

One of the things that I’m interested in, we’ve talked about it from a few different angles on this show, and specifically I’m thinking of our listeners who remember Emer Coleman, who was talking about this idea of the monoculture. It’s important, if we’re doing very important, influential things, to bring a variety of perspectives to it because it affects so many people. So you wanna have a diversity of opinions, and thoughts, and morals, and all those things that go into designing core processes. And I’m curious, do you see that, or how do you see that playing out in specifically the tech space, the Big Data realm, in the large companies that you work with?

Cornelia: Yeah, it’s really extremely important, and that’s a really great observation on your part. So there are some great stories of the past. One of the stories that’s talked about in the film is how voice recognition systems, initially, were engineered by men, and they were designed and they were tested by men. So when the first voice recognition systems came out they couldn’t recognize female voices at all. They simply couldn’t hear them. Female voices could not be heard because none of the data that they used for development or testing was data of women speaking. And what was just a perspective, it was that monoculture, it was that mono-perspective that was brought to it.

Cornelia Davis:
Voice recognition systems, initially, were engineered by men, and they were designed and they were tested by men. So when the first voice recognition systems came out they couldn’t recognize female voices at all. They simply couldn’t hear them. Female voices could not be heard because none of the data that they used for development or testing was data of women speaking. And what was just a perspective, it was that monoculture, it was that mono-perspective that was brought to it. … More than 50% of the purchases made online are made by women. And so wouldn’t it be fantastic if we had a female perspective in the mix when we’re generating those digital experiences?

When we think about digital experiences now, though, let’s take something simple like ecommerce. More than 50% of the purchases made online are made by women. And so wouldn’t it be fantastic if we had a female perspective in the mix when we’re generating those digital experiences? So it’s absolutely crucial to have all of those different viewpoints. And really, the punchline here is that everything that we talked about so far in terms of gender diversity, we could’ve said, well, the reason for making sure that there’s a balance in gender is something that we need to do just because it’s the right thing to do. And yeah, I would argue that doing things just because it’s the right thing to do is absolutely valuable, but what we just talked about here demonstrates that it’s also just fundamentally good for business. And so having that gender diversity, and I love what you said when we transitioned over into this topic as well. You said you wanted to bring more people into the equation. And so we have this tremendous need for technologists, not just in the United States, but globally. And we have a tremendous need to educate more individuals and attract and retain more individuals across all diverse groups. Whether it be gender diversity, socioeconomic diversity, people of color, all of those things are really equally important to bring a diverse viewpoint to that pool of individuals.

James: Yeah, and I think you hit on something important. We’re not saying people are doing this intentionally, there’s sort of all these reasons of why things exist the way they are. I think the point more is in the same way that if you want to get an accurate prediction from a survey it’s best to have a randomized trial or statistically relevant sampling, right? A diverse enough sampling of people that you’re, oh, sorry, the word I’m looking for, a representative sample, to actually predict what you’re doing might impact the entire population. So it sort of just makes good business sense to broaden the diversity of perspectives from whatever avenue they come from, right? And I think that’s the more compelling case that’s going to drive change, is the idea that hey, look, it actually is business savvy to sort of proof our ideas and our concepts and everything that we’re bringing out into production. It’s better if it’s tested by more diverse perspectives, because that’s how you pick up on things that one perspective will miss. So I really like that angle as to it’s actually business savvy, it’s not necessarily just some moral high horse aspiration. There’s a bit more to it.

Cornelia: Yeah, absolutely true. And I wanted to touch on something that you just said, which is that people are not doing it on purpose. And that couldn’t be more true. Sure, there’s some individuals out there who are just sexist, and they’re overtly sexist and that’s a problem. But those individuals are very few and far between. For the most part, a lot of what drives this gender inequity is what we call implicit bias. And every one of us has biases that are just ingrained in us because we, as human beings, we get so much data coming in that we need to categorize. And when we categorize we risk including biases in those categorizations. And so every one of us has implicit biases. So I spend a lot of time talking about these issues, and then, without fail, after I have chatted with a group of individuals, without fail I have individuals come up to me afterward and say, you know, I never thought of that, I never thought about that. I did a talk earlier this year at an internal event, at a company event, a fairly large event, we had over 1,000 people there, where I talked about how these perceptions that young women have, and that really, adults also have, around who looks like an engineer and who doesn’t. And I had colleagues of mine come up afterward and say, you know what, I signed my son up for coding camp this summer. I never thought to sign up my daughter. To which I always respond, you’re signing her up tomorrow, right? Yep, signing her up tomorrow!

James: [laughs]

Cornelia: So it really is, it’s even parents, and parents want nothing but the best for their kids. But even they have their implicit biases and they think to sign up their sons for coding camps but not their daughters. And so the other big campaign, the other big platform that I stand on in my activism is really just getting us all to be a little bit introspective and to try to recognize our biases, and then compensate for those. So that when that bias comes in I can recognize it and say ah, that’s just my bias. Let me see if there’s a different way that I can think about this. And you’ll get different outcomes.

James: Absolutely, and that’s probably true for so many other aspects of life, right? That’s how really great people make themselves better. That’s what CEOs do to assemble the best teams around them, they understand their weaknesses and try and bring people on board who can complement them, right?

Cornelia: Yes.

James: So I think that’s a general, good rule of thumb for the audience. But interesting to see how it actually all ties back to data, and sort of how you’re getting value out of these investments, whether it’s bringing more people, whether it’s understanding that there’s a whole lot more that you can gain, and it’s actually good business sense to have a representative sample when you’re designing things.

Cornelia: Yes.

James: So Cornelia, with that, we covered a whole lot in this podcast. I know our audience is gonna have a lot of things to take away from this. I’m curious if there’s any other sort of parting shots that you wanna leave the audience with?

Cornelia: So I think what I would say in parting is that we are at the precipice of, really, a new era in computing and we sometimes refer to that, and when I say we, the industry sometimes refers to that as the third platform. The first platform being mainframes, the second platform being client/server, and the third platform now being cloud native. Now when I say cloud, it’s ironic, this is the first time I think that I’ve used the term cloud in this whole conversation. When I talk about cloud native I’m not talking about simply writing things that run in somebody else’s data center, so in Amazon’s data centers, for example. I’m talking about cloud as a fundamentally different architecture. A highly distributed, highly malleable, always changing architecture. That means we’re using commodity servers, they come in and out, we’re constantly upgrading things, and so on. And this is really, truly, a new era. That new era is also marked by almost limitless capacity. And with almost limitless capacity we are able to do things, not only on computing platforms, but on data platforms, that were unimaginable 10 or 20 years ago. So I can’t imagine a more exciting time to be in the technology field. And so I guess I would close my closing with this. One of my favorite movies of all time is The Matrix, and it’s not because I’m a sci-fi fan and it’s definitely not because I’m a special effects fan. It has everything to do with one scene in that film, and it’s the scene where Morpheus and Neo are standing on a rooftop, and Morpheus is telling Neo, really, you can jump from this rooftop to this rooftop. And Neo’s like, yeah, no [laughs]. And Morpheus says, free your mind, and then he jumps from rooftop to rooftop. That’s my parting words, is free your mind. We have an opportunity here to really, truly change the world.

James: Absolutely. And Cornelia, if our audience wants to follow up, learn more, and sort of dive into what it means and talk to you about what it means to free their mind, where are we going to find out more about you?

Cornelia: So on Twitter you can find me @cdavisafc and you can also find me on the Pivotal blog. I work for Pivotal, we have a blog, you can find some blog posts there so you can find me through Pivotal that way. I think those are probably the best ways. And I’m often out and about at various conferences. I know that this is gonna publish after the Hadoop Summit, but I’ll be at SpringOne Platform for anybody who’s gonna be there, I’ll be there at the first week of August in Vegas. And yeah, just check me out on Twitter, that’s probably the best way to figure out where I’m gonna be.

James: Okay, well Cornelia, it’s been an absolute pleasure to have you on the show. Thanks so much for coming on.

Cornelia: James, thank you so much for having me, it’s been a real pleasure.

Posted on by admin