David Garrison - #TrueDataOps Ep. 38

[00:00:00] Speaker A: True DataOps podcast. We'll start here in a few more seconds to allow people to get logged in on the live stream. See you in a few. All right, well, welcome to this episode and season three of our show, True Data Ops. I'm your host, Kraziano the Data Warrior. Each episode, we try to bring you a podcast discussing the world of data ops with the people that are making dataOps what it is today. So be sure to look up and subscribe to the DataOps Live YouTube channel, where you can find all the recordings from our past episodes. If you missed any of our prior episodes, now's a good time to catch up as we start season three. Better yet, you can go to truedataops.org and subscribe to the podcast and then you'll get proactive notifications. My guest today is a data ops practitioner, Snowflake Data Superhero, and a senior data architect at National Grid, David Garrison. Welcome to the show, David. [00:01:13] Speaker B: Yeah, thanks for having me. [00:01:16] Speaker A: So, for those that don't know you, could you give us a little bit about your background in data architecture and data ops and what you've been up to? [00:01:24] Speaker B: Yeah, sure. Well, I've been in data pretty much since I got out of college, so 2012 years, I've had most roles that have the word data in the title. Data analyst, BI data engineer, data architect, Snowflake data superhero. So one thing that I love about data is that it's such a transferable skill for different industries. So it's, you know, you can hop from mobile games to real estate. So ultimately, a lot of my experience in DataOps data architecture comes from making a lot of mistakes, seeing other people mistake, make mistakes, try not to make the same mistake twice if it can be helped. [00:02:18] Speaker A: Yeah, that's one of those repeatable things you don't want to repeat, right? Well, for this season, we want to take a step back and we're thinking about how the world of true DataOps has evolved over the last couple of years and what we've learned. So you've been a practitioner in the field and worked on DataOps on several different organizations and industries, so I'm really looking forward to hearing your thoughts on all this for our listeners. If you've not looked at the seven pillars of True Data Ops recently, you can find [email protected] seven pillars. So that's a good reference for you. Now, it's been like four years since we first published the truedataops.org website. The Philosophy and the Seven Pillars. Also the dummies Guide to Data Ops. So in that time, do you think that the seven pillars of true data ops still resonate with people? [00:03:22] Speaker B: I certainly think so. I mean, I've seen various different iterations of these types of pillars. Seven pillars, six pillars, five foundations. So I think there's certainly a lot of value there in kind of defining these are the fundamental aspects. Yeah, I think they hold true. [00:03:49] Speaker A: So we've got this AI and gen AI stuff going on all the time. So how do you think that plays with the concepts of data ops, true data ops, and what we need to do to make things work and really be effective with all of this new technology and kind of the speed at which things are going today? [00:04:14] Speaker B: Sure, it's an interesting question. My initial gut reaction is that it hasn't really changed things fundamentally. I suppose, if anything, AI makes it a little bit more important because if you're not setting up a test environment and managing source control, then you're not doing data science. But I think if anything, it's been undervalued in, I don't know, decades past. So I don't know if it's gotten more important, but maybe there's more talk about it, more awareness, more conversations about it. [00:04:55] Speaker A: Yeah, I think that part of it, like you said, you mentioned fundamentals and we get talking about AI and gen AI and it seems like things that were fundamental principles of data governance, data quality, data modeling, all of that that has been out there for literally decades, have kind of become more important because things are moving so fast. I mean, it's one of those good news, bad news things. Right, is things are moving so fast that if you do something fundamentally wrong, you'll probably find out pretty quick. But at the same time, if you're generating things using Genai and you don't understand the fundamentals, do you even know that you actually built something that's not quite right or isn't going to get you the right result if you don't have some sort of controls in place? [00:05:51] Speaker B: Yeah, well, Genai is something that kind of inherently, just because of the scale of it, requires scaling up. And if you're at that point of scaling up, you need to be following a lot of these principles. I think that kind of goes for any company that's scaling up, even if you're not using AI. It's just that most companies that are getting that big at that scale or getting into data science a lot more so. [00:06:18] Speaker A: But even if, like you said, even if they're not scaling up to that point, if they're Just you know, building their data analytics platform on Snowflake. What have you seen in that area? It's like, you know, how important has it been for them to follow some of these data ops principles? [00:06:38] Speaker B: Yeah, well I can't say that I've worked at a company that has gotten all seven of these principles right. There's, there's always something lacking. I haven't worked at your big Amazon's, Google's, Microsoft's, but yeah, but I mean if a company did have all of those pillars right then you know my not be inclined to leave. So maybe that's why they're hiring us. It's tough to find. [00:07:14] Speaker A: So what kind of things have you seen organizations struggle with? [00:07:19] Speaker B: The biggest ones that I see struggling with are really getting into CI CD for data. Like they'll have CI CD for web applications and, and whatnot. But data is, seems to be a hard challenge there really getting into the automation of testing, you'll see good testing practices but really getting to that step of having it be automatic and having it be test driven can be a challenge. Some of them. I'm just coming from the world of Snowflake. The thing that I see the most often is actually issues with grant management governance. I one thing one of my biggest frustrations with Snowflake and I've told them this, and I should maybe tell them this more often, but they have excellent tools for handling very fine grain grant management, RBAC controls, security management, but they don't provide built in tools for like the structure of following best practices. So being able to set up a read only role for a database requires managing those fine grain controls. So having that structure around creating business roles in Snowflake in particular is something that I see frequently. So part of that just comes from the fact that I've been working with Snowflake so much in the last six, seven years. [00:08:50] Speaker A: Yeah, so yeah, so some of that's not like you said, it's not built into Snowflake. The capability is there but you got to do a lot of building yourself in order to make it work. Have you found any reasonable ways of managing that? [00:09:09] Speaker B: I have certainly there are a few tools that help provide that structure, provide the semantic layers and templates. DataOps Live is one of them. [00:09:24] Speaker A: It's okay, you're allowed to say that on this. [00:09:26] Speaker B: Yeah, yeah. Big fan of DataOps live for that purpose. I don't get to use it at every job I work at. Even now at National Grid part of the company uses DataOps Live, but that's a Big part of the company doesn't and that's the part that I usually end up working with. So. [00:09:46] Speaker A: Yeah, and you know, there's an easier way, but we're not going there. [00:09:52] Speaker B: Yeah, that can be frustrating for sure. But I've certainly seen homegrown solutions work as well. Even just building out templates and having good documentation can be the difference maker. But sometimes that gets in the way as much as not. [00:10:12] Speaker A: Yeah, yeah. I mean, what you're talking about like being able to have a template for, you know, people who, you know, I'll say don't fundamentally understand the underpinnings and the fine grained level, but know that. Okay, we need read only access to the analytics for the finance team. How do we set that up? Right. Because we don't want the finance people necessarily going in and changing the data in Snowflake, but they certainly need to be able to access it, whether it's via Tableau or Power Bi or even in some of the built in tools inside of Snowflake. I can see that'd be a fairly big challenge, especially coming from a business perspective rather than a technical perspective. [00:10:58] Speaker B: Right, yeah. And you can certainly set up scripts that say, I want a read only role, create that role, this is what it's called. And then run through all the objects and make it appropriate for an analyst. But that requires a decent understanding of the business and any database standards that that company is using. Environment. It tends to be a little bit specific to the company unless you're using something a little more standard industry best practice. [00:11:32] Speaker A: Yeah, and I guess that's really the challenge there because even if you've got some scripts that say you wrote a bunch of scripts and you hand them off to the data analyst or the data architect or the data engineer and one of these other teams, how do they know they ran it right? How they know that it's being kept up? The governance portion of it. So yeah, it's one of the, the seven pillars there. I think you mentioned that before. It's like, yeah, that the governance, the security and then change control on just that sort of thing. Right. It's like forget about the change control on the schemas. Right. You need change control on the schemas and the table definitions and the load processes and the pipelines and all that. But you got to have the change control even on the governance aspect to. [00:12:23] Speaker B: Pretty much every company I've worked at has had at least one admin who just goes and runs grant scripts and doesn't get them into source control. I mean they'll usually have a ticket or they'll log it, but there's always a little bit of disconnect between what the grants are in the system and what's been documented and what's rerunable. Repeatable. [00:12:45] Speaker A: Yeah, yeah, yeah. So I guess that's another piece when you start looking at the, the pillar for collaboration and self service is if you don't have a good framework around that, then it's a lot harder to do the collaboration. I mean people may think they're doing self service. It's like, yes, it's working fine for them, but it's very, very centered on that particular individual and what they know needs to be run and all that. But then there's not that cross team collaboration, let alone the governance like approval process. Is there an approval process? Or somebody just puts a JIRA ticket in and says, hey, we need analyst number five over here in finance needs access, please go grant them access. Okay. And they execute that and say okay, they close the JIRA ticket. But, but did anybody do any due diligence? What's really the role of analyst 5 and should they have access to all of the finance data or should they only have access to part of the finance data? And that level of granularity is missing in a lot of the homegrown stuff. [00:14:01] Speaker B: Yeah, we certainly have our, at National Grid we have plenty of layers of approvals. If anything, it sometimes gets in the way. But translating those approvals into documented scripts and governed scripts, source control has been the challenge. [00:14:24] Speaker A: I think like you said, in a lot of organizations these controls and things are in place for application development, but have never really been in place. On the data side they might, maybe they used an ELT tool that had version control built into it, but there wasn't any governance. They just went in and the developer went in and drew a couple of different lines and hit save and hit deploy and okay, it's been versioned in the repository but nobody actually looked at it before it got deployed. Right, sure. Yeah. [00:15:06] Speaker B: That layer of approvals and code reviews mixed in is certainly important and often lacking. [00:15:13] Speaker A: Yeah. Oh, looks like. Oh, there you are, you're back. I'm not hearing you. So we lost, lost your audio. Sorry about that, folks. Every once in a while that's the downside of live. Okay, I hear you. [00:15:50] Speaker B: Okay, there we go. [00:15:51] Speaker A: Okay, we're back. Okay, you were in the middle of saying something and you went gone. [00:15:55] Speaker B: Yep. Let me fix my video settings. Okay. Nope, that one made me go away. [00:16:02] Speaker A: That made you go away so your video was fine. [00:16:05] Speaker B: Okay. [00:16:06] Speaker A: Video and your audio. [00:16:14] Speaker B: Yeah, it's gonna. [00:16:15] Speaker A: There we go, There we go. Now I get you back. [00:16:18] Speaker B: So sorry, I bumped a cord. [00:16:20] Speaker A: Don't touch anything. [00:16:21] Speaker B: Not gonna touch that again. [00:16:24] Speaker A: All right, I want to do, I want to do a quick walkthrough on our seven pillars. You've got elt, in the spirit of elt, agility and CI cd, Component design and maintainability, Environment Management, Governance, Security, change control, which we've been talking a lot about, automated testing and monitoring that you mentioned and collaboration and self service. So I guess my question is, do you think that covers it? If everybody ideally were to do all of these things, does that really cover what's going to make us successful in building out our data states? [00:17:09] Speaker B: I think it's a good list. I would probably separate a couple of them. Like, I think governance and security are kind of separate enough at least. But again, I work a lot on the governance side on my day to day, so I see that as kind of more separate from security. [00:17:35] Speaker A: Yeah, yeah, yeah. [00:17:37] Speaker B: And I also kind of think of source control as being very tied to CI cd. So I don't know if I were laying it out, I might do it a little bit differently. But I think you've got all the right points here. It's really half a dozen. One, six and the other. [00:17:55] Speaker A: Yeah, it's kind of what needs to be covered effectively. Right. So is there anything that you think is missing? Do you think every. Do we really have it all covered like you said, other than maybe you could split governance and security into separate things? [00:18:17] Speaker B: I don't know of any major pillar that I would say is missing these days. I often see these types of pillars include something specific for AI and I think there's some merit to that because machine learning, engineering, the process of training and retraining models, is itself kind of its own level of infrastructure and operations. So I think it's valid to have that. [00:18:46] Speaker A: MLOps, right? [00:18:48] Speaker B: Yeah, yeah. So I think that's a valid thing to have, but it doesn't apply to everyone in the same way that most of these do pretty much apply to most companies in most data projects. [00:19:02] Speaker A: So you, you've been a consultant a fair amount in your career as well as Annie, you currently work at National Grid, but how do you approach the buy versus build conversation with them? Because like I know you, obviously you're familiar with data Ops Live, which is obviously a buy scenario. [00:19:22] Speaker B: Right. But yeah, I do certainly see buy as being kind of too quickly dismissed by a lot of teams, companies I'm sure many are familiar with the manager who will balk at $10,000 for a tool to solve a problem, but will happily throw 10,000 person hours at that same problem just because it comes from a different budget. I've certainly seen that where, you know, buying the tool comes out of my team's budget, but paying for an employee comes out of, I don't know, some other visibility of numbers can certainly come into play there. And that's frustrating to see. But of course, buying a tool isn't exactly a zero person hours proposition either. There's certainly valid concerns of getting locked into a tool buying something that you don't actually need. Sometimes you just have a marketing guy trying to sell a tool who's just over the top and it's hard to identify if what they're talking about is just marketing a good sales pitch or if it's actually the tool that you need. So I certainly don't like to see buying a tool as the default option either. In my personal experience, the biggest headaches that I've had have come from trying to split the difference, which is to say paying or buying a team of consultants to build the tool. And I'm sure there's occasions where that can make sense, but I've yet to personally see one. [00:21:08] Speaker A: Okay, good. Yeah, because I haven't, because I know that I've seen that approach as well where they specifically rejected buying a tool that did at least 50% of what we needed and might have covered half of the, half of the seven pillars. We'll say in this case and then they spend even more money. They have consultants that are actually doing the work and writing scripts and all of that. But then when that the budget runs out for them and they leave, you've got all this stuff that nobody knows how to use. It's not well documented. You know, you might have hired, you know, this, you know, little side like hired a tableau expert to build all this stuff out. But then when the tableau expert left, nobody knew how to modify the reports, nobody knew how to modify the displays. I imagine doing like manual, which you've experienced the CICD using, was it Jenkins and Cucumber and all of these other vegetable names. Right. [00:22:17] Speaker B: Or worst case that I've seen is like a data vault implementation that then you lose your data vault expert. And so you now have an entire modeling system that nobody at the company knows how to maintain or make changes to. [00:22:35] Speaker A: And they probably rejected buying one of the data wallet automation tools like Data Vault or Data Vault Builder. [00:22:41] Speaker B: Of course, that would come from the tool budget. We don't have money for that. [00:22:45] Speaker A: Yeah. You know, it's funny, all these years that I've gone through doing this sort of thing and I've certainly experienced it, where we built spreadsheets to even show, here's the cost of the people and how many hours we think it's going to take. Here's the cost of the tool and it's clear roi. Okay, yes. There's ramp up time with a tool. Yes. But there's development time with building it yourself that takes even longer no matter what the developer says, as you know. Right, developers. Oh yeah. I can build that tool in two months. Okay. That's two months to initially write the code and then it's nine months of iterations and testing and it doesn't quite do what we want. And then that guy leaves. Right, right. It's like, okay, now what do we do? But I had never, I guess it had never come through as clear as what you said is, like, it's a different budget. [00:23:37] Speaker B: Yeah, I see it all the time. I've seen so much sense now, like, yeah, the, the guy who's looking at that spreadsheet says this money for the tool comes straight out of my department's budget and so I can't afford it. And then they'll go and look at the spreadsheet of it of, you know, the person hours and where that those salaries come from, like, oh, that's fine, I'll pay as much as you want on that side of the things. It's kind of wild. [00:24:07] Speaker A: Yeah, yeah, that's. And so the buy versus building is. Yeah, it's not a straightforward technical conversation like you say is in some cases it's budgetary, but it may even be political at that point because who owns the budget for that? And yeah, that's a, that's a different challenge. Right. It's like. So he's got to have a very compelling case for, for buying at National Grid too. [00:24:37] Speaker B: Because we deal with utilities and government regulations. We also have a security component to it as well. Getting sign off to even hand over metadata, to even open up channels that could be used for sharing, even if they're not used for sharing. Just opening up those channels requires a ton of security review. And I mean, and that takes time too. So for us, sometimes the default decision is, well, let's just build the thing because then we don't have to even get into the politics. Which is kind of ironic in a way. It ends up taking just as long or longer. But it's interesting how like that default option changes because of non monetary factors. [00:25:29] Speaker A: Yeah, yeah, like I said, thought about that. But you know, a lot of times we talk about people, processes and things thing and technology and the technology in many cases is the simplest of them. It's the people and processes that get in the way of the progress, make it harder to do some of these things. [00:25:51] Speaker B: Somebody doing construction in the room? [00:25:53] Speaker A: Yes, unfortunately. Yeah, I asked them not to work on the thing right behind me. Yeah, they don't do anything there for the next 30 minutes. So. Yeah, yeah, unfortunately, you know, workers got to do what they got to do. Yeah, it's bound to happen again. That's what happens with these live things anyway. So what do you got coming up next? Going to any events you're going to or organizing? [00:26:27] Speaker B: I'm running a few events. So I run the Boston Snowflake user group and we're doing stuff monthly. We've got a few things coming up. We're doing a Snowflake build event workshop. We do socials every month. So that's certainly taking up a fair bit of my get out and do technical things. [00:26:51] Speaker A: So you do little social meetups as well as the technical meetups? [00:26:56] Speaker B: Yeah. Typically every other month is a social. We'll go to a pub or we'll go to a brewery or something and just meet up, get a chance for people to talk. Snowflake provides a little bit of budget for food and drinks. And then on the alternate months we will do workshops, presentations, lightning talks, various structured meetup events. The next one coming up in November is a workshop that me and the other organizer will be actually running. But we bring in speakers for all sorts of topics, all Snowflake related. And then we occasionally also work. [00:27:54] Speaker A: Snowflake user group? [00:27:56] Speaker B: Yeah, yeah. Then we occasionally also work with some of the other user groups. So we've done a collab with the remote data Vault user group. We're looking at doing one with the Woman in Snowflake user group as well. [00:28:11] Speaker A: Great, good. What's the best way for people to get a hold of you? [00:28:19] Speaker B: I'm most responsive on LinkedIn. That's the place where I will actually see your message if you want to get a hold of me. I'm also at the Boston Snowflake user group. We have a Slack channel for that as well. That requires getting an invite sent out, so that's not a great place to start, but LinkedIn. [00:28:42] Speaker A: Great. And yeah, the QR code is right there on the screen for everybody if they want to track you down. All right. Well, given I got this noisy construction going on outside, I think we're going to call this one done here. Thank you so much for being on David and providing your insights. It's interesting. Like I said, I have a new perspective now on the whole buy versus build thing and the different budgets that I hadn't even thought of before. So that's a good one. But and I want to thank everyone else for who's online joining those you watching replay later. Just a reminder, join me again in two weeks and my guest is going to be thought leader and industry analyst Wayne Eckerson, who's also one of the co authors of the Data Ops for Dummies book and the philosophy and seven pillars of True Data Ops. So as always, be sure to like the replays from today's show once they're up on the YouTube channel. Tell your friends about the True Data Ops podcast and don't forget to go to true dataops.org and subscribe to the podcast so you don't miss any episodes. Until next time, this is Ken Graciano, the Data Warrior, signing off. For now.