BYTE the Cloud

Chris 0:00
All right, cloud engineers today, we're going deep on something that can make a real difference in how we work.

Kelly 0:07
Oh yeah, the

Chris 0:08
AWS cost and usage report.

Kelly 0:10
Okay, I'm listening.

Chris 0:11
I mean, we could build the coolest things in the cloud, right? Sure. But as we all know, it's got to be smart too, absolutely. And that comes down to, well, cost.

Kelly 0:19
Definitely, keeping an eye on those costs for sure,

Chris 0:23
exactly, knowing where every dollar is going in AWS, yeah. So this deep dive, it's gonna be clutch, especially if you're thinking about those AWS exams, you know, like the Solutions Architect Associate, yeah, yeah. Makes sense. This service pops up all the time there. It does. It does. So what's really cool about this cost usage report is it gives you, like, an x ray vision into your AWS spending x ray vision, okay? I like that. Think of it as, like, you get a detailed receipt for everything you're using, every little thing. Oh, wow, broken down, like, to the resource level, and it goes right to your S3 bucket, okay? And you can, like, totally customize it, yeah. Oh, really. Daily, hourly, you name it. That's handy. You can even see by like service or tagged resources, whatever you need. Wow, that's

Kelly 1:10
that's a lot of flexibility. It's powerful stuff. It

Chris 1:12
helps you make those smart choices, how you use your resources, how you design your architecture, yeah,

Kelly 1:17
yeah, no, that makes a lot of sense. So it's not just like saving money. It's about making smarter choices with the resources we've got.

Chris 1:24
Exactly like last month, my project went like, way over budget. This would have to figure out why? Oh, for

Kelly 1:30
sure. Or let's say you're moving a big app to the cloud, this report tracks your costs every step of the way. Oh, I see, yeah, you can catch any of those surprise spikes, and make sure that you know your architecture is really as efficient as it can be,

Chris 1:47
right? No more nasty surprises on the bill, exactly. I'm starting to see how, how useful this could be, yeah, not just for me, but for my whole team. Definitely, like say, I can prove a feature is a total flop. Nobody's using it. Well, then maybe we can shift our focus to, you know, something more impactful, something people actually use exactly

Kelly 2:05
that's the strategic thinking. Or imagine you're on the security team, you see some wacky data transfer costs.

Chris 2:13
Okay, that could be a red flag, oh, like some data is walking out the door

Kelly 2:16
exactly this report. It gives you that close up view to you know, see if anything's fishy.

Chris 2:23
Okay, I'm sold on the why. Now let's get into the how. All right, so this report lives in my S3 bucket. Yeah, we know it's customizable, right? What does that look like in practice? Sure. What kind of formats are we talking about? Well,

Kelly 2:38
the report itself can be a few different things, like a CSV or parquet file, okay, okay, those are good if you're already working with those data tools, you know, yeah, for analysis and all that, exactly, and you get to call the shots. Oh, how? So you decide how often you want this report, okay, how much detail you need, and where it ends up in this three right? Once it's set up, AWS, does the rest? Just delivers it to your

Chris 3:01
bucket. So it's like setting up a data feed. But it's all about me.

Kelly 3:05
That's a good way to think about it.

Chris 3:06
I like it. Let's get down to brass tacks. What are the key features that make this thing so powerful? One

Kelly 3:12
of the most impressive things is how detailed the data is. You can see the costs broken down for each individual resource? Well, that's granular. It is. You're not just getting a lump sum for, say, all your EC2 instances, right, right? You know exactly which instance is costing. What that's

Chris 3:30
that's gotta be crucial for optimizing. Oh, absolutely. Can you play with that data, slice and dice it in different ways?

Kelly 3:37
Absolutely. You can group the data by service, by resource type, even by custom tags. Oh, cool. Say you tagged all your resources for a specific project. You can instantly see the costs just for that project that's slick. It's really helpful for internal billing and things like that.

Chris 3:55
Okay, I'm seeing the power here, yeah, but you mentioned needing some data skills, right? Can you talk about, like, how we actually use this report once it hits S3 Sure,

Kelly 4:05
since it lives in S3 you'll want to know your way around some data tools. Yeah, makes sense. But the good news is, AWS has a whole set of tools to help you out. Oh, good for querying and analysis. You've got things like Athena Glue and EMR, yeah, I've heard of those, and you can crunch those numbers without managing any servers. Nice. So

Chris 4:27
less headache for us. Exactly, what about visuals? Can we, like, make charts and graphs from this data? Something easier to digest. You're

Kelly 4:34
talking about dashboards, right? Yeah, yeah. Amazon, QuickSight is your best friend for that. Okay, it plugs right into your S3 bucket, and you can make dashboards and reports that are super easy to understand, sweet.

Chris 4:45
So we've got the raw data, the tools to analyze it, and then we can make it pretty with visuals. That's the idea. It sounds like AWS has really given us everything we need to be cost conscious. You know they have, but I know what our listeners really want. Oh, yeah, exam prep. Up. Are you ready to put this knowledge to the test? Get me with it. All right, let's pair up some questions and see what we can learn. Let's

Kelly 5:06
do it. Okay, picture this. You're in the hot seat taking the Solutions Architect Associate exam. Oh, and this question pops up, which of the following is not a valid use case for the AWS cost and usage report.

Chris 5:22
Oh, man, the pressure's on. I'd definitely be taking a deep breath right about now, thinking back to what we talked about. You know, yeah, the report's all about cost and usage data. That's got to be my guiding principle here,

Kelly 5:33
exactly. And let's say the options are a tracking the cost of a cloud migration. Project B, monitoring real time network traffic for security threats. C, optimizing resource utilization for a web application, okay? And D, analyzing usage patterns to, you know, inform future development. All

Chris 5:55
right. So, Option A, tracking cost during a migration. Yeah, talked about that. It's super valuable as you move stuff to

Kelly 6:01
the cloud. Definitely, you want to make sure you're on track your architecture is lean and mean and you're not blowing the budget. Right? Option C, optimizing a web app that fits too Right? Yeah,

Chris 6:11
yeah. The report shows you, if you're like, using too many resources exactly, helps you right size those instances and options will make sense. You see what features people are actually using, yeah, that helps you focus on the right stuff. Prioritize development Absolutely. So the odd one out seems to be B monitoring network traffic. That's

Kelly 6:30
the one. Remember, this report is all about cost and usage, not network traffic, right, right? For that, you'd need something like VPC flow logs. Okay, they love to throw those curve balls in the exam. You know, word it so it sounds right, but it's not the core function. Yeah,

Chris 6:45
those tricky questions. So the answer is, B. B is out. Makes sense. I feel like I'm getting the hang of this. What's next? Let's

Kelly 6:54
try another one. Your company needs to analyze cost data, but in real time, they need to make quick decisions. What service would you use along with the cost and usage report?

Chris 7:04
Hmm, okay, so this is about combining things, it is. And the key word here is real time. We know this report has like a little delay, so what would give us that instant view?

Kelly 7:15
You're thinking along the right lines. So your options are AWS Athena, AWS Kinesis, data streams, Amazon, QuickSight and AWS Cost Explorer,

Chris 7:26
okay, so Athena and QuickSight, they're great for the rapport, but not real time exactly.

Kelly 7:30
They analyze it once it's in S3 right? And

Chris 7:32
Kinesis, that's for streaming data, not history, right? So that leaves cost

Kelly 7:38
explorer, bingo. Cost Explorer is all about that real time view. So the answer is, d it is. It's like having a live dashboard for your AWS spending, and then the cost and usage report gives you all the details historically. Oh, that makes sense. It's like the best of both worlds. It is understanding what each tool is good for and how they work together, right? All right. How about this? You're setting up the cost and usage report for a brand new project, and you need the most detailed cost breakdown you can get. What data granularity setting would you pick? Okay,

Chris 8:08
so this is all about the right level of detail. My options are daily, hourly, resource level and monthly. And we want the nitty gritty, right? You got it.

Kelly 8:20
We want to see as much as possible.

Chris 8:22
So the answer has to be resource level.

Kelly 8:25
That's the one. It goes down to each individual resource. You see every little bit of cost, awesome,

Chris 8:30
daily and hourly. Those give you some insight. Yeah, but monthly is way too zoomed out for this too high

Kelly 8:34
level. Yeah, it's like choosing the right lens for a camera. You need the close up exactly.

Chris 8:39
So it's C, resource level, E, it is.

Kelly 8:42
Now, imagine your company's got tons of AWS accounts. Oh, yeah, that's pretty common, right? It is, and they need to look at all the cost data in one place. What's the smartest way to do that?

Chris 8:52
We talked about combining reports manually, and it sounded like a nightmare. It can be. There's got to be a better way. There

Kelly 8:58
is, remember AWS Organizations, yeah, vaguely, it lets you organize all your accounts like a hierarchy, okay, and manage them all from one place, including cost data.

Chris 9:08
Oh, I see. So we can bring all that data into our main account, exactly the

Kelly 9:12
master account of your organization, and then you can use tools like Athena and QuickSight to analyze it all way

Chris 9:18
more manageable than separate reports for everything, much more efficient. Okay, that makes a lot of sense. Yeah. So the answer is, using AWS Organizations, yep,

Kelly 9:26
especially for those bigger companies with tons of accounts, all right, what's the next cost puzzle we gotta solve? Let's say you're on cost anomaly duty. Gotta spot those unexpected spikes, okay, big jumps in spending for a specific service. Yeah. How would you set that up?

Chris 9:42
So we're being proactive here, right? Exactly. We

Kelly 9:44
want to catch those spikes before they become a problem. We

Chris 9:47
need some sort of alarm system, something automated. You

Kelly 9:49
got it. And this is where two AWS services team up, AWS Budgets and AWS cloud. Watch.

Chris 9:58
Okay, I've used both. But how do they work together? Think of CloudWatch

Kelly 10:01
as your watch dog. It's watching your spending non stop for that specific service, and you set a limit if the spending goes over. CloudWatch barks, barks, meaning it triggers an alarm, and that's where budget steps in. Okay, I'm with you. You link that alarm to a budget you've already set up with your spending limits. Okay, so alarm goes off, budget sends you an alert.

Chris 10:25
Wow, pretty clever. It's like a two step system. It is, CloudWatch sees the problem. Budgets tells you about it. So the answer is, combine CloudWatch and budgets. You

Kelly 10:35
got it. That kind of system can save you from a lot of headaches later on. Definitely worth setting up. All right. Next scenario, you're on a project that needs real time cost visibility. You gotta see the spending as it happens. Okay, this is different, not historical analysis this time. Nope. We need something live up to the minute, gotcha. And the tool for that is AWS Cost Explorer.

Chris 10:56
Oh, yeah, the dashboard we talked about that earlier, perfect for this, right? It is

Kelly 11:01
real time view of all your costs and usage.

Chris 11:04
So for real time tracking, the answer is, Cost Explorer. Cost

Kelly 11:07
explorer, it is gives you that instant insight so you can react quickly. Nice. Okay, let's shift gears a bit. Data formats your company wants to analyze this report, but with a big data tool, big data, okay, what format would give them the best performance?

Chris 11:24
So we need a format that can handle a ton of data quickly. Exactly.

Kelly 11:27
Your options might be CSV, JSON and parquet.

Chris 11:31
Okay, so CSV, that's good for smaller stuff, but for big data, it gets clunky, right? It

Kelly 11:36
does and JSON, it's more structured, but still not ideal for tons of data, right? So that leaves parquet. Parquet, yeah, that's made for big data, right? It is. It's a columnar storage format. I

Chris 11:47
remember reading about that, yeah, it stores data by column, not row, which makes it faster.

Kelly 11:52
You got it, and it compresses the data so you save on storage too. Oh, nice bonus.

Chris 11:56
Yeah, so for big data analysis of the cost and usage report, the answer is parquet.

Kelly 12:01
Parquet is the winner. It's all about using the right tool for the job. Definitely.

Chris 12:05
Adding that to my mental notes, yeah. Okay. What else we got? All right, common

Kelly 12:08
EC, two problem. You've got instances running, but they're not being used much. Yeah, that happens. How do you fix that and save some money?

Chris 12:16
We got to right size those instances, right? You got it. Pick the instance

Kelly 12:19
type that actually matches the workload. But

Chris 12:22
where does the cost and usage report come in? Remember, it

Kelly 12:25
gives you all those usage details, right, right? You can see the CPU utilization, memory usage, all

Chris 12:31
of it. Oh, so we can see if we're using way more than we need exactly.

Kelly 12:34
And then you can pick a smaller instance type save some money, right? You could even use Auto Scaling, you know,

Chris 12:41
yeah, to adjust the instance size automatically based on real usage patterns. Part. So the answer is, use the report to see how much we're actually using those EC2 instances, and then

Kelly 12:51
right size them based on the data. Sense. Okay, let's talk security. Your company's got a rule, all data has to be encrypted. Uh huh, got to protect that data. How would you make sure the cost and usage report is encrypted

Chris 13:02
too, we need to treat it like any other sensitive data, right? Exactly,

Kelly 13:05
encryption all the way in transit and at rest.

Chris 13:08
So what are our options? Well,

Kelly 13:11
AWS gives you a couple of ways to do this. First, you can encrypt your S3 bucket server side encryption. Okay, so AWS encrypts the report automatically when it gets delivered, so

Chris 13:22
even if someone got into our bucket, they couldn't read the report without the key exactly,

Kelly 13:26
good protection for data at rest. What if we want more control, though? Then you bring in the big guns, AWS, KMS, KMS,

Chris 13:34
right? The key management service, yep. You create your own encryption keys, manage them yourself and use those to lock down the cost and usage report precisely.

Kelly 13:42
KMS gives you that granular control who has access all that.

Chris 13:46
So we can choose between the S3 encryption or KMS for more fine grained control. That's

Kelly 13:52
right, both options can help you sleep soundly at night knowing your data is safe. Okay, good to know what's next. Last one for today, cost allocation. Tags. Okay, your company tags all resources with the project they belong to. Uh huh, smart. How do you make sure the cost and usage report shows that breakdown by project,

Chris 14:10
right? So we can see how much each project is spending. How do we configure the report to show us that it's

Kelly 14:15
actually super easy. When you set up the report, you just say which tags you want to

Chris 14:19
include. Oh, that's it. So we just tell it to include the project tag, and it'll group the cost accordingly, exactly.

Kelly 14:25
No complex setup. Just tell the report what you want to see

Chris 14:29
that's fantastic, so simple, but so powerful it

Kelly 14:33
is, and that way you can track those project costs make sure nothing gets out of hand. Yeah, it keeps things organized and helps you stay on budget,

Chris 14:40
it's amazing how much control you have with this report.

Kelly 14:43
It really is a game changer, not just for individuals, but for whole organizations. Yeah,

Chris 14:47
I'm seeing that now like this is the bedrock of any serious cost management strategy on AWS, wouldn't you say

Kelly 14:54
absolutely, it's the foundation the data you need to make those smart decisions about your spending.

Chris 14:59
And like we've seen today, it works with a ton of other AWS services to create some really powerful solutions the

Kelly 15:07
whole ecosystem. I feel like I've learned a ton today. Good, good. That's what we like to hear. What's

Chris 15:12
like, the one thing you want our listeners to remember about this service. Don't

Kelly 15:15
just like, file this report away and forget about it, right? It's the gold mine of information, you know, dig into it, analyze it, play around with the customization options. Yeah, you'll be surprised at what you learn about your AWS environment. Great

Chris 15:29
advice. Unlock the potential of this report, and you're on your way to becoming a cost ninja. Cost

Kelly 15:36
ninja. I like that, right?

Chris 15:38
And for those of you studying for those AWS exams. Well, this is a must know. It pops up all the time, so make sure you know it inside and out, definitely. Well, thanks for joining us on this deep dive into the AWS cost and usage report. It's been fun. I feel way more confident about my AWS skills now, and I hope you do too. I hope so. Yeah, we'll be back next week with another deep dive into the world of cloud computing, always learning new stuff, until then, keep learning, keep building and keep those costs in check. See you next time.