Ep. 132 Bonus Ep. 3 | Decoding AWS Databases: Choosing the Right Service in Exam Questions
Chris 0:00
Welcome back to the deep dive. If you're a mid level cloud engineer, maybe you're navigating the complexities of AWS certification, or perhaps just looking to sharpen your cloud architecture skills. Well, this deep dive is specifically for you. Yeah, definitely. Today we're tackling one of the most critical and frankly, most asked about areas for any AWS professional, how to confidently choose the right AWS database service. You know, for any given scenario, it really is key. And this isn't just about passing an exam, although we want you to do that too. It's about building resilient, high performing systems out there in the real world. Okay, let's unpack this all right. Our mission today is to arm you with the specific insights you need to ace those tricky scenario based questions and maybe more importantly, make truly informed design decisions
Kelly 0:49
Exactly. Think of it like this. We're here to give you a bit of a shortcut to being well informed, packed with the kind of nuanced facts and maybe just enough humour to
Chris 0:58
keep you hooked. We can try on the humour part, right?
Kelly 1:01
But seriously, it's not enough to just know what a service is. The real trick, especially for certification and definitely for your career, is understanding when and why to use it. It's that classic right tool for the right job principle every single time,
Chris 1:17
absolutely and to make those informed decisions, you first need to know the core capabilities of our key players, right? Got to know the lineup. So we're going to break down four of the main AWS database services that are, well, pretty central to almost any cloud architecture discussion.
Kelly 1:33
Okay, let's start with Amazon relational database service, or RDS. You've probably heard of it at its heart, RDS is a managed service for let's call them traditional relational databases, okay, like SQL Exactly. This is where your data is meticulously organised. Think tables, rows, columns, typically enforcing a rigid schema and relying heavily on Structured Query Language. SQL got it and RDS supports a whole array of popular engines like MySQL, PostgreSQL, Oracle, SQL Server, more ADB, and importantly, Amazon Aurora, which, well, we'll definitely get into next is also an option under that RDS umbrella. So when we talk about scaling for standard RDS, we're primarily talking about vertical scaling, meaning, meaning you increase the instance size, you add more CPU, more memory, to handle heavier loads. Think of it like upgrading to a bigger, more powerful single server,
Chris 2:24
right? So RDS basically takes the operational burden out of managing those traditional SQL databases. It's kind of like having a DBA on demand, but without the full time cost. That's a good way
Kelly 2:34
to put it. It handles patching, backups, failover, all that stuff. So what's next in our lineup? Next
Chris 2:39
up we have Amazon, DynamoDB.
Kelly 2:42
Ah, DynamoDB. Big difference here? Yes,
Chris 2:46
this is AWS. Is fully managed, highly available NoSQL database service. And that NoSQL part is really key, because it means it supports flexible schemas, very data storage models, like key value pairs or documents,
Kelly 2:59
so not the rigid tables and rows like RDS, exactly
Chris 3:03
unlike relational databases, where you often define all your relationships up front, DynamoDB offers Well, incredible agility. It's designed for truly massive workloads. We're talking millions of requests per second, trillions of rows, hundreds of terabytes of storage. It's built for scale. Its defining characteristic is lightning fast, consistent performance, often with sub millisecond latency on data retrieval, even at extreme scales. That's fast, it is and for capacity, you've got choices. You can opt for Auto Scaling, where capacity adjusts dynamically as needed, which is great for unpredictable loads. Or you can use provisioned throughput, where you specify your exact Read, Write capacity units, rcus and wcus up front, so you get
Kelly 3:43
control over cost and performance if you know your workload precisely, that control gives you predictability. Okay, massive workloads and sub millisecond latency for DynamoDB definitely makes it sound like a powerhouse. Now, you mentioned Amazon Aurora earlier under the RDS umbrella, it's also relational so how does it differentiate itself from, say, standard RDS with my school or PostgreSQL? Great
Chris 4:06
question, and this is really where Aurora shines. Amazon. Aurora is AWS proprietary relational database service, but it's designed for cloud native performance. It's fully compatible with my school and PostgreSQL, which is honestly fantastic for migrations, so easy to switch over relatively Yes, but the real game changer is its architecture. Aurora actually separates its compute and storage layers. It offloads all the IO operations to this distributed, self healing storage service that automatically scales up to 128
Kelly 4:37
terabytes. Interesting. So the database engine itself isn't handling the storage directly, right? It means
Chris 4:43
your database instance isn't bogged down managing data blocks. It can focus purely on query processing. And this unique design, this fundamental reimagining of how a relational database operates in the cloud, is why it boasts up to well, five times faster performance than standard MySQL. And three times faster than standard PostgreSQL. Okay, that's a significant jump. It really is. And for scalability, Aurora also supports up to 15 read replicas. That's way more than the typical five for standard RDS engines. And these replicas can also span across availability zones for even greater resilience, 15
Kelly 5:16
replicas. Wow. That decoupled architecture and the performance boost that sounds like a massive advantage for relational workloads, especially the read heavy ones,
Chris 5:23
definitely and finally, what about Amazon Redshift? How does it fit into this whole database puzzle? Because it sounds different. Again,
Kelly 5:30
it is fundamentally different from the others we've discussed. Amazon Redshift is a fully managed, petabyte scale cloud data warehouse service. It's specifically optimised for analytical processing, like bi queries
Chris 5:44
or reporting data warehouse, okay, not transactional like the others, primarily
Kelly 5:49
analytical, yes. And it uses Columnar storage, which is a key differentiator, unlike traditional row based storage, which is great for fetching a whole record quickly, Columnar storage. Store is data by column. How does that help? Well, think about analytical queries. They often scan and aggregate only a few columns across millions or even billions of rows. Columnar storage makes this incredibly efficient, because you only read the columns you need, drastically reducing IO and speeding up those analytics. Ah, that makes sense, and Redshift operates as a managed cluster service. It automates the provisioning, the scaling of those clusters based on your processing needs. It supports both temporary clusters, maybe for ad hoc analysis, and Long Term CLusters for continuous bi requirements.
Chris 6:31
All right, now that we have a, I think, a solid understanding of our players, RDS, DynamoDB, Aurora, Redshift, here's where the rubber meets the road. The fun part Exactly. It's not just about what these services are, but the crucial why and when to pick one over another in real world scenarios, and especially on your certification exam, this is really the heart of making smart architectural
Kelly 6:54
decisions. Couldn't agree more, it's all about identifying the core requirements of your application or your workload. There's really no one size fits all database. So let's start with maybe the most common type transactional workloads where data integrity and consistency are absolutely paramount.
Chris 7:09
Okay, so when would you confidently lean towards RDS or maybe Aurora? For these transactional
Kelly 7:16
needs, you'd choose RDS or Aurora when your application demands a city compliance, remember acid, Atomicity, Consistency, Isolation, durability,
Chris 7:25
right the bedrock of relational
Kelly 7:27
databases Exactly. Think about financial transactions, inventory management systems, CRMs, anywhere, every single update must be perfectly consistent and reliable. They also Excel when you need complex SQL queries, multi table joins, that kind of strong transactional consistency, these are really your go to candidates for traditional enterprise
Chris 7:45
applications. Makes sense. And what about ensuring high availability and disaster recovery for these mission critical relational databases? How do you handle that right
Kelly 7:55
critical question for high availability within a single region, you immediately think of multi-AZ deployments. This provides synchronous replication to a standby instance in a different
Chris 8:05
availability zone, synchronous, so the data is always consistent between the two. Yes,
Kelly 8:09
synchronous. And what's crucial here is the automated failover. If your primary instance fails for any reason, hardware failure, AZ outage, AWS, automatically switches your endpoint over to the standby, often in just minutes, with minimal disruption to your application. It also means backups are typically taken from the standby, which minimises the performance impact on your active database.
Chris 8:32
Okay, multi. AZ, for resilience, what a performance especially if you have lots of reads.
Kelly 8:38
Ah, for read heavy workloads, you implement read replicas now, these use asynchronous replication, allowing you to scale out reads significantly by directing read traffic to one or more replicas, asynchronous so potentially a slight delay, a small replication lag, yes, but usually very brief. And remember, Aurora truly shines here with support for up to 15 read replicas, and it even gives you a single reader endpoint that automatically load balances connections across all available replicas. Standard RDS typically supports up to five read replicas,
Chris 9:10
so multi-AZ for resilience against AZ failures, protects your writes, read replicas to handle high read traffic, scale up performance. That's a clear distinction and security for these relational databases, anything specific to keep in mind, definitely
Kelly 9:24
for security, both RDS and Aurora support encryption at rest using AES 256 bit encryption managed by AWS Key Management Service or KMS. And the good news is this has a negligible performance impact. But here's a crucial exam gotcha, and a real world one too. Encryption at rest can only be enabled when you first create the database instance, only at creation, only at creation. You cannot enable it later on in existing unencrypted instance, nor can you disable it once it's enabled. If you forget or your requirements change, your only option is to take a snapshot create a. New encrypted instance from that snapshot, migrate your application connections and then delete the old one. It's a significant operational hurdle you want to avoid.
Chris 10:09
Wow. Okay, that's a critical detail, definitely something that could trip someone up in an exam, or worse in production. What about data in transit?
Kelly 10:16
Yeah, good point. Ssltls encryption is also available for in transit connections, protecting data as it moves across the network between your application and the database. Always recommend
Chris 10:26
it got it now shifting to the high scale side. When would DynamoDB be the better choice, especially given its non relational nature, feels like a different world. It
Kelly 10:36
is kind of a different world. Yeah, you'd choose DynamoDB when you need extreme like virtually limitless scalability, incredibly low latency data access, things single digit milliseconds or less, and your data naturally fits a non relational or NoSQL model.
Chris 10:51
So key value document right? While it offers
Kelly 10:55
flexible schemas, which is powerful, that freedom comes with a need for careful access pattern design, unlike relational databases, where you can often easily add new ad hoc query patterns later, DynamoDB performs best when your access patterns are known upfront. That's because how you design your primary key and any secondary indexes directly impacts performance
Chris 11:14
and cost. Ah, so you need to think ahead more about how you'll query the data
Kelly 11:18
precisely it achieves its massive scale and low latency, partly by allowing eventual consistency for most reads by default, this means a recent write might not be immediately visible across all the underlying data replicas, but it usually propagates very quickly, often within milliseconds. You can request strongly consistent reads, but they have higher latency and consume more read capacity. This eventual consistency trade off is often crucial for highly distributed systems, where demanding strict, immediate consistency across all replicas would introduce significant bottlenecks.
Chris 11:52
Okay, eventual consistency is the default for speed and scale. Can you give us some concrete examples, like, what kind of real world use cases? Scream? DynamoDB?
Kelly 12:00
Absolutely. DynamoDB is perfect for things like high performance gaming applications. Imagine a real time leaderboard with millions of players, or storing player profiles accessed constantly. It's also ideal for IoT devices generating Constant Data. Think millions of sensors sending updates every second. You need somewhere to land that data quickly and reliably. Right that Firehose of data exactly, or for large scale web and content management systems storing massive amounts of user data, session state or content catalogues. And many mobile applications leverage DynamoDB for their back ends, often integrating seamlessly via something like API Gateway for a truly serverless architecture,
Chris 12:39
okay, those are great examples. And are there any specific features unique to DynamoDB that are worth knowing, maybe for optimising performance or integration? Definitely
Kelly 12:49
a couple to highlight. First, DynamoDB streams. This provides an ordered flow of information about changes, creates, updates, deletes, happening to items in your table. This allows for real time processing. For instance, you could trigger a Lambda function to react to changes, maybe to remove sensitive PII data as soon as it's written, or to replicate data to another system, like a data lake or a search
Chris 13:10
index, so reacting to data changes in real time. Cool. What else? Then
Kelly 13:14
there's DynamoDB accelerator or DAX. DAX is essentially a fully managed, highly available in memory cache specifically for DynamoDB. It sits in front of your DynamoDB table and can reduce read response times from milliseconds down to microseconds for eventually consistent reads, it's transparent to your application, so you often don't need to change your code, but it dramatically reduces the read load on your underlying DynamoDB table for those cacheable
Chris 13:40
items, microseconds. Okay, so DAX for boosting read performance even further, and you mentioned capacity planning earlier,
Kelly 13:46
right? Remember to choose provision throughput if your read and write capacity requirements, rcus and wcus are fairly predictable. This gives you cost predictability. But use Auto Scaling when demand is unpredictable or spiky. Auto Scaling will automatically adjust your provision capacity up or down based on actual traffic, helping you save costs during quiet periods and handle sudden bursts without throttling.
Chris 14:08
That makes a lot of sense. So DynamoDB for speed, scale flexibility, especially when your access patterns are pretty well defined. Now switching gears completely again, back to analytical workloads. When should we be thinking Amazon, Redshift? Yeah,
Kelly 14:24
back to analytics. Redshift is really your purpose built solution here. Its primary use case is to analyse vast data sets. Think business intelligence, complex reporting dashboards or feeding data into machine learning models. It's engineered to aggregate and process that data very efficiently, mainly because of that Columnar storage we talked about right optimised for scanning columns Exactly. And Redshift integrates really nicely with other AWS services. You'll often find it connected to Amazon S3 with S3 acting as a massive, cost effective data lake. Source data gets loaded from S3 into Redshift for analysis. It also works well with Amazon Q. Nieces for ingesting streaming data directly, and AWS Glue is often used for the Extract, Transform, load or ETL operations needed to prepare and move data into Redshift. It even supports querying data directly in S3 using Redshift spectrum blending data lake and data warehouse approaches.
Chris 15:15
Okay, so Redshift is the heavy hitter for analytics and bi playing well with S3 and other data services that covers the main players and their tweet spots pretty well. Now, while we've touched on security and cost for individual services, what about some overarching security and cost considerations that apply across all these AWS databases? Things You should always be thinking about,
Kelly 15:36
absolutely critical security should never, ever be an afterthought. It's foundational for all AWS databases. You should be utilising AWS Identity and Access Management IAM for fine grained access control, so controlling who can do what precisely IAM lets you define exactly who or what, whether it's a human user, an EC2 instance, via an IAM role or a Lambda function, can access your database and what actions they're allowed to perform. It's also considered best practice and strongly recommended to leverage AWS Secrets Manager to handle database credentials, API keys or other application secrets instead of hard coding them. Please, yes, instead of embedding credentials in your code or configuration files, Secrets Manager securely stores them, but crucially, it also supports automatic rotation of credentials for supported databases like RDS and Redshift. This significantly minimises operational overhead and drastically reduces the risk associated with long lived potentially compromised credentials. Automatic rotation sounds like a lifesaver. It really is, and for private connectivity, making sure your database, traffic doesn't traverse the public Internet. Always consider VPC endpoints. There are two main types relevant here. A gateway. Endpoint is used specifically for S3 and DynamoDB, and it's often free. It allows resources within your VPC to reach these services without needing an internet gateway or NAT device. For most other AWS services, including RDS, Aurora and Redshift connections within AWS, you'd use an interface endpoint which uses elastic network interfaces, Enis in your subnets. These provide private, secure connectivity powered by AWS. Privatelink. Okay,
Chris 17:11
so IAM for permissions, secrets, manager for credentials, VPC endpoints for private network access. Those seem like non negotiable best practices. What about optimising costs across these services? Because, let's be honest, those database bills can add up fast if you're not careful.
Kelly 17:28
Oh, absolutely. Cost optimization is always a huge factor. First principle, always aim to right size your resources, monitor your usage, understand your actual needs, and choose instant sizes or provisioned capacity that matches your average use, not just your absolute peak leverage. Auto Scaling where it makes sense, like with DynamoDB or Aurora serverless to handle variability efficiently. So don't over provision, just in case. Try not to, if you can avoid it with scaling mechanisms for databases with continuous, predictable demand over long periods, think one or three years definitely look into reserved instances where RDS, Aurora, Redshift and ElastiCache, which often sits alongside databases, rushes, offer significant discounts, often 40, 70% compared to on demand pricing. In exchange for that commitment,
Chris 18:14
that's a big saving for steady workloads, huge now
Kelly 18:18
for flexible, interruptible workloads, maybe batch processing jobs, data analysis tasks that can be stopped and restarted. You might consider Spot Instances for the compute nodes, for example, and some Redshift configurations, or for EC2 instances accessing the database. Spot offers massive savings, but you have to design for interruption. You'd never use spot for your primary stateful database instance itself, but potentially for associated processing
Chris 18:43
right you spot with caution and only where appropriate Exactly. And
Kelly 18:47
one more architectural point on cost, remember that decoupling your application components using services like Amazon, SQS, simple queue service can also help right size your database servers by putting a queue in front of your database you can buffer incoming requests during peak loads. This allows your database to process work at a steadier pace, meaning you might be able to use smaller, less costly instances rather than over provisioning just to handle those occasional spikes.
Chris 19:12
Ah, SQS is a shock absorber. That's clever. Okay, this has been a truly deep dive into making informed database decisions. I feel like anyone listening is now much better equipped with those nuggets of knowledge for critical exam scenarios and those real world architectural challenges. Hope so. So what does this all mean for your certification journey? Here's where we pivot, maybe to some rapid fire strategy for navigating those tricky scenario based questions you'll definitely encounter. Yeah,
Kelly 19:43
let's do it. The key thing the exam tests is your understanding of how these technologies work together and how to apply them to solve specific problems. It's usually less about clicking buttons in the console and more about choosing the right service or feature for the requirements given in the scenario. So
Chris 19:59
think about. Purpose of the application, the workload requirements exactly and always,
Kelly 20:03
always look for keywords in the scenario description. They are crucial clues that will point you towards the appropriate AWS service or configuration.
Chris 20:11
All right, let's run through some common scenarios. Ready scenario an application is experiencing high latency, and the monitoring shows it's due to heavy read activity on the database. What's your media thought?
Kelly 20:22
RDS or Aurora, read replicas. That's the classic solution to offload read traffic, scale horizontally for reads and improve performance without impacting the primary write instance.
Chris 20:33
Okay. Next scenario, you need a highly available database, one that's resilient to an entire availability zone failure within a single AWS region,
Kelly 20:42
RDS or Aurora multi-AZ deployment that synchronous replication and automatic failover to a standby instance in a different AZ is designed precisely for this requirement, maintaining continuous database operations perfect.
Chris 20:55
What if you're storing large amounts of unstructured or maybe semi structured data coming in from millions of IoT devices, and the throughput is unpredictable, but potentially extremely high.
Kelly 21:06
That screams DynamoDB. Its massive scalability, flexible, no SQL model and consistent, low latency make it ideal for ingesting high volume, unpredictable data streams like IOT sensor data or clickstream logs,
Chris 21:18
right? How about analysing vast historical data sets, maybe terabytes or petabytes, or building out a data lake for business intelligence and complex reporting
Kelly 21:27
Amazon Redshift often used in conjunction with Amazon S3 as the primary data lake storage. Redshift is purpose built for these large scale analytical queries and data warehousing tasks, leveraging its Columnar storage for speed.
Chris 21:40
Okay. Scenario, the requirement is to minimise the operational overhead of managing database credentials securely for applications running on EC2 or Lambda,
Kelly 21:50
AWS, Secrets Manager, it's the absolute best practice for securely storing and automatically rotating database credentials, API keys, etc, takes the burden off developers and ops teams,
Chris 22:00
good one ensuring messages related orders maybe are processed in strict first in first out order exactly once, with no duplicates allowed in a distributed system, perhaps before they hit the database
Kelly 22:11
Amazon SQS, FIFO queue. The FIFO first in first out queue type is critical for maintaining message order and providing exactly once processing semantics often used to reliably decouple application components. Standard SQS queues offer high throughput, but only best effort. Ordering.
Chris 22:28
Okay? FIFO for order. Now you need to migrate an extremely large data set, maybe 50 terabytes, from your on premises data centre, to AWS, perhaps landing it in S3 for analytics. It's time sensitive, and you can't saturate your existing internet connection
Kelly 22:44
for weEKS. AWS Direct Connect, this provides a dedicated, private, high bandwidth network connection between your premises and AWS bypassing the public Internet. It ensures fast, secure and reliable data transfer for these large scale migration scenarios, or maybe AWS Snowball, if it's a one off physical transfer, but for ongoing, large transfers, Direct Connect,
Chris 23:04
right direct connects for that dedicated pipe, how about securely accessing S3 or DynamoDB from EC2 instances located in a private subnet, without that traffic having to go out to the public internet
Kelly 23:14
and back a gateway VPC endpoint for S3 or DynamoDB, this allows private, secure connectivity, directly from your VPC to these specific services, keeping all traffic within the AWS network. Backbone
Chris 23:27
excellent, optimising storage costs for data in S3 where the access patterns are unpredictable or maybe just unknown. You don't know how often you'll need the data. S3
Kelly 23:36
intelligent tiering. This storage class automatically monitors access patterns and moves your data between frequent and infrequent access tiers and archive tiers, if configured to optimise costs without any performance impact or operational overhead for you, it's brilliant for unknown or changing patterns.
Chris 23:51
Love that one last one protecting critical data stored in S3 from accidental deletion by users or applications.
Kelly 23:58
Enable S3 versioning on the bucket. This keeps multiple versions of an object. Then for extra protection, enable MFA Delete. This requires multi factor authentication for permanently deleting an object version or changing the buckets versioning state, adding a strong layer of security against accidental or malicious deletions. And remember overarching all these scenarios, read the question carefully. Don't just memorise definitions. Focus on the purpose and the specific requirements. Mentioned keywords like real time, high performance, cost effective, highly available, structured data, unstructured data, transactional workload, analytical workload, or decoupled components, these are your best friends and guiding you straight to the right service or feature,
Chris 24:42
that's incredible advice. Seriously, anyone listening, you are now so much better equipped with both the nuanced knowledge and those strategic insights to tackle those critical exam scenarios and, more importantly, make truly informed database decisions for your AWS journey.
Kelly 24:56
Yeah, hope this really helps remember choosing the right database. It sounds simple, maybe, but it's perhaps one of the most fundamental decisions you'll make when building scalable, resilient and cost effective cloud architectures. It's really not just about passing an exam, although that's great, but it's about truly understanding the tools you have at your disposal to build genuinely well architected solutions that meet complex, real world demands. So
Chris 25:20
true, which leads to a final thought for you to mull over as cloud engineers, how do these fundamental database choices made early in the design process influence the entire security and performance posture of complex, multi tiered applications down the line? Think about those ripple effects if you choose poorly early on, something to think about until our next deep dive.
