BYTE the Cloud

Chris 0:00
Hey everybody, and welcome to another deep dive. Today, we're going to be talking about AWS Step Functions, and this is something that's become pretty essential in the serverless world and for modern cloud applications. Yeah,

Kelly 0:13
it's a really interesting service and really useful for orchestrating complex workflows, especially if you're working with serverless architectures. Yeah, absolutely.

Chris 0:22
And I've been, I've been given some really interesting material to dig into for this deep dive. So let's just jump right in. Okay, for those who might be new to step functions, can you give us just a high level overview of what it is?

Kelly 0:37
Yeah, sure. So AWS Step Functions, it's a serverless service that lets you design and manage workflows visually. You can kind of think of it as a choreographer for your application coordinates all the different steps and services that are involved in a complex process. Okay?

Chris 0:51
I like that analogy a choreographer, so it kind of brings it all together, makes it all flow smoothly. Now, I always like to ask for a real world example. Can you give me a real world example of how step functions would be useful. Let's

Kelly 1:02
say you're building an E commerce platform, and a customer places an order. Now there's a whole bunch of things that need to happen, right? You need to process the payment, you've got to check the inventory, you need to arrange shipping, send notifications to the customer, all this stuff, and instead of writing a bunch of custom code to manage all of that right step functions lets you visually design this workflow so you can see what's going on. Makes it much easier to develop and troubleshoot,

Chris 1:30
so it's like a flow chart that actually executes the steps. Yeah, you can think of it that way. Okay, I like that now. E commerce is just one example. What are some other scenarios where there are tons step functions would be really useful. Data

Kelly 1:43
processing pipelines are another great example. So you might need to ingest data from a whole bunch of different sources, transform it, validate it, load it into a data warehouse. Step functions can orchestrate all of those steps

Chris 1:59
really easily. So it's about moving data around, but it's also about applying logic Exactly. It's about making decisions and making sure things happen in the correct order. That's right. Really powerful stuff. So let's dive a little deeper into step functions. What are some of the key features and benefits?

Kelly 2:17
One of the coolest things about step functions is the visual workflow designer. You can literally drag and drop different states that represent actions or tasks and connect them together to define the flow of your workflow. Here, wait

Chris 2:31
back up. What are these states? So

Kelly 2:33
states are kind of like the building blocks of your workflow. There are different types of states, each with a specific purpose. Okay? So for example, you might have a task state that invokes a Lambda function, or you might have a choice state that lets you branch your workflow based on certain conditions. Got

Chris 2:49
it so I can create these intricate workflows, yeah, with all this different logic, yeah, different paths, yeah, and I can do it all visually. That's the idea. Okay, now I imagine that this also integrates with other AWS services. Oh, absolutely

Kelly 3:01
right. Step functions plays really nicely with a whole bunch of different services. We mentioned Lambda. You can also trigger workflows from S3 events, process messages from SQS queues, interact with DynamoDB, all kinds of

Chris 3:14
stuff. So it's like the glue that holds your serverless architecture together. Yeah, that's a good way to put it. Okay, cool. Now no system is perfect. What are some of the limitations? Sure, Every tool has its limits or gotchas that we need to be aware of with step functions. One thing to

Kelly 3:29
keep in mind is the maximum execution time for a workflow. Okay, so if you have a process that's gonna run for a really long time, you might need to consider a different approach. So you're

Chris 3:39
saying there are limits to how long a step function can run. Yeah, that's right. Okay, all right. So how does step functions fit into the broader AWS ecosystem? Step

Kelly 3:48
Functions is a core part of serverless architectures and event driven architectures. It's the orchestrator coordinating interactions between all these different services, Lambda, SQS, SNS, API gateway, all that good stuff. Okay,

Chris 4:03
so it's like the conductor of your serverless orchestra. All right, I think we've laid a good foundation here. We understand what step functions is, why it matters, how it fits into the cloud landscape. Sounds good, but now let's switch gears a little bit and talk about something that I know a lot of you are interested in, which is exam prep. Okay, so let's dive into some questions that you might encounter on the AWS certification exam.

Unknown Speaker 4:29
All right, let's do it.

Chris 4:30
So first question, if I asked you to explain the core components of a step function, state machine, what would you say? Well,

Kelly 4:38
a step machine is essentially the blueprint for your workflow, and it's made up of a few key things. First you have the states, which we've already talked about, those are the individual actions or tasks. Then you have transitions, which define the flow between those states, how you move from one state to the next. And then you have the input, which is the data that kicks off the whole execution. So. States,

Chris 5:00
transitions and input. That's right, those are the three key things to remember. Okay, now let's get a little more specific describe the different types of step function states. Okay, so

Kelly 5:09
we've touched on task states and choice states, but there are a few others as well. Okay, you have wait states which introduce delays. You have succeed states which signal a successful termination. You have fail states for explicit failures, and you have parallel states which allow you to run multiple

Chris 5:29
tasks concurrently, like cooking multiple dishes at the same time. Yeah, exactly. Okay. I like that. So the type of state you choose really depends on the logic of your workflow. That's right. Okay. Now let's say you're designing a workflow that involves invoking a Lambda function, but there's a chance that that function could fail. Okay. How would you design your step function to handle that potential error?

Kelly 5:53
Error Handling is super important for robust workflows within a task state that invokes a Lambda function. You can define retry behavior. So you can specify the number of retry attempts and even implement a back off strategy. So

Chris 6:04
it's like saying, hey, if this Lambda function fails, let's try it again a few times

Kelly 6:08
before we give it exactly you want to give it a chance to recover. Okay, but

Chris 6:11
what happens if the Lambda function still fails, even after all the retries?

Kelly 6:16
Well, that's where the catch feel comes in. You can define a different execution path to take if those retries are unsuccessful, maybe you log the error, maybe you send a notification, or you try invoking a different Lambda function as a fallback.

Chris 6:28
So it's like having a backup plan exactly case things go wrong. Okay, that's good to know. All right, now, let's talk about why someone would choose to use step functions in the first place, instead of just managing workflows manually with custom code.

Kelly 6:42
That's a great question. Step functions offers a number of advantages overwriting your own custom code. First of all, the visual workflow designer makes development much easier. It's so much simpler to see what's going on than if you have a bunch of code. Okay. Second, the built in error handling and retry mechanisms make your work clothes more resilient, so you don't have to write all that logic yourself, right? And finally, step functions takes care of all the underlying infrastructure so you can focus on the logic of your application, not worrying about servers and scaling. So it's all

Chris 7:13
about simplicity, resilience, yeah, and scalability, exactly, okay, that makes sense. Now, let's talk about how step functions integrates with other AWS services, particularly when you're building serverless applications. Step

Kelly 7:25
functions can be triggered by a whole bunch of different events. You can have API gateway requests trigger a workflow messages in an SQS queue changes to objects in an S3 bucket, all sorts of things. And once the workflow is triggered. It can invoke Lambda functions, interact with DynamoDB tables, send messages via SNS, whatever you need it to do. So

Chris 7:48
it's like the central nervous system, yeah, of your serverless application, yeah, I like that. It receives signals, coordinates, responses. Okay, so let's dive into some more specific step functions concepts, all right, that often come up on the exam you explain the difference between standard and express workflows in step functions.

Kelly 8:04
Sure, standard workflows are designed for long running processes. They offer features like error handling, retries, wait states, all that good stuff. Express workflows, on the other hand, are optimized for short lived high volume events. They're much faster and more cost effective, so

Chris 8:21
standard for endurance, express for speed. Yeah, you guys. Okay, now let's talk about a common design pattern, the fan out pattern. How can you use step functions to implement a fan out pattern?

Kelly 8:34
The fan out pattern is all about executing multiple tasks concurrently and then aggregating the results. And step functions makes this really easy. With the parallel state, you can define multiple branches of execution, each performing a different task, and once all the branches are complete, the workflow can proceed. So it's

Chris 8:50
like having a team of workers tackling different parts of a project, okay, well, I think we've covered a lot of ground here. We have good understanding of what step functions is why it matters and how to use it sounds good, but let's take a quick break. When we come back, we'll dive into some more advanced concepts and real world applications of AWS Step Functions. Stay

Kelly 9:11
tuned. All right. Welcome back. All

Chris 9:12
right. So where are we headed for round two? Let's

Kelly 9:14
talk about IAM, okay, which stands for identity and access management. IAM is super important for security in AWS, right? Security

Chris 9:21
is always a big deal, yeah. So how does this relate to step functions? Well, every

Kelly 9:25
step functions workflow runs with an IAM role, okay? And that role defines what permissions the workflow has. So for example, it controls whether the workflow can invoke Lambda functions, read from S3 buckets, write to DynamoDB, tables, all that stuff. So it's

Chris 9:42
like an ID card for your step function. Yeah,

Kelly 9:44
that's a good analogy of what it's allowed to do. What are some things to keep in mind about IAM with step functions, especially for the exam,

Chris 9:52
one common exam scenario is figuring out the minimum permissions that a workflow needs to function correctly. Yeah. Okay, so let's say your workflow needs to read data from an S3 bucket. You wouldn't want to give it full access to S3 right? You'd want to create an IAM policy that only allows it to read from that specific bucket. So

Kelly 10:12
least privilege Exactly. Now let's talk about VPCs, virtual

Chris 10:16
private clouds. How do they relate to step functions? Step

Kelly 10:19
functions workflows usually run in a shared environment, but sometimes your workflow might need to access resources that are inside your private VPC, like a database that's not publicly accessible. Oh, yeah, in those cases, you can configure your step functions workflow to run inside your VPC, and that gives it access to those private resources.

Chris 10:41
So it's like creating a secure tunnel from your step function into your private network. Yeah, you could think of it that way. Okay, so we talked about logging and monitoring before. Are there specific tools or techniques that we should be aware of?

Kelly 10:54
Step functions integrates with AWS CloudTrail, which logs all the API calls made to the step function service. This is really useful for auditing and security investigations. You can see who created or updated a workflow, who started executions, all that stuff. So it's

Chris 11:09
like having a surveillance camera, yeah, okay, but what about monitoring the actual execution of workflows? For that,

Kelly 11:15
you can use CloudWatch logs. Step functions can send detailed execution logs to CloudWatch, you can track the progress of each step, identify any errors and monitor performance metrics.

Chris 11:27
So CloudTrail for the big picture, CloudWatch logs for the details exactly. Now, cost optimization is always a big concern in the cloud it is any tips for managing costs with step functions. Step

Kelly 11:37
functions pricing is based on the number of state transitions and the execution time of your workflows. Okay, so one way to save money is to design efficient workflows, minimize unnecessary steps, reduce execution time. So it's like

Chris 11:52
streamlining the process, yeah, eliminate wasted steps, okay.

Kelly 11:55
What else? Another key factor is choosing the right workflow type. Remember standard versus Express workflows. Yes, standard workflows are more durable, they have more features, but they cost a bit more. Express workflows are faster and cheaper. So if you have a high volume, short lived use case, express workflows can save you a lot of money. Okay,

Chris 12:13
right tool for the job, and no matter how good we are designing things, things can still go wrong. Let's talk about troubleshooting. What are some common issues with step functions? One

Kelly 12:24
common problem is a workflow getting stuck in a particular state. Okay, this could be because of an error in a Lambda function, or maybe a missing resource or a dependency that's not available. The first step in troubleshooting is to look at the execution history, which gives you a step by step log of what happened. You can see the input and output of each state, and that can help you pinpoint the problem, like

Chris 12:49
retracing your steps exactly. Okay, but what if the execution history isn't enough,

Kelly 12:53
then you can dive into the CloudWatch logs, which can give you more detailed information about what happened during the execution, you might find error messages, exceptions, other clues that can help you solve the problem. So execution history

Chris 13:05
first, then CloudWatch logs, yep, okay. What if you need to make changes to a running workflow?

Kelly 13:12
You can't modify a running execution directly, but you can stop the execution and then restart it from a specific step, and when you restart it, you can incorporate your change

Chris 13:21
because you're hitting pause, yeah, making adjustments and then resuming, yeah, like a checkpoint. Okay, so we talked about the fan out pattern. What are some other design patterns?

Kelly 13:30
The sequential pattern is the most basic one. Each step happens one after the other, like a recipe. You follow the steps in order. And there's the branching pattern right, which lets you create different paths based on certain conditions, like if a payment is successful, you do one thing. If it fails, you do something else. Okay,

Chris 13:46
so sequential for simple flows, branching for decisions. What else we already

Kelly 13:51
talked about, the parallel pattern for running tasks concurrently. Then there's the scatter gather pattern, which is useful for distributing tasks to multiple workers and then collecting the results. Okay? And finally, there's the map pattern, which lets you iterate over a collection of items performing the same actions on each item. So

Chris 14:11
many patterns, a pattern for every occasion, got it so besides e commerce and data processing, what are some other real world use cases for step functions.

Kelly 14:20
Step functions is being used in all sorts of areas. One really interesting area is machine learning, building and deploying machine learning models often involves a complex series of steps. You have to pre process the data, train the model, evaluate the model, deploy the model. Step functions can orchestrate that whole pipeline, so it's like

Chris 14:41
a project manager for your machine learning. Exactly cool. What else? Serverless

Kelly 14:44
web applications are another good example. Step functions can manage user interactions like authentication, data validation, API calls. So instead of writing all that code yourself, yeah, you can use step functions to visually design those interactions. Well, it's not

Chris 14:59
just. Back end processes. It can also be used for front end user experiences. That's right. Okay, that's interesting. So we've been focused on step functions. But are there other ways to orchestrate workflows? Of

Kelly 15:10
course, step functions isn't the only tool out there. Some developers might use AWS Lambda directly to orchestrate their workflows. You can chain multiple Lambda functions together using destinations, and that essentially creates a workflow,

Chris 15:24
okay, so like a chain of dominoes, where each Domino is a Lambda function. Yeah, that's

Kelly 15:29
a good way to visualize it. But as workflows get more complex, this approach can become difficult to manage Correct. You can end up with a tangled mess of functions and it can be hard to debug. Okay?

Chris 15:41
So for simple workflows, Lambda chaining might be enough, but for more complex ones, you might want something like step functions. That's right. What other alternatives are? There?

Kelly 15:51
Another option is to use a container orchestration service like AWS ECS or EKS. You can define your workflow steps as containerized tasks, and then manage the execution using ECS or EKS. Okay.

Chris 16:07
So if you're already using containers, this might be a good option. It could be. What are the downsides of

Kelly 16:12
that? Well, container orchestration services usually require more infrastructure management, and they might be overkill for simpler workflows. Step functions being Serverless is a lot easier to manage. Okay,

Chris 16:22
so again, right tool for the job? Yeah, consider the complexity, yeah, scalability, ease of management, right? We've talked about a lot with step functions. We have what are some key takeaways for our listeners? First,

Kelly 16:32
step functions is a serverless workflow orchestration service. Makes it easier to design, build, manage complex processes in the cloud. Second, it has a visual workflow. Designer integrates with other AWS services built in error handling retries and supports a bunch of different design patterns. Third, sip functions is a key part of serverless and event driven architectures. And finally, understanding key concepts like IAM, VPCs, logging, monitoring, cost optimization, all that stuff is crucial, both for real world applications and for the exam. Those are

Chris 17:06
excellent takeaways. Now let's move on to the final part of our deep dive, where we're going to focus on some of the trickier aspects of step functions that might pop up on the exam. Okay, sounds good. Stay tuned. Okay, so we're back and ready to tackle some of the tougher stuff when it comes to step functions on the exam, sounds good. What are some of the areas where people often trip up? One area

Kelly 17:25
that can be tricky is step functions integration with other AWS services. Exam questions might throw you into a scenario where you need to use step functions with SQS or SNS or something like that. So for example, you might be asked how to trigger a step function when a new message arrives in an SQS queue, or how to use SNS to send notifications at different stages of a workflow. So

Chris 17:50
it's not enough to just know step functions. You have to know how it works with other services Exactly.

Kelly 17:55
It's all about the bigger picture. Okay, what else another thing that people sometimes struggle with is understanding the nuances of the different state types. We've talked about the common ones like task choice and parallel, but there are also some more specialized states, like weight and map that can be confusing. Okay, so make sure you understand those less common states, yeah, and how to use them in different scenarios. Any other areas that tend to cause problems, security and error handling are always popular topics on exams. Might be asked about IAM roles and policies for securing your workflows, or how to design workflows that can handle errors gracefully.

Chris 18:30
So security and resilience, those are

Kelly 18:33
key concepts All right, so you're good at

Chris 18:36
anticipating these exam questions. Can you give us a real world scenario that might show up on the exam, sure,

Kelly 18:43
let's say a company needs to process a large batch of images. They have all these images stored in an S3 bucket. They need to resize the images, convert them to a different format, and then upload them to a different S3 bucket. An exam question might ask you to design a step functions workflow to handle this. So

Chris 19:02
it's a classic image processing pipeline, yeah. How would you approach that? First,

Kelly 19:06
I would break the problem down into smaller steps, download the image, resize it, convert it, upload it. Then I would create a task state for each step, and I'd probably use Lambda functions to do the actual image processing. Now, since you can process each image independently, I would use a parallel state to process multiple images at the same time, and of course, I would add error handling and retry mechanisms So break

Chris 19:29
it down, parallelize it, make it resilient. Exactly how could they make this question even harder? They

Kelly 19:34
could bring in the concept of cost optimization. They might ask you to choose the most cost effective instance type for those Lambda functions, or they might ask you to use the execution history to find and eliminate unnecessary steps. So always

Chris 19:48
be thinking about cost. It's a big part of the cloud. Okay. Last question, what's your top tip for someone preparing for the Solutions Architect exam? When it comes to step functions,

Kelly 19:58
don't just. Memorize facts and definitions, focus on understanding the concepts and how to apply them to real problems. Okay, the best way to do that is to get hands on, build some workflows, experiment. The more you practice, the better you'll understand stuff, functions. So

Chris 20:13
understand the why, not just the what, exactly. All right, I think we've covered a lot today. We've gone from the basics to some pretty advanced stuff, and we've even talked about how to prepare for the exam. Hopefully you've learned a lot and you feel more confident about using AWS Step Functions. It's a powerful service, it is, and thanks for guiding us through it. My pleasure. Well, that's it for this deep dive, until next time. Happy cloud architecting.