AWS Glue vs. AWS Lambda

When the data which requires ETL has different formats, schema, and volume, this service suits them best. So, when the volume is not consistent (typical use-case of healthcare and online shopping), AWS Glue can be the prime choice. When the data is available in both batch and streaming mode, the developer needs to generate a separate codebase. This increases the source code management efforts. So, prefer to go with Glue when the nature of the data is the same (either batched or streamed).

Incentivized

Scenarios where AWS Lambda is well suited: 1. When we need to run a periodic task few times in a day or every hour, we may deploy it on AWS Lambda so it would not increase load on our server which is handling client requests and at the same time we don't have to pay for AWS Lambda when it is not running. So, overall we only pay for few function invocations. 2. When some compute intensive processing is to be done but the number of requests per unit of time fluctuates. For example, we had deployed an AWS Lambda for processing images into different sizes and storing them on AWS S3 once user uploads them. Now, this is something that may happen few times every hour on a particular day or may not happen even once on other days. To handle this kind of tasks AWS Lambda is a better choice as we don't have to pay for the idle time of the server and also we don't have to worry about scaling when the load is high. Scenarios where AWS Lambda is not appropriate to use: 1. When we expect a large request volume continuously on the server. 2. When we don't want latency even in case of concurrent requests.

Incentivized

• After data cleansing, the team also implemented the best practices for using AWS platform services as a Data Lake, such as job bookmarking for AWS Glue jobs, proper delimiter for the AWS Glue crawlers, partitioning in AWS S3, and transformation to parquet file for compression and faster querying time in Amazon Athena.
• Data modernization through combining data from multiple sources into a functioning datasets, rebuilding DW, and resctructuring data sources.
• Aims to lessen customer complaints, eliminate manual data extraction requests via SR from different data sources, and Increase accuracy, consistency and speed up reconciliation process.

Incentivized

• AWS Lambda is a welcoming platform, supporting several languages, including Java, Go, PowerShell, Node.js, C#, Python, and Ruby. And if you need to deploy a Lambda function in another language, AWS offers a Runtime API for integration.
• We really appreciate how AWS Lambda is always-on for our functions, with only a brief "cold-start" waiting period the first time a function is called after being dormant.
• In addition to only generating costs when it's actually being used, AWS Lambda really puts the "serverless" in serverless architecture, offering turnkey scaleability and high availability for our code with zero effort on our part.

Incentivized

• It’s integration with other cloud vendors is bit difficult
• If it can support non SQL based databases as well, it would be powerful.
• Real time data synchronisation in data source is missing

Incentivized

• The UI and Developer experience is not so great. IF you use an abstraction like Serverless Application Model (SAM), things get pretty easy, but it's still AWS UI/DX you're working with after that (which is to say, not their strength).
• Documentation is always a mixed bag. Sometimes it's just easier to google your specific problem and see how others have solved it. This can be much faster than trying to find an example that may or may not be there in the documentation (which oftentimes has multiple versions and revisions).

Incentivized

I personally found it very usable for a data engineer's day job, particularly for performing ETL and managing the data pipelines.

Incentivized

It is very easy to get started with AWS Lambda and create your first function. The user interface makes it easy to add AWS services to be inputs or outputs to the function, meaning it can be configured in many different ways for different needs. This makes it ideal for various scenarios in AWS.

Incentivized

Amazon responds in good time once the ticket has been generated but needs to generate tickets frequent because very few sample codes are available, and it's not cover all the scenarios.

Incentivized

As this is a product where a great part of errors can be at the source code level, AWS support team doesn't dive that further. I mean they don't evaluate problems more complex related to your code, [which] is totally understandable, but this make[s] debug process more tough and painful.

Incentivized

The cataloging of data objects is the best in the case of AWS Glue. We use AWS Glue in all of our data pipelines to sync external and internal data sources and to automatically produce SQL-based ETL based on AWS Glue catalog objects. Integration with Amazon products is the other advantage.

Incentivized

It's fine, it works as the others would have, except EC2. We are migrating back to EC2 for dedicated compute because we have scaled to a point where we have consistent traffic. The tradeoff of maintaining infrastructure in-house outweighs the benefits of moving quickly through our roadmap.

Incentivized

• Positive Impact :- after ETL we can able to do some kind of automation
• Negative :- At some point of time it can hamper the cost but not really

Incentivized

• We have simplified log fiie ingestion using Lambda functions. The return has been less time worrying about getting logs from source to ingestion; one the process is in place the team is nearly 100% hands off.
• We have begun taking a more API focused approach by using API Gateway as the interface to business processes and Lambda as the back end compute. Moving away from server based back ends places us on a path to reducing overall spend in compute costs.
• Lambda functions allow us to easily interface with third party services through APIs. This simplifies access management since the function can be granted permissions and access to the function can be gated with API keys and other authentication methods.

Incentivized