AWS Database Migration Service vs. AWS Glue

As stated previously, AWS Database Migration Service excels when replicating very specific data elements between environments. AWS Database Migration Service handles replication tasks to load equipment assets or customer job sites from production to QA databases very well. Full load replication - e.g., we need an exact copy of the production database in another region - works well. But when we need to load the QA database with the latest production data - it does work as well. AWS Database Migration Service comes up short because we do not want to wipe the QA database completely and make an exact copy. We want to keep what's already in the QA database and add production data so that we can QA with that level of volume. And at least with our database design, we end up having to do a lot of manual data manipulation and de-duplication.

Incentivized

When the data which requires ETL has different formats, schema, and volume, this service suits them best. So, when the volume is not consistent (typical use-case of healthcare and online shopping), AWS Glue can be the prime choice. When the data is available in both batch and streaming mode, the developer needs to generate a separate codebase. This increases the source code management efforts. So, prefer to go with Glue when the nature of the data is the same (either batched or streamed).

Incentivized

• After data cleansing, the team also implemented the best practices for using AWS platform services as a Data Lake, such as job bookmarking for AWS Glue jobs, proper delimiter for the AWS Glue crawlers, partitioning in AWS S3, and transformation to parquet file for compression and faster querying time in Amazon Athena.
• Data modernization through combining data from multiple sources into a functioning datasets, rebuilding DW, and resctructuring data sources.
• Aims to lessen customer complaints, eliminate manual data extraction requests via SR from different data sources, and Increase accuracy, consistency and speed up reconciliation process.

Incentivized

• It’s integration with other cloud vendors is bit difficult
• If it can support non SQL based databases as well, it would be powerful.
• Real time data synchronisation in data source is missing

Incentivized

I personally found it very usable for a data engineer's day job, particularly for performing ETL and managing the data pipelines.

Incentivized

Amazon responds in good time once the ticket has been generated but needs to generate tickets frequent because very few sample codes are available, and it's not cover all the scenarios.

Incentivized

The cataloging of data objects is the best in the case of AWS Glue. We use AWS Glue in all of our data pipelines to sync external and internal data sources and to automatically produce SQL-based ETL based on AWS Glue catalog objects. Integration with Amazon products is the other advantage.

Incentivized

• Set it and forget it for maintaining data consistency - moderate cost, zero to minimal maintenance
• High touch production data loads into QA - negative ROI engineering time investment far exceeded our predictions

Incentivized

• Positive Impact :- after ETL we can able to do some kind of automation
• Negative :- At some point of time it can hamper the cost but not really

Incentivized