AWS Glue vs. Fivetran

When the data which requires ETL has different formats, schema, and volume, this service suits them best. So, when the volume is not consistent (typical use-case of healthcare and online shopping), AWS Glue can be the prime choice. When the data is available in both batch and streaming mode, the developer needs to generate a separate codebase. This increases the source code management efforts. So, prefer to go with Glue when the nature of the data is the same (either batched or streamed).

Incentivized

[Fivetran is] very well suited when you are using popular and common data sources, such as the major ad platforms, and SaaS platforms such as Salesforce. If the majority of your data sources are custom internal applications or databases, may be less value as you aren't leveraging the delivered connectors.

Incentivized

• After data cleansing, the team also implemented the best practices for using AWS platform services as a Data Lake, such as job bookmarking for AWS Glue jobs, proper delimiter for the AWS Glue crawlers, partitioning in AWS S3, and transformation to parquet file for compression and faster querying time in Amazon Athena.
• Data modernization through combining data from multiple sources into a functioning datasets, rebuilding DW, and resctructuring data sources.
• Aims to lessen customer complaints, eliminate manual data extraction requests via SR from different data sources, and Increase accuracy, consistency and speed up reconciliation process.

Incentivized

• Simplified ETL from a wide range of data sources
• Stable and painless data pipeline
• Granular control over what parts of the data source are loaded

Incentivized

• It’s integration with other cloud vendors is bit difficult
• If it can support non SQL based databases as well, it would be powerful.
• Real time data synchronisation in data source is missing

Incentivized

• Doesn't include support for a Kinesis stream as a data source so couldn't be used for some use cases under consideration
• Doesn't support the use of "BEFORE DELETE" triggers
• No support for serverless Aurora

Incentivized

I personally found it very usable for a data engineer's day job, particularly for performing ETL and managing the data pipelines.

Incentivized

Very easy and intuitive to setup and maintain as there usually are not that many options. Very well documented (e.g. how to setup each connector, how the schema looks like, any specific features of this connector etc.). Also the operation is intuitive, e.g. you have status pages, log pages, configuration pages etc. for each connector.

Incentivized

It runs pretty well and gets our data from point A to point cluster quickly enough. Honestly, it's not something I think about unless it breaks and that's pretty rare.

Incentivized

Amazon responds in good time once the ticket has been generated but needs to generate tickets frequent because very few sample codes are available, and it's not cover all the scenarios.

Incentivized

The cataloging of data objects is the best in the case of AWS Glue. We use AWS Glue in all of our data pipelines to sync external and internal data sources and to automatically produce SQL-based ETL based on AWS Glue catalog objects. Integration with Amazon products is the other advantage.

Incentivized

Fivetran came well with the connectors' availability and updates with the source changes. We had an idea on data requirements in our case which helped us to work out on cost implication and take a decision for Fivetran as a data provider for our organization. These were 2 places where Fivetran out-performed, other vendors.

Incentivized

• Positive Impact :- after ETL we can able to do some kind of automation
• Negative :- At some point of time it can hamper the cost but not really

Incentivized

• Saved a lot of manual development days (unable to quantify)
• Accelerated the time to add a new source to the data warehouse a lot

Incentivized