Google Cloud Dataflow vs. SQL Server Integration Services

Based on my experience, streaming / real time / machine learning / AI type of processing and batch processing which needs less transformation are very well suited. Work load that needs complex transformation / multiple hops gets very complicated to implement. New feature like Dataflow SQL option will come in handy for sql heavy users.

Ideal for daily standard ETL use cases whether the data is sourced from / transferred to the native connectors (like SQL Server) or FTP. Best if the company uses MS suite of tools. There are better options in the market for chaining tasks where you want a custom flow of executions depending on the outcome of each process or if you want advanced functionality like API connections, etc.

Incentivized

• Streaming, Real time work load
• Batch processing
• Auto scaling
• flexible pricing

• SSIS works very well pulling well-defined data into SQL Server from a wide variety of data sources.
• It comes free with the SQL Server so it is hard not to consider using it providing you have a team who is trained and experienced using SSIS.
• When SSIS doesn't have exactly what you need you can use C# or VBA to extend its functionality.

Incentivized

• inbuild template options can be expanded
• more data connector options
• easy of use

• SSIS memory usage can be quite high particularly when SSI and SQL server are on the same machine
• SSIS is not available on any environment other than Microsoft Windows
• SSIS does not function with any database engine back-end other than Microsoft SQL Server

Incentivized

SSIS is responsible for running core business processed managing core business data. It can be managed, improved and expanded using minimal internal resources. It is also able to support all of our current data infrastructure. Replacing SSIS would be time consuming and costly with no apparent ROI.

Incentivized

SSIS has a drag and drop based developer interface, so it is relatively straight forward to get started. You can start to get into the weeds pretty quickly as your solution becomes more complex. However, most of the base functions are right in front of you for a developer. You can also set project and solution level parameters, so when you deploy to new environments, you don't have to jump into each package to change your variables and settings. (For example, default directory to ingest flat files).

Incentivized

Raw performance is great. At times, depending on the machine you are using for development, the IDE can have issues. Deploying projects is very easy and the tool set they give you to monitor jobs out of the box is decent. If you do very much with it you will have to write into your projects performance tracking though.

Incentivized

The support, when necessary, is excellent. But beyond that, it is very rarely necessary because the user community is so large, vibrant and knowledgable, a simple Google query or forum question can answer almost everything you want to know. You can also get prewritten script tasks with a variety of functionality that saves a lot of time.

Incentivized

The implementation may be different in each case, it is important to properly analyze all the existing infrastructure to understand the kind of work needed, the type of software used and the compatibility between these, the features that you want to exploit, to understand what is possible and which ones require integration with third-party tools

Incentivized

Google Cloud Dataproc Cloud Datafusion

I think SQL Server Integration Services is better suited for on-premises data movement and ADF is more suited for the cloud. Though ADF has more connectors, SQL Server Integration Services is more robust and has better functionality just because it has been around much longer

Incentivized

• cost saving from managing our own data center for ETL servers
• consumption based pricing
• with auto scaling feature, we were able to expand components to support work load

• Without this, we would have to manually update a spreadsheet of our SQL Server inventory
• We would also have poor alerting; if an instance was down we wouldn't know until it was reported by a user
• We only have one other person who uses SQL Server Integration Services , he's the expert. It would fall to me without him and I would not enjoy being responsible for it.

Incentivized