dbt vs. SQL Server Integration Services

dbt (Data Build Tool) is best suited for doing the data transformation. dbt is just a transformation tool and it is not suitable for building a data pipeline which requires extraction of data and loading. dbt is well suited for SQL based transformation logic and it is less appropriate when transformation logic requires python.

Incentivized

Ideal for daily standard ETL use cases whether the data is sourced from / transferred to the native connectors (like SQL Server) or FTP. Best if the company uses MS suite of tools. There are better options in the market for chaining tasks where you want a custom flow of executions depending on the outcome of each process or if you want advanced functionality like API connections, etc.

Incentivized

• user experience makes it easy to work with SQL and version control
• customer success team and the dbt (data build tool) community help establish best practices
• thorough and clear documentation

• SSIS works very well pulling well-defined data into SQL Server from a wide variety of data sources.
• It comes free with the SQL Server so it is hard not to consider using it providing you have a team who is trained and experienced using SSIS.
• When SSIS doesn't have exactly what you need you can use C# or VBA to extend its functionality.

Incentivized

• Slow load times of the dbt cloud environment (they're working on it via a new UI though)
• More out-of-the-box solutions for managing procedures, functions, etc would be nice to have, but honestly, it's pretty easy to figure out how to adapt dbt macros

Incentivized

• SSIS memory usage can be quite high particularly when SSI and SQL server are on the same machine
• SSIS is not available on any environment other than Microsoft Windows
• SSIS does not function with any database engine back-end other than Microsoft SQL Server

Incentivized

SSIS is responsible for running core business processed managing core business data. It can be managed, improved and expanded using minimal internal resources. It is also able to support all of our current data infrastructure. Replacing SSIS would be time consuming and costly with no apparent ROI.

Incentivized

dbt is very easy to use. Basically if you can write SQL, you will be able to use dbt to get what you need done. Of course more advanced users with more technical skills can do more things.

Incentivized

SSIS has a drag and drop based developer interface, so it is relatively straight forward to get started. You can start to get into the weeds pretty quickly as your solution becomes more complex. However, most of the base functions are right in front of you for a developer. You can also set project and solution level parameters, so when you deploy to new environments, you don't have to jump into each package to change your variables and settings. (For example, default directory to ingest flat files).

Incentivized

Raw performance is great. At times, depending on the machine you are using for development, the IDE can have issues. Deploying projects is very easy and the tool set they give you to monitor jobs out of the box is decent. If you do very much with it you will have to write into your projects performance tracking though.

Incentivized

The support, when necessary, is excellent. But beyond that, it is very rarely necessary because the user community is so large, vibrant and knowledgable, a simple Google query or forum question can answer almost everything you want to know. You can also get prewritten script tasks with a variety of functionality that saves a lot of time.

Incentivized

The implementation may be different in each case, it is important to properly analyze all the existing infrastructure to understand the kind of work needed, the type of software used and the compatibility between these, the features that you want to exploit, to understand what is possible and which ones require integration with third-party tools

Incentivized

Matillion is graphical versus dbt, which is SQL code-based (that, of course, is a matter of personal preference and not an objective advantage). The integrated testing, documentation generation, lineage, etc., were additional criteria that led us to choose dbt.

Incentivized

I think SQL Server Integration Services is better suited for on-premises data movement and ADF is more suited for the cloud. Though ADF has more connectors, SQL Server Integration Services is more robust and has better functionality just because it has been around much longer

Incentivized

• In 3 months we re-wrote the data warehouse (15-20 sources) in dbt with 3 developers.
• We are using it continually for the past year with no issues.
• Sorry, I don't have ROI numbers but the impact was huge.

Incentivized

• Without this, we would have to manually update a spreadsheet of our SQL Server inventory
• We would also have poor alerting; if an instance was down we wouldn't know until it was reported by a user
• We only have one other person who uses SQL Server Integration Services , he's the expert. It would fall to me without him and I would not enjoy being responsible for it.

Incentivized