Apache Airflow vs. Tidal by Redwood

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

1. Tidal Automation is a super robust application for Regular SQL tasks or other file maintenance which in-turn can help employee's to free up their time which they spend on working on repetitive tasks. 2.This has significantly reduced the time and effort required for setting up and managing workflows, ultimately increasing productivity. 3. Although we faced few problems while integrating the software with existing systems and was time consuming.

Incentivized

• Apache Airflow is one of the best Orchestration platforms and a go-to scheduler for teams building a data platform or pipelines.
• Apache Airflow supports multiple operators, such as the Databricks, Spark, and Python operators. All of these provide us with functionality to implement any business logic.
• Apache Airflow is highly scalable, and we can run a large number of DAGs with ease. It provided HA and replication for workers. Maintaining airflow deployments is very easy, even for smaller teams, and we also get lots of metrics for observability.

Incentivized

• Tidal Automation is good in it's integration with Oracle ERP
• Tidal Automation was robust enough to perform all the workloads smoothly on SAP Integrations.
• Tidal Automation has the robust HA and DR features
• Tidal Automation has the best and simple user interface.
• Tidal Automation documentation is the best compared to other Workload Automation solutions in the market.

Incentivized

• A local "dry run" or IDE plugin that can validate and simulate DAG execution without needing a full environment.
• Better feedback on DAG parse errors in the UI or CLI.
• Navigating large DAGs with hundreds of tasks can be slow and hard to understand visually.

Incentivized

• Still a bit slow when navigating. If you close a job you have to wait a few seconds to open another one. Even when you made no changes.
• When viewing a job and make no changes, the "ok" button changes the last modified date as if you made a change. No big deal, but wastes time when troubleshooting a problem and looking into what jobs were changed last.
• You can see the parameters column in the "job activity", but not in "Job definitions".
• Can't search the parameters field in the filter.
• Changing a variable name does not change it on the job. It still works because Tidal Automation uses the ID number. It just causes confusion when you see a variable on a job and can't find the variable under "Variables". On top of that, Tidal Automation does not show the ID column under "Variables" making it even more difficult to find the variable.

We are on the fence. The increased pricing for renewals is staggering. With new automation options like Microsoft's Power Automate and Event Driven Ansible on the field, there are other options now available.

For its capability to connect with multicloud environments. Access Control management is something that we don't get in all the schedulers and orchestrators. But although it provides so many flexibility and options to due to python , some level of knowledge of python is needed to be able to build workflows.

Incentivized

Having provided consulting services for years on Tidal by Redwood, I recommend going with a solutions partner or consultant to deploy it. I believe there are sizing and tuning guidelines that should be followed for environments of scale. I believe they are not critical when first lighting up the product, but if you are not aware of them you will encounter performance degradation after a few thousand job objects are added.

Apache Airflow is suited for a much wider set of use cases compared to Databricks. You can run it anywhere, and there is also no vendor lock-in. With Airflow, we can utilize almost any compute engine. Same thing we want to do with Databricks. There might be some level of difficulty based on the support.

Incentivized

Tidal by Redwood excels at performing complex workflows, event-driven automation, and compliance-focused procedures in big companies with varied IT infrastructures. This includes claims, policy, and billing processes are included in this. The sending of our documents and checks to our printers for automatic printing has also been automated with TA. The majority of our regular file transfers to and from our company are done utilizing TA and SFTP. Key characteristics:- Control over access and security. Resource management and error correction. Tools for reporting and observing. Capacity for extensive automation and job scheduling. Scalability for large enterprises. Orchestration of workflow for intricate operations. Uses:- A good fit for big businesses with complicated IT environments. Ideal for managing dependencies and automating complex operations. effective at organizing cross-platform and system functions. strong support for governance and compliance.

Incentivized

• Most of the ETL processes were automated, cutting down on human labor.
• Apache Airflow's user interface (UI) was very informative and straightforward.
• Since ETL processes were providing data via airflow, we were able to gain a deeper comprehension of the data at hand.

Incentivized

• It has a positive impact on factors like increase in productivity, easy to implement as there some options pre-built in it which automates and perform.
• It also reduces human error mostly as it involve less manual performance and tool is time saving in this perspective.
• It also has negative impacts like cost of the tool as its expensive and if it's not properly utilized it may lead huge revenue loss as all will be scheduled per plan.
• We have to continually monitor its effectiveness to ensure a positive return on investment.