Apache JMeter vs. Azure DevOps Services

The key questions I would pose are: how high of a concurrent load do you need to generate against the system? How technically skilled are your personnel that will perform the load testing? How much programmable/scripting/code based control do you desire from the tool? Do you prefer more of a GUI based tool or a programmatic code/scripting based tool? JMeter will be for those that want a GUI, moderate to low scripting/code based control/programmability, moderate to low load generation, and for users experienced in the novice to expert range.

ADO is well suited for the visibility of day-to-day tasks and responsibilities as well as things such as Features, user stories, etc. Off the top of my head, I can't think of any scenario where it might not be well suited, as you can customize ADO to your liking to a degree.

• Easy to construct Webservices script - You just need to add a sampler with the endpoint, payload and header information. And boom you can run a load test on the service with varying load and capture the response time and throughput. In a commercial tool like Loadrunner or Silkperformer, this process would take at least couple of hours, while with Jmeter we can do this in 10 minutes or less.
• Modularizing the scripts - Jmeter has a very good way of organizing the scripts and the reusability of the existing script (transactions) is also as easy as copy pasting the script.
• Jmeter tests can be run from anywhere, so you can run it from the idle webserver itself if you have some firewall restrictions to test it from your local or other environment.
• Jmeter distributed tests can be spawn from the AWS and results collated to see the performance from different regions in the world.

• Flexible Requirements Hierarchy Management: AZDO makes it easy to track items such as features or epics as a flat list, or as a hierarchy in which you can track the parent-child relationship.
• Fast Data Entry: AZDO was designed to facilitate quick data entry to capture work items quickly, while still enabling detailed capture of acceptance criteria and item properties.
• Excel Integration: AZDO stands out for its integration with MS Excel, which enables quick updates for bulk items.

Incentivized

• Even though it has very easy installation, you can run into issues easily. I was ab;e to trigger a test at browser level but couldn't stop the tests. I had to kill it from task manager and then had to reinstall from begining.
• There are so many things in JMeter you can do especially with the sampler for selenium for UI tests. More documentation/webinars etc.. would help the community better.
• The jmx files should have an easy way to be integrated with ci/cd tools and cloud.

Incentivized

• Need to make the changes so that it doesn't occupy most of the CPU utilization and memory
• Execution of Bulky SQl Queries leads to either the SQl being out of exception or the VS being unresponsive
• Integration with Microsoft products is easy, but with non-Microsoft products it is more difficult, and you have to make a lot of configuration changes to integrate
• With every upgrade of the Visual Studio, like from VS 2010 to VS 2013 , we need to upgrade our hardware/machine, as the VS hardware requirement also increases
• If code is getting compiled in one visual studio, like in VS 2010, that the same code could possibly give an error when compiled in VS 2013, due to certain changes in keyword, data format, etc., with the VS upgrade

Incentivized

Price, Wiki and user sharing. Having access to the information provided by the developers and other open source providers is key for me. The ability to share information and get answers directly is very important to success in software testing. And the price of this product currently is amazing. Too many companies charge way too much money for products that are far behind in their value and pertinence

I don't think our organization will stray from using VSTS/TFS as we are now looking to upgrade to the 2012 version. Since our business is software development and we want to meet the requirements of CMMI to deliver consistent and high quality software, this SDLC management tool is here to stay. In addition, our company uses a lot of Microsoft products, such as Office 365, Asp.net, etc, and since VSTS/TFS has proved itself invaluable to our own processes and is within the Microsoft family of products, we will continue to use VSTS/TFS for a long, long time.

Incentivized

I can jump right into a new test plan and start building from scratch. The natural progression from test plan to thread group and then designing the basic format of the process is very streamlined and smooth. With only slight modifications I can build out a very complex model from a very basic beginning.

Azure DevOps is a powerful, complex cloud application. As such there are a number of things it does great and something where there is room for improvement. One of those areas would be in usability. In my opinion it relies too much on search. There is no easy way to view all projects or to group them in a logical way. You need to search for everything.

Incentivized

I have been using JMeter for the last year. By using this tool, you can make sure the system will work under varied loads. It helps us to simulate real time scenarios by creating required virtual users and make sure the application will work under load. Perform load, stress, and stability testing using JMeter.

Incentivized

When we've had issues, both Microsoft support and the user community have been very responsive. DevOps has an active developer community and frankly, you can find most of your questions already asked and answered there. Microsoft also does a better job than most software vendors I've worked with creating detailed and frequently updated documentation.

Incentivized

Was not part of the process.

Incentivized

I have evaluated LOADUI (web free version) and it was a very unstable tool and I could not rely on those results completely as I was not sure how the tool was performing. It only generated the top 10 less transaction times and when Jmeter was in use, it was very effective in using plugins to accommodate more outputs.

Incentivized

Jira is fantastic for project management and customer facing portal. It is not good for pure development (no integration with Git, pipeline management, automated testing features). If DevOps were to integrate and adopt the project features of Jira as well as the customer facing interfaces, I feel it would be a complete project management system.

Incentivized

• It helped to estimate resources required for a particular client requirement. For example if the client wants the application to be used by 500 users for 1 hour, based on the load test we perform for 500 users and 1 hour we recommend resources and their capabilities.
• Jmeter was used to identify bottleneck. This would lead to better performance.
• Jmeter was used to identify failures as well.

• Increased dev team efficiency through more streamlined development processes and task automation.
• Improved quality of software deployments due to better source control, automated testing, and release management options available in DevOps.
• Better collaboration between the dev team, business analysts, and agile project managers.

Incentivized