Apache JMeter vs. Tricentis NeoLoad

The key questions I would pose are: how high of a concurrent load do you need to generate against the system? How technically skilled are your personnel that will perform the load testing? How much programmable/scripting/code based control do you desire from the tool? Do you prefer more of a GUI based tool or a programmatic code/scripting based tool? JMeter will be for those that want a GUI, moderate to low scripting/code based control/programmability, moderate to low load generation, and for users experienced in the novice to expert range.

While Tricentis NeoLoad supports SAP GUI protocols, recording and maintaining test scripts for complex SAP transactions (like those in the OTC module) can be cumbersome.


For instance, we had challenges recording a VA01 (Sales Order Creation) transaction because of the dynamic nature of SAP screen fields and session handling. NeoLoad sometimes struggles to properly correlate or handle SAP’s SAPGUI Dynpro fields automatically.

Incentivized

• Easy to construct Webservices script - You just need to add a sampler with the endpoint, payload and header information. And boom you can run a load test on the service with varying load and capture the response time and throughput. In a commercial tool like Loadrunner or Silkperformer, this process would take at least couple of hours, while with Jmeter we can do this in 10 minutes or less.
• Modularizing the scripts - Jmeter has a very good way of organizing the scripts and the reusability of the existing script (transactions) is also as easy as copy pasting the script.
• Jmeter tests can be run from anywhere, so you can run it from the idle webserver itself if you have some firewall restrictions to test it from your local or other environment.
• Jmeter distributed tests can be spawn from the AWS and results collated to see the performance from different regions in the world.

• NeoLoad supports performance test engineers with a strong feature set to analyse the results. The first metric you have in mind is response time of the application. The metric satisfies the stakeholder if it meets the requirements. Otherwise, the stakeholder wants to know some comments and suggestions for improvement. I rely on the data gathered by monitoring agents when I analyse test results. The monitoring agent is a service which collects different counters from an operating system, databases, web server, etc. The results can be analysed in individual composed graphs. This is a powerful feature to deliver conclusive reports to stakeholders. Even the export of reports in various formats is helpful for communication with different stakeholders.
• Our customers like the flexible license model of NeoLoad. A usual performance test project starts with the development of tests and configuration of the test environment. Finally, there is a first test execution followed by a couple of additional test executions. Usually, it is hard to say how long it takes to implement the issues based on the test results. So we usually do not know how long is the period between first and last test execution. It does not matter how long the period might be. In most of our projects we prefer daily licenses of NeoLoad. If you buy e.g. a four-day license you can use NeoLoad at four different days.
• As a performance test engineer, I like the impressive APIs of NeoLoad. NeoLoad is just a performance test tool and the vendor Neotys is focused on this job. Whenever you would like to add something into a test scenario which is not mainly a performance issue you can integrate these parts by APIs which are easy to handle and very powerful. They also support you in case technologies which are not supported yet by NeoLoad. Maybe there is a new device which you want to monitor. Therefore you can report the measurements of the specific counters by the data exchange API during test execution. So you can analyse the data in the usual way. In our company we developed already a couple of features based on the APIs of NeoLoad and we are very happy with this flexibility.

Incentivized

• Even though it has very easy installation, you can run into issues easily. I was ab;e to trigger a test at browser level but couldn't stop the tests. I had to kill it from task manager and then had to reinstall from begining.
• There are so many things in JMeter you can do especially with the sampler for selenium for UI tests. More documentation/webinars etc.. would help the community better.
• The jmx files should have an easy way to be integrated with ci/cd tools and cloud.

Incentivized

• There is no support for VT3270 terminal emulation to support mainframe-hosted applications.
• The ramp-up and ramp-down time data are not automatically excluded from the average response time calculations. Instead, you must exclude these times manually each time you run a test.
• There is not a summary graphic that shows the distribution of all of the response times on a single graph. We currently export the data as a coma separated variable (CSV) file and import into Excel to generate this summary graph.

Incentivized

Price, Wiki and user sharing. Having access to the information provided by the developers and other open source providers is key for me. The ability to share information and get answers directly is very important to success in software testing. And the price of this product currently is amazing. Too many companies charge way too much money for products that are far behind in their value and pertinence

We find NeoLoad easy to use compared to their competitors. It requires minimal coding knowledge, also it is easy to train newcomers with NeoLoad. It also offers good integration and automation capability where with less code you can automate your testing process

Incentivized

I can jump right into a new test plan and start building from scratch. The natural progression from test plan to thread group and then designing the basic format of the process is very streamlined and smooth. With only slight modifications I can build out a very complex model from a very basic beginning.

In being flexible to meet all needs, it is very bare bones by design, but this could be a considered a feature and not a bug.

I have been using JMeter for the last year. By using this tool, you can make sure the system will work under varied loads. It helps us to simulate real time scenarios by creating required virtual users and make sure the application will work under load. Perform load, stress, and stability testing using JMeter.

Incentivized

This is the only area where we strongly feel Tricentis needs to improve. Their support process is inadequate, representatives are not so capable of providing solutions at one shot without consulting their internal engineering teams. Most importantly, the time they take to resolve simple issues is huge and at many times hampered our work.

Incentivized

1. Their documentation is scattered over many places like GitHub read me files, their internal website etc;
2. Support team takes huge time just to acknowledge our ticket
3. No support to automatically migrate scripts from competetor tool into

Incentivized

I have evaluated LOADUI (web free version) and it was a very unstable tool and I could not rely on those results completely as I was not sure how the tool was performing. It only generated the top 10 less transaction times and when Jmeter was in use, it was very effective in using plugins to accommodate more outputs.

Incentivized

Tricentis NeoLoad is easier to use, with simpler steps to follow to record/write test scripts and to design scenarios. It is scalable and comparatively provides overall benefits in a fair cost. Neoload also provides APIs for anyone to leverage and build on top of the existing features. This adds up as an advantage.

Incentivized

• It helped to estimate resources required for a particular client requirement. For example if the client wants the application to be used by 500 users for 1 hour, based on the load test we perform for 500 users and 1 hour we recommend resources and their capabilities.
• Jmeter was used to identify bottleneck. This would lead to better performance.
• Jmeter was used to identify failures as well.

• Improved user experience: By identifying and fixing performance issues during testing, Tricentis NeoLoad has helped improve the overall user experience of our applications. This has increased customer satisfaction and loyalty, as well as increased revenue as users are now more likely to use the application or purchase products or services as a result of a better user experience.
• Reduced risk of downtime: By identifying and fixing performance issues before they occur in production, Tricentis NeoLoad has helped reduce the risk of downtime. Downtime can be costly in terms of lost revenue and customer dissatisfaction, so reducing the risk of downtime has had a positive impact on overall business objectives.
• Cost savings: We have been able to identify and fix performance issues before they occur in production, avoiding the costs associated with downtime, such as lost revenue and customer dissatisfaction.
• Faster performance testing: Tricentis NeoLoad has enabled our teams to perform performance testing more quickly by automating the process of simulating user loads and measuring response times, throughput, and error rates.

Incentivized