As one of the key community contributors to Hyku, Notch8 was grateful for a recent opportunity to perform a load test on the application, solidifying our understanding of its ability to handle multiple concurrent users. In March 2019, we conducted this testing through a project for Carleton University in Ottawa, Canada. Carleton’s intention was to set a baseline for acceptable usage, so the goal was deliberately free-form: hit the system and see what happens.

Given the complexity of a Samvera application, an average response time of 500 milliseconds is considered acceptable, with 300 milliseconds considered exceptionally good. Tests were conducted on three machine sizes: t2.medium, m5.large, and m5.xlarge. All services (Redis, Solr, Postgersql, Fedora and Rails) were run on the same instance for these baseline tests. The graphs below illustrate performance in each size machine, with 50 and 100 concurrent users respectively, reporting the time it took to complete a set script of the following actions:

  • Visit the login page
  • Logging in
  • Visiting the Dashboard Works page
  • Visiting a work in the list
  • Visiting the Dashboard Works page
  • Searching for a work
  • Visiting the Search page (not dashboard)
  • Performing a search for a Work
  • Visiting a work in the results

Visiting a page entails loading the entire page and all assets on the page (i.e. images, css, javascript, etc.)

These reports start with 1 user and ramp up to 50 or 100 concurrent users over a ten minute period.

Results t2.medium – 50 users

Results t2.medium – 100 users

Results m5.large – 50 users

Results m5.large – 100 users

Results m5.xlarge – 50 users

Results m5.xlarge – 100 users

The data per user in the graphs above represents multiple page views, so that the expectation of not exceeding 500 milliseconds per page is met when we consider that activity may be up to tenfold per user here. Load testing is fundamentally imperfect, and these conclusions are drawn on a relatively random amount of activity across those users, but still we are able to conclude that the response time for a single page view comes well within the desired expectations.

Results indicate that any of the machine sizes are sufficient to support a load of 50 concurrent users, with each test staying in the 3 second range at full test load. Only the largest machine (m5.xlarge), however, was capable of handling 100 concurrent users without an exponential increase in response times, remaining solidly in the 2 second range. Notch8 concluded before beginning this test that it is not advisable to utilize anything smaller than an AWS t2.medium (or equivalent 2 core 4GB RAM) machine, though it’s also reasonable to expect that moving Fedora and Solr to their own machines would increase response time significantly.

These findings are a win for Hyku. They confirm our confidence in Hyku’s performance, and will help the community partners make accurate system recommendations to adopters.