We took two pages of a real sales report: a sales page with 14 charts and a wholesale page with 10, over about 3 million sales rows. Simulated users open the pages one after another, with at most 6 charts loading at once, as in a web browser, and a year picked at random each time.13
The number of users doubles at every step: 2, 4, 8 and on. Each step warms up for 8 seconds and is then measured for 40. A series stops when the slowest 5% of pages take more than 30 seconds.
The simulated users ran on the server itself, fenced to the same 8 of its 32 threads for both engines, so each engine had the same share of the machine. Each engine ran alone.14 OpenBI read the sales from Parquet files, with a few percent fewer rows than Analysis Services’ copy of the same model.15 Both engines keep a cache: Analysis Services keeps whole answers, OpenBI keeps building blocks.7