Benchmarks

In the following, I would like to explain my benchmarks and the results, as well as the ability to interpret them, and describe how you can test them yourself on your machine.

All code for the benchmarks is available in the benchmarks folder of the repository. The test data is generated with the generate_test_data.py script, and the benchmarks are executed with the run_tests.sh script. Read more about how to run the benchmarks at the end of this page.

Test framework

Machine:

Ryzen 7 5700X
32 GB RAM
NVME SSD

OS:

Linux

Looksyk:

Backend compiled in release profile (As in server operation (with Docker). For desktop operation, Electron's RAM consumption should also be taken into account.)

Test results

graph name	size	size comparison (1)	parse and index time (full graph (2))	RAM usage (3)	metainfo-request (4)	page load (5)	backlinks (6)
200-small-notes	16.586 words	0,02 x the bible 0,01 x complete Harry Potter series	7ms	26,38MB	0,57ms	0,37ms	0,34ms
200-mixed-notes	82.321 words	0,11 x the bible 0,07 x complete Harry Potter series	24ms	28,62MB	0,58ms	0,40ms	0,34ms
200-large-notes	275.459 words	0,35 x the bible 0,25 x complete Harry Potter series	74ms	33,88MB	0,51ms	0,47ms	0,36ms
2500-small-notes	207.192 words	0,26 x the bible 0,19 x complete Harry Potter series	93ms	37,36MB	1,68ms	0,40ms	0,32ms
2500-mixed-notes	1.033.963 words	1,30 x the bible 0,95 x complete Harry Potter series	310ms	65,41MB	2,17ms	0,31ms	0,36ms
2500-large-notes	3.449.801 words	4,41 x the bible 3,18 x complete Harry Potter series	910ms	130,37MB	2,25ms	0,46ms	0,35ms
10000-mixed-notes	4.136.753 words	5,28 x the bible 3,81 x complete Harry Potter series	1277ms	186,87MB	6,9ms	0,41ms	0,38ms

Notes:

(1) Assumptions:
- The Bible has 783,137 words
- The complete Harry Potter Series has 1,084,170 words
(2) Time to parse the graph and build the index for the graph. This is the time it takes to start the server and have the graph ready to be used. Updates to the graph are incremental and much faster.
(3) RAM usage of the backend process after parsing and indexing the graph and after processing the requests for the metainfo, page load and backlinks. This is the RAM usage of the backend process, not the entire system. When using the desktop application, Electron's RAM usage should also be taken into account.
(4) Time to process a request for the metainfo of the complete graph (used for navigation and content assist). This includes the time to retrieve the metainfo from the index and the time to process the request and send the response to the client. Parsing the response on the client side is not included in this time. Request executed with curl on /api/metainfo/.
(5) Time to process a request for the content of a page. This includes the time to retrieve the content from the index and the time to process the request and send the response to the client. Parsing the response on the client side and rendering the markdown as html is not included in this time. Request executed with curl on /api/pages/System.
(6) Time to process a request for the backlinks of a note. This includes the time to retrieve the backlinks from the index and the time to process the request and send the response to the client. Parsing the response on the client side is not included in this time. Request executed with curl onapi/metainfo/. Request executed with curl on /api/backlinks/System.

Steps to reproduce the benchmarks

Requirements:

Python 3
Cargo
sh
bc

Steps to reproduce the benchmarks:

Build the project in release mode:

cd backend
cargo build --release

Generate the testdata

cd benchmarks
python2 generate_test_data.py

Run the benchmarks

cd benchmarks
sh run_tests.sh