All benchmarks in this section were performend on hardware described in Infrastructure Requirements section.
To measure performance of TIM DB, we upload 1000 datasets. POST /datasets calls are executed in sequence, persistence of data is running in parallel. When all the data is successfully uploaded, we update all the datasets - we add 24 rows to each dataset. PUT /datasets/{id} calls are executed in sequence, persistence of data is running in parallel. When all updates are finished, we delete all the datasets in sequence.
Action Configuration Execution time Upload 1000 datasets Each dataset: 12 variables, 96384 rows, 7.5MB CSV size 1h 51m 15s Update 1000 datasets with new rows Each dataset: 12 variables, 24 rows, 2KB CSV size 1h 28m 10s Delete 1000 datasets with 2 versions Each dataset: 12 variables, 96408 rows 10m 40s
Total time: 3h 30m 06s
The measurement was performed by Python 3.11 script with use of TIM Python client v5.2.0. The TIM Python client was used in the way that corresponds to a documentation .
Following tables show different aspects of TIM DB with respect to different number of rows and columns in a dataset. In Each table, rows represent number of rows in a dataset, columns represent number of columns in a dataset.
Size of CSV file 1 5 10 50 100 150 500 1000 100 2.6 kB 5.0 kB 8.0 kB 32.0 kB 62.0 kB 92.0 kB 302.4 kB 602.9 kB 1000 25.9 kB 49.6 kB 79.1 kB 315.6 kB 611.0 kB 906.6 kB 3.0 MB 5.9 MB 10000 259.0 kB 495.0 kB 790.0 kB 3.1 MB 6.1 MB 9.1 MB 29.7 MB 59.2 MB 100000 2.6 MB 4.9 MB 7.9 MB 31.5 MB 61.0 MB 90.5 MB 297.0 MB 592.0 MB 1000000 25.9 MB 49.5 MB 79.0 MB 315.0 MB 610.0 MB 905.0 MB 3.0 GB 5.9 GB
Dataset size in DB 1 5 10 50 100 150 500 1000 100 8.2 kB 41.0 kB 41.0 kB 73.7 kB 122.9 kB 163.8 kB 434.2 kB 843.8 kB 1000 90.1 kB 122.9 kB 163.8 kB 483.3 kB 942.1 kB 1.4 MB 4.1 MB 8.2 MB 10000 630.8 kB 958.5 kB 1.4 MB 4.6 MB 9.1 MB 13.7 MB 41.0 MB 82.0 MB 100000 6.1 MB 9.3 MB 13.5 MB 45.6 MB 91.1 MB 136.6 MB N/A N/A 1000000 60.3 MB 93.1 MB 134.4 MB N/A N/A N/A N/A N/A
Duration of POST /datasets API call 1 5 10 50 100 150 500 1000 100 1s 1s 1s 1s 1s 1s 1s 1s 1000 1s 1s 1s 1s 1s 1s 4s 5s 10000 1s 1s 1s 2s 3s 8s 22s 39s 100000 2s 3s 4s 24s 30s 58s N/A N/A 1000000 13s 26s 40s N/A N/A N/A N/A N/A
Duration of dataset upload 1 5 10 50 100 150 500 1000 100 4s 4s 4s 4s 6s 9s 25s 1m 14s 1000 4s 5s 5s 5s 8s 10s 33s 1m 11s 10000 5s 5s 5s 14s 23s 41s 3m 59s 6m 10s 100000 9s 14s 18s 1m 58s 3m 8s 5m 24s N/A N/A 1000000 1m 7s 2m 24s 3m 35s N/A N/A N/A N/A N/A
Duration of dataset delete 1 5 10 50 100 150 500 1000 100 14s 14s 14s 14s 16s 19s 36s 1m 24s 1000 15s 15s 15s 15s 17m 20m 41m 1m 17s 10000 15s 15s 15s 22s 31s 45s 3m 49s 5m 45s 100000 18s 21s 24s 1m 45s 2m 49s 4m 36s N/A N/A 1000000 1m 5s 2m 8s 3m 5s N/A N/A N/A N/A N/A
We measure the performance of TIM Forecasting capabilities by running 3 scenarios (described below) with different dataset sizes.
scenario: Register and execute forecast build-model jobs with the following registration payload: { "useCase" : { "id" : "<use-case-id>" } } scenario: Register and execute forecast predict jobs with the model built in scenario 1 and with following registration payload: { "data" : { "outOfSampleRows" : [ { "from" : "0001-01-01 00:00:00" , "to" : "3000-01-01 00:00:00" } ] } } scenario: Register and execute forecast predict jobs with the model built in scenario 1 and with empty JSON in the registration payload: Scenario 1 Max CPU usage in % 1 5 10 50 100 150 500 1000 100 0.06 0.06 0.19 0.33 0.26 0.19 N/A N/A 1000 N/A 0.26 0.47 0.37 0.15 0.14 N/A N/A 10000 N/A 0.21 0.82 0.19 1.08 0.4 N/A N/A 100000 0.72 0.79 1.45 0.99 1.31 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Max RAM usage in MiB 1 5 10 50 100 150 500 1000 100 3250.0 3271.0 3263.0 3269.0 3274.0 3304.0 N/A N/A 1000 N/A 3325.0 3336.0 3382.0 3400.0 3424.0 N/A N/A 10000 N/A 3506.0 3716.0 4523.0 4623.0 4600.0 N/A N/A 100000 2511.0 4961.0 5700.0 3562.0 4558.0 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Model building and prediction time in seconds 1 5 10 50 100 150 500 1000 100 1.13 1.22 0.9 1.14 1.22 1.28 N/A N/A 1000 N/A 1.09 1.11 1.55 1.93 2.35 N/A N/A 10000 N/A 3.23 4.8 22.07 26.98 30.57 N/A N/A 100000 15.62 28.52 38.19 52.64 85.1 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Total execution time in seconds 1 5 10 50 100 150 500 1000 100 7.53 7.49 9.68 21.31 41.99 50.48 312.4 311.64 1000 N/A 8.26 9.53 24.24 36.69 49.87 312.33 313.21 10000 N/A 9.9 14.13 49.02 64.64 83.31 309.52 312.62 100000 28.93 43.77 53.0 90.17 151.38 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Forecasting result table size in DB 1 5 10 50 100 150 500 1000 100 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB N/A N/A 1000 N/A 147.5 kB 147.5 kB 147.5 kB 147.5 kB 147.5 kB N/A N/A 10000 N/A 1.4 MB 1.4 MB 1.4 MB 1.4 MB 1.4 MB N/A N/A 100000 14.1 MB 14.1 MB 14.1 MB 14.1 MB 14.1 MB N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Scenario 2 Max CPU usage in % 1 5 10 50 100 150 500 1000 100 0.12 0.05 0.68 0.33 0.11 0.2 N/A N/A 1000 N/A 0.7 0.48 0.29 0.52 0.21 N/A N/A 10000 N/A 1.1 0.03 0.25 0.29 0.5 N/A N/A 100000 0.21 0.42 0.35 0.66 1.1 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Max RAM usage in MiB 1 5 10 50 100 150 500 1000 100 3270.0 3265.0 3260.0 3279.0 3277.0 3293.0 N/A N/A 1000 N/A 3317.0 3344.0 3378.0 3412.0 3407.0 N/A N/A 10000 N/A 2003.0 1869.0 2005.0 1962.0 2219.0 N/A N/A 100000 2161.0 2159.0 2255.0 3136.0 4198.0 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Model building and prediction time in seconds 1 5 10 50 100 150 500 1000 100 1.77 0.18 0.17 0.3 0.29 0.29 N/A N/A 1000 N/A 0.18 0.2 0.36 0.46 0.53 N/A N/A 10000 N/A 0.86 0.95 5.21 6.28 11.42 N/A N/A 100000 11.43 14.07 18.13 45.66 80.33 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Total execution time in seconds 1 5 10 50 100 150 500 1000 100 10.17 7.49 7.47 19.12 38.62 47.79 N/A N/A 1000 N/A 7.2 9.53 23.01 35.53 49.84 N/A N/A 10000 N/A 9.62 9.64 33.5 44.86 74.72 N/A N/A 100000 23.17 28.41 34.96 80.79 150.37 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Forecasting result table size in DB 1 5 10 50 100 150 500 1000 100 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB N/A N/A 1000 N/A 147.5 kB 147.5 kB 147.5 kB 147.5 kB 147.5 kB N/A N/A 10000 N/A 1.4 MB 1.4 MB 1.4 MB 1.4 MB 1.4 MB N/A N/A 100000 14.1 MB 14.1 MB 14.1 MB 14.1 MB 14.1 MB N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Scenario 3 Max CPU usage in % 1 5 10 50 100 150 500 1000 100 0.11 0.04 0.63 0.41 0.11 0.49 N/A N/A 1000 N/A 0.64 0.4 0.17 0.45 1.12 N/A N/A 10000 N/A 0.52 0.74 0.2 0.44 0.03 N/A N/A 100000 0.02 0.18 0.09 0.12 0.12 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Max RAM usage in MiB 1 5 10 50 100 150 500 1000 100 3241.0 3276.0 3255.0 3278.0 3281.0 3285.0 N/A N/A 1000 N/A 3315.0 3332.0 3354.0 3349.0 3379.0 N/A N/A 10000 N/A 3387.0 3369.0 3381.0 3444.0 3598.0 N/A N/A 100000 3570.0 3610.0 3601.0 3601.0 3594.0 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Model building and prediction time in seconds 1 5 10 50 100 150 500 1000 100 0.15 0.73 0.14 0.14 0.15 0.16 N/A N/A 1000 N/A 0.14 0.14 0.14 0.15 0.16 N/A N/A 10000 N/A 0.15 0.19 0.15 0.15 0.23 N/A N/A 100000 0.14 0.18 0.16 0.15 0.2 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Total execution time in seconds 1 5 10 50 100 150 500 1000 100 5.08 7.18 7.4 14.51 28.25 36.28 N/A N/A 1000 N/A 5.03 7.3 19.2 26.23 38.16 N/A N/A 10000 N/A 4.97 7.3 21.61 27.89 41.58 N/A N/A 100000 5.17 7.21 7.52 16.75 31.38 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Forecasting result table size in DB 1 5 10 50 100 150 500 1000 100 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A 1000 N/A 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A 10000 N/A 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A 100000 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
We measure the performance of TIM Detection capabilities by running 3 scenarios (described below) with different dataset sizes.
scenario: Register and execute detection build-model jobs with the following registration payload: { "useCase" : { "id" : "<use-case-id>" } } scenario: Register and execute detection detect jobs with the model built in scenario 1 and with empty JSON in the registration payload: scenario: Register and execute detection detect jobs with the model built in scenario 1 and with the following registration payload: { "data" : { "rows" : { "baseUnit" : "Sample" , "value" : 1 } } } Scenario 1 Max CPU usage in % 1 5 10 50 100 150 500 1000 100 0.78 0.21 0.07 0.26 0.02 1.83 N/A N/A 1000 0.24 0.1 0.75 0.32 0.11 0.33 N/A N/A 10000 1.35 1.14 1.42 1.6 0.89 0.55 N/A N/A 100000 3.33 4.15 5.21 7.63 3.39 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Max RAM usage in MiB 1 5 10 50 100 150 500 1000 100 3601.0 3620.0 3610.0 3619.0 3640.0 3659.0 N/A N/A 1000 3667.0 3688.0 3703.0 3733.0 3744.0 3754.0 N/A N/A 10000 3847.0 3879.0 4012.0 4938.0 5081.0 4837.0 N/A N/A 100000 2401.0 4530.0 5741.0 3372.0 3058.0 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Model building and detection time in seconds 1 5 10 50 100 150 500 1000 100 1.3 2.68 1.4 1.38 1.54 1.56 N/A N/A 1000 2.32 2.18 2.22 2.41 2.68 2.46 N/A N/A 10000 13.5 16.84 19.9 32.23 33.3 35.31 N/A N/A 100000 79.88 127.28 102.11 123.86 104.35 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Total execution time in seconds 1 5 10 50 100 150 500 1000 100 5.05 7.2 7.97 17.13 26.32 38.32 311.15 311.3 1000 9.7 7.59 9.7 16.82 31.46 38.95 312.07 311.62 10000 20.05 24.74 26.67 47.82 61.35 75.59 320.2 334.63 100000 98.02 145.21 117.01 148.01 153.97 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Detection result table size in DB 1 5 10 50 100 150 500 1000 100 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB N/A N/A 1000 114.7 kB 114.7 kB 114.7 kB 114.7 kB 114.7 kB 114.7 kB N/A N/A 10000 1.1 MB 1.1 MB 1.1 MB 1.1 MB 1.1 MB 1.1 MB N/A N/A 100000 10.9 MB 10.9 MB 10.9 MB 10.9 MB 10.9 MB N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Scenario 2 Max CPU usage in % 1 5 10 50 100 150 500 1000 100 0.71 0.18 0.13 0.18 0.16 1.95 N/A N/A 1000 0.19 0.08 0.63 0.25 0.14 0.34 N/A N/A 10000 1.31 1.03 1.18 1.25 0.62 1.13 N/A N/A 100000 0.9 0.24 0.47 0.6 0.35 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Max RAM usage in MiB 1 5 10 50 100 150 500 1000 100 3605.0 3615.0 3642.0 3611.0 3638.0 3656.0 N/A N/A 1000 3675.0 3683.0 3757.0 3729.0 3735.0 3753.0 N/A N/A 10000 3844.0 3816.0 3857.0 3872.0 3901.0 3910.0 N/A N/A 100000 3893.0 3955.0 3986.0 4588.0 5119.0 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Model building and detection time in seconds 1 5 10 50 100 150 500 1000 100 0.74 0.86 0.83 0.76 0.79 0.76 N/A N/A 1000 0.99 0.88 0.92 0.82 0.81 0.86 N/A N/A 10000 1.13 1.06 1.28 2.27 1.94 1.87 N/A N/A 100000 4.69 4.58 5.53 11.31 13.88 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Total execution time in seconds 1 5 10 50 100 150 500 1000 100 5.1 7.12 10.09 16.67 28.83 38.41 N/A N/A 1000 9.17 8.01 7.22 16.78 35.72 38.23 N/A N/A 10000 7.56 7.39 7.39 19.22 29.59 41.52 N/A N/A 100000 14.22 14.3 17.15 36.63 51.86 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Detection result table size in DB 1 5 10 50 100 150 500 1000 100 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB 16.4 kB N/A N/A 1000 114.7 kB 114.7 kB 114.7 kB 114.7 kB 114.7 kB 114.7 kB N/A N/A 10000 1.1 MB 1.1 MB 1.1 MB 1.1 MB 1.1 MB 1.1 MB N/A N/A 100000 10.9 MB 10.9 MB 10.9 MB 10.9 MB 10.9 MB N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Scenario 3 Max CPU usage in % 1 5 10 50 100 150 500 1000 100 0.66 0.17 0.11 0.22 0.1 2.09 N/A N/A 1000 0.17 0.16 0.53 0.18 0.08 0.22 N/A N/A 10000 1.1 0.87 0.99 1.52 0.37 0.93 N/A N/A 100000 0.83 0.22 0.39 0.49 0.28 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Max RAM usage in MiB 1 5 10 50 100 150 500 1000 100 3606.0 3612.0 3623.0 3622.0 3632.0 3647.0 N/A N/A 1000 3657.0 3715.0 3698.0 3723.0 3701.0 3764.0 N/A N/A 10000 3797.0 3798.0 3817.0 3838.0 3855.0 3819.0 N/A N/A 100000 3835.0 3850.0 3825.0 3857.0 4045.0 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Model building and detection time in seconds 1 5 10 50 100 150 500 1000 100 0.66 0.27 0.3 0.29 0.29 0.27 N/A N/A 1000 0.38 0.3 0.28 0.27 0.33 0.28 N/A N/A 10000 0.33 0.28 0.27 0.26 0.3 0.26 N/A N/A 100000 0.29 0.25 0.25 0.29 0.27 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Total execution time in seconds 1 5 10 50 100 150 500 1000 100 4.85 4.93 7.38 14.4 29.06 38.84 N/A N/A 1000 4.96 7.26 7.57 16.76 34.68 36.21 N/A N/A 10000 7.84 5.03 7.47 17.15 26.7 37.73 N/A N/A 100000 7.77 7.23 7.2 16.88 27.07 N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Detection result table size in DB 1 5 10 50 100 150 500 1000 100 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A 1000 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A 10000 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A 100000 8.2 kB 8.2 kB 8.2 kB 8.2 kB 8.2 kB N/A N/A N/A 1000000 N/A N/A N/A N/A N/A N/A N/A N/A
Multiple datasets are uploaded to TIM from the perspective of a certain date in the past. To simulate a production scenario, the following data points are patched to the datasets, creating multiple versions. Multiple jobs are then created, combining each dataset version with the same configuration. These jobs are then executed in parallel to give the user production grade results.
Datasets Dataset Versions Configurations Models Execution time 16 (11 variables, 953 rows each) 32 (11 variables, 43 rows each) for each dataset 2 1024 in parallel 2h 17m 57s