At VI Company, we store a lot of time-series data. This type of data accumulates considerably every second. An example includes the quotes of financial instruments we store for banks, each of which represents a change in said instrument’s price on the stock exchange.
There can be more than a billion of these every day, and it's paramount that the order in which they occur is accurate to the millisecond. This requires a database which is not only adequate but also remains fast, even when requesting information about multiple years. To meet our clients' needs in this regard, we embarked on a quote experiment to select a better, faster solution. Allow us to elaborate!
The experiment: from long list to short list
Most of our projects involve lots of data, and the ocean of information only expands over time. In terms of databases, this poses a challenge. When calculating a product's price over the course of five years, for example, the database needs to consider and aggregate all the requested quotes based on the query we send – a process that costs quite some time as it involves millions of data points. Therefore, we wanted to take it up a notch in terms of 'read performance.' On top of that, we felt that the 'write performance' (the solution's ability to write and store new data points) could be improved, too.
Therefore, we decided to start an experiment to see if we can go about it in a faster, better way. First, we listed the requirements our new solution should meet. Then, we made a comprehensive overview of potential solutions, which we subsequently narrowed down to three favorites. In the final ‘round,’ we created realistic scenarios and put all three of them to the test. Our main question in this stage: How fast is this solution, and is it easy to implement and use?
The 'winner:' Timescale
Ultimately, we opted for a solution called 'Timescale.' It checks all the boxes – from user-friendliness to price/quality ratio – and it doesn't require us to learn an entirely new programming language. Ninety percent of it is familiar because we've worked in a similar environment.
When testing on our development machines, Timescale could congest around 100k 'quotes' in under 1 second, while competing products took around 1.5 seconds. The 'read performance' was where the Timescale really stood out, by being able to convert 1 million quotes over a timespan of a year to 1 per day in under 1 second, while others took more than 3 seconds.
Currently, we're using Timescale for an internal project to test whether it works as expected. If so, we'll start using it for our clients as well. We believe our use of this solution will be very beneficial for them. We can quickly retrieve data over a timespan of years should clients wish us to do so. On top of that, Timescale can store a wealth of data and allows us to scale up when projects expand – in terms of project size, the sky is the limit with Timescale!