How does DataTableDev work: what’s on the other side of the development

DataTableDev is more than just another grid component on the market you can choose from. For our team, it’s a big step further in the whole sphere of data visualization and analytics on the web. This piece brings in all the reasons and our opinions on the product’s creation and the technology behind it.

What is DataTableDev: a simple grid or a new powerful technology?

The idea implemented in DataTableDev is much deeper than just a good web grid component. In fact, the product is a prototype and testing ground for a larger concept we’ve been developing for quite some time. 

Behind the simple grid stands an entirely new and robust algorithm for working with data, which destroys the stereotype about the time-consuming and complex nature of working with extensive data and dealing with high-quality, large-scale analysis.

DataTableDev presents only a specific part of the possibilities we are trying to implement. However, it already shows all the main points:

• works with actually colossal data volumes (like millions of rows and gigabytes of data), 
• visualizes and updates the display quickly without delays, 
• and gives an immediate response to any user action like sorting, filtering, searching, and scrolling.

We aim to focus the analyst’s attention and capabilities on an essential thing – data and insights. The grid should be only a tool for the visualization and operations of information. Therefore, its process should be as inconspicuous as possible for the user, and the product should be considered a time-saver in data analysis.

Why do we want the grids to work faster?

Our team also faces the need for data grids throughout our work time, and we use them often. So we are more than aware of the trouble of unproductive, irritating, and lengthy data analysis processes because of the extra big amount of data. We noticed the following trends in others, too:

✊ Users always want the software to be faster and more powerful, so it would be pleasant to use.

☝️ In the modern world, more and more people need to operate with big data consisting of millions of rows but still need the classic analysis and correlation search. 

✌️ Vast amounts of data often cause overtime and complicate the work of the aforementioned software.

So stable, smooth, and trouble-free performance in data tables is a must-have for us, but we couldn’t find any component that would fit our understanding of “fast” and deal with such extensive datasets.

The solution was obvious – create it ourselves! 

But before that, we needed to conduct ample technical research to understand what we would create.

We tested and examined more than 60+ different options of similar components – nothing matched our vision. So we turned to statistics and user behavior to find the root of the problem. It turns out that software’s responsiveness heavily influences how users work and feel about it. 

Ideally, the user should not notice any intermediary between the actions and the result, which means that all user’s actions should have an immediate visible impact.

As stated in multiple studies on how the human brain perceives information when it comes to web-based applications, there are three primary time limits within which the app should respond to user input:

• 0.1 second – people regard it as an immediate action
• > 1 second – users can spot an intermediary
• > 10 seconds – people start losing attention

0.1 seconds became our reference point 🚩 – during this time, the grid must handle the event, connect to the data source, request the data needed, receive it back, and display it on the screen. It sounds like much more than 0.1 seconds, doesn’t it?

But no one was going to retreat, so we started developing a special approach to working with tabular data providing our users and us with ideal performance regardless of the data size.

How are we developing the approach to the fastest grid?

Shortly to say, we used the following steps to build the perfect algorithm for such tasks:

1. Сlearly defined a particular understanding and anatomy of frames and developed all subsequent processes based on this understanding.
2. Optimized all processes and cooperation between them to achieve maximum productivity and minimum time used.
3. Implemented special callbacks to improve further rendering speed and proper resource utilization.

Now let’s have a more detailed overview of each step.

Frames and their anatomy

When the screen is updated and repainted by the browser, the processes are encapsulated in a frame. The rate at which each frame is generated is used to measure the interface responsiveness. 

A high and steady frame rate is required to provide smooth interactions. For a user interface to be perceived as responsive, the frame rate target is 60 frames per second, and all of them should be equal in length, which helps steer clear of jaggy and rigid animations.

Calculations prove that each frame needs to take up about 16.7 milliseconds: 

1000 (milliseconds) / 60 (frames per second) ≈ 16.7 (milliseconds).

You can fit six frames of this length in 100 milliseconds (0.1 seconds as it is our 🚩). Therefore, our task was to put all the processes in these six frames and order and prioritize them effectively. The diagram above illustrates the correspondence of processes that allow us to reach the target. 

Processes optimization 

Each frame gets 16.7 milliseconds and has a specific structure: data calculations, styles recalculation, the layout is rewritten, and the changes are drawn on the browser screen.

Some big, lengthy, and time-consuming operations can overload the frame and take too much time, canceling other essential tasks like redrawing the component’s state. One of the most common subsequences is that a particular frame is not drawn in time and is getting skipped by a browser. Because what is the matter of spending resources on something that isn’t relevant anymore?

When the frames are delayed or lost – it will look like a glitch on the screen. The delay of one process leads to disruption and delay in the structure of all processes. As a result, the picture on the screen isn’t smooth and integral, which can distract and irritate. So basically, maintaining a stable frame rate strongly affects the UX of your product.

Therefore, it is critical to adjust each process’s operation properly, assign it a clearly limited scope and time of operation, and optimize switching between tasks. For this, we used the following simple but very effective ✨tips and tricks✨:

• Browsers have a lot of work to do but limited resources for that. Therefore, we should prioritize all tasks and actions so that everything is on time.

• Each millisecond should be used. So loading and calculation processing during the browser’s idle states can be performed to optimize spare free time. 

• Remember about rendering and avoid unnecessary redraws. Some actions with the DOM or individual attributes of visual elements may cause the need for additional changes for styles and layouts, which also takes time and breaks the clear sequence of frames.

• It would be correct to avoid resource-intensive tasks. If something similar needs to be performed, the task should be divided into smaller subtasks.

• In most situations, when working with large data sets, not the whole data set needs to be loaded fully at once into the browser. Instead, the data should be loaded by parts on demand during runtime.

Each of these small steps helps to keep the frame in place and processes running. To implement some of these approaches, we used some extra help.

Pseudo-multithreading 

For reaching smooth navigation (e.g., scrolling, searching for a certain result), the result needs to be rendered in a smooth and even fashion. That way, the display won’t be choppy, jerky, or uneven without sudden jolts. 

All this has something to do with how the frame is redrawn and the animation is presented. To avoid unnecessary redraws, we used requestAnimationFrame().

requestAnimationFrame() is a method used to execute the rendering code on the next available screen repaint to avoid possible reflows caused by the recalculation of previously rendered elements. This means the drawing process is executed when changes are ready, thus providing a more efficient repaint, resulting in smoother and more pleasant user interaction.

This is a great place to run code that doesn’t deal with handling user input before redrawing directly. However, specific calculations should be avoided in the method, as this can lead to forced layout synchronization. In this phenomenon, the layout change happens before the styles are recalculated.

Optimizing redrawing time, as well as other operations, can free up some time in a frame. And that’s a perfect possibility to complete some small but essential tasks. 

requestIdleCallback() is an elegant way to identify the most appropriate moment for executing under-the-hood tasks, as well as predict how much time is left until the idle stage changes. Therefore it makes it possible to estimate whether a certain process can be completed at the given moment, thence carrying out the procedures explicitly during idle times. The main idea is to use the browser’s resources as efficiently as possible; that way, we’ll prioritize event handling, rendering, etc., and the script we pass to execute in requestIdleCallback() will run when possible, having a lower priority than other processes.

In combination, these methods allow us to fit the constraint of 16.7 milliseconds per frame and maintain a steady frame rate. Nothing extra is done, and free time is used as productively as possible.

What have we already achieved?

Everything described above was successfully implemented in the technology used by DataTableDev and showed excellent results.

Currently, we have a working demo that can demonstrate the current development of the approach. It works perfectly well with 11 million records, shows excellent performance, clean, smooth navigation, and super fast execution and visualization of the main functions of the grid. You can play with it and evaluate the level of grid development.

Of course, behind all this, there is also a server part and a smart, fast data transfer algorithm, which are an extremely important part of the technology and its main basis.

We don’t immerse you so much in it yet, but we promise there will soon be something to see there too.

What are the future plans?

For further development, we want to be sure that it is necessary and exciting for users and will really be the kind of breakthrough we are counting on. 

We continue to work and research to make the technology even more perfect and unique because we plan to use it not only for DataTableDev. In the future, this approach will be integrated into other products of our line and may also become the basis for other data visualization technologies that we have long dreamed of.

But for this, we need the help of data analytics and visualization experts in the form of your thoughts, advice, and feedback on the idea of our technology and product in time. 

How can you help us right now?

1. Check the demo and explore the capabilities of the grid
2. Write in our feedback form your thoughts and recommendations
3. Share this article with someone you think would be interested in having the tool we are developing.

We would appreciate such actions and are happy to continue our work to make the world of data visualization accessible and open to all.