Sparse Engine concept

  • 6 March 2024
  • 3 replies

Userlevel 1
Badge +1

Hi Pigmenteers!

While I’m taking Pigment trainings and explore my personal application, I figure out that there is no information about the size of my application or total number of cells, etc.. and the reason behind is that Pigment is a sparse engine.

Therefore, can you help explain to me more about the concept of sparse engine - what is a sparse engine, how does it actually work and what is the benefit compared to non-sparse engine?


Best answer by Igor P 6 March 2024, 09:05

View original

3 replies

Userlevel 2
Badge +3

Hello @Thu Mai !

Long story short, a sparse engine does not take into account the blank cells, therefore it can perform better and faster compared to other calculation engines (where blanks are filled - densified - with 0 or falses).

Please take look in this article for more details.

Let me know if you have any questions!



Userlevel 6
Badge +14

While a sparse engine is indeed optimised when you do not need all intersections of your dimensions, cells do still count, and if you really want to optimise your model you should try to compute as little as possible.

You’ll find a guide below on how to add the number of cells in each metric in the block explorer.

If you feel a particular block is especially slow, it might be useful to try to analyse wether you absolutely need all cells. 


Hope that helps too!

Userlevel 1
Badge +1

oh got it! thanks @Igor P and @francois a lot!! 😊