Sunday, November 19, 2017

Decimate Data with JavaScript's Filter Function

In 2011, the Javascript standard added several new array functions that greatly simplify the process of working with large datasets.  One of the challenges when working with "big data" is when you go to chart that data, rendering every individual data point can seriously hamper the performance of your chart, increasing render times or locking your browser for a while, or crashing.

As I skimmed through several internet postings on different elaborate methods for decimating data (that is, reducing the resolution of a dataset through the application of some function), I realized that the filter function would perform all the work easily.

Consider an array of values:

var a = "4.901712886872069,4.905571030847647,4.909414346738851,4.913242948087607,4.91705694713669,4.92085645484947,4.92464158092928,4.928412433838424,4.932169120816823,4.93591174790032,4.939640419938632,4.94335524061298,4.947056312453386,4.950743736855651,4.954417614098028,4.958078043357581,4.961725122726249,4.965358949226622,4.968979618827425,4.972587226458727,4.976181866026878,4.97976363042917,4.983332611568248,4.986888900366257,4.990432586778738,4.9939637598082856,4.997550054547602,5.001123533723662,5.004684288602818,5.008232409479947,5.011767985692191,5.015291105632452,5.018801856762651,5.022300325626761,5.025786597863601,5.029260758219422,5.032722890560263,5.036173077884096,5.039611402332775,5.043037945203758,5.046452786961651,5.049856007249539,5.053247684900134,5.05662789794673,5.059996723633982,5.063354238428486,5.066700518029206,5.070035637377705,5.07335967066822,5.076672691357564,5.079974772174871,5.083265985131165,5.086546401528796,5.0898160919706985,5.0930751263695155,5.096323573956563,5.099561503290659,5.102788982266802,5.106006078124715,5.109212857457251,5.112469599977525,5.115715770546765,5.1189514375797165,5.122176668829165,5.125391531394446,5.128596091729822,5.131790415652724,5.134974568351865,5.138148614395217,5.141312617737873,5.144466641729777,5.147610749123334,5.150745002080902,5.153869462182165,5.156984190431394,5.160089247264595,5.163184692556541,5.166270585627705,5.169346985251077,5.172413949658882,5.175471536549192,5.178519803092442,5.181558805937842,5.184588601219694,5.187609244563616,5.190620791092663,5.193623295433372,5.1966168117216975,5.199601393608879,5.202577094267201,5.205543966395682,5.208556759340414,5.211560502621755,5.214555250442706,5.21754105652075,5.220517974093625,5.223486055925034,5.2264453543102425,5.229395921081616,5.232337807614062,5.235271064830402,5.238195743206656,5.241111892777258,5.2440195631401885,5.246918803462037,5.249809662482995,5.252692188521762,5.255566429480405,5.258432432849123,5.261290245710962,5.264139914746451,5.266981486238185,5.269815006075326,5.272640519758055,5.275458072401956,5.278267708742335,5.281069473138485,5.283863409577888,5.286649561680356,5.289427972702121,5.292198685539861,5.295011909539312,5.2978172415063565,5.300614725596722,5.303404405596596,5.306186324926734,5.308960526646518,5.311727053457957,5.314485947709622".split(',');

Now let's say I want to decimate that so that a small status chart only ever shows around 10 data points.  More or less are fine as long as I'm not continuing to show more data as I add several more batches to this initial set.  The following filter function uses the modulo operator to make a simple reduction function.

a.filter(function(elem,index,array){return index % (Math.round(array.length/10)) ==0;});

...yields a set of 10 data points, one for every time the array index of the value divided the dynamic value  array.length/10 is equal to zero.  Math.round simply makes our dynamic value a whole number, which works well with modulo (%) against the integer index numbers in the array.

"4.901712886872069,4.950743736855651,4.997550054547602,5.043037945203758,5.086546401528796,5.128596091729822,5.169346985251077,5.208556759340414,5.246918803462037,5.283863409577888"

Now any charting we may do with this data will continue to tell the story, sacrificing resolution in exchange for performance.