What it is
ViiksetJS https://github.com/jamestthompson3/viiksetjs is a data visualization library for React that is built on D3 https://d3js.org and vx. https://vx-demo.now.shIt aims for high performance and ease-of-use so that developers who are not familiar with D3 or other lower level charting libraries can visualize large datasets easily.
Cool features
- Support for livestreaming data via websockets
- Custom chart annotations via text or React components
- Non-blocking data processing
- Fast canvas renderer
Why it was created
ViiksetJS was created at Kamu, the company behind EasyAntiCheat https://www.easy.ac/en-us/ for charting large sets of anti-cheat analysis data. We had tried many of the out-of-the-box charting libraries for React, but none of them fit our requirements of having a easy to understand API and being performant when charting large amounts of data.
While D3 offered a solution to the performance bottlenecks, it often requires a non-trivial time commitment to get past the learning curve. Being a small team with a high workload, many developers felt that this learning curve was too steep and the library too low-level for creating the type of visualizations required by our customers. That's why ViiksetJS aimed to be as simple as possible so that any developer could pick it up and start being productive right away.
What I learned along the way
This was my first ever open source project as well as my first attempt at making a reusable library. Needless to say, it has been a great learning experience! Here are some of the lessons I learned:
Documentation is hard
Documenting all the features you've built and keeping that documentation up to date is almost as difficult a task as writing the code! Luckily now that the library is in Typescript, the code is a bit more self-documenting, but types are not a replacement for well written documentation.
Validate early, validate often
Spending a lot of time on an API of a component or the architecture of a library only for it to be confusing to others is tough. Asking for feedback early and often in the process helps point out weak spots in your thinking!
Rethink dependencies
Getting something out of your mind and into code will always be a sloppy process and there are many libraries that can help smooth out the creases, however, keeping the dependencies used to validate an idea long term can lead to big pains down the road. Creating a library with as few dependencies as possible helps keep the scope down and doesn't tie you to a certain API structure based on your dependencies.
Look at the cause, not just the symptoms
On the surface, ViiksetJS was about solving the problem of high performance + ease-of-use, but those were merely the symptoms of the actual challenge of complex React state management and creating highly configurable components. Asking why something is difficult would have allowed me to avoid some of the mistakes in the API design of ViiksetJS.
What I would change if I were to do it again
Having used ViiksetJS in production for nearly two years, and having gone through many iterations of the API, if I had to create a graphing library again, there are a few things I would change. Here are some examples:
More composable components
One of the most painful parts of using ViiksetJS is creating highly customized components. This is in part due to a poorly thought out API from the beginning, but also in part due to the underlying dependency on VX. With basic components being pulled straight from VX, there is a lot of prop drilling that needs to happen, so components tend to have a lot of available props that take huge config objects. Components that have a large number props or props with complex configurations tend to be fragile and prone to breaking when updating the library or the underlying data structures. Instead of the current design, I would focus on having clear interfaces between components so that they can be easily composed. I would also remove the underlying dependency on VX so that I have a bit more flexibility in the API design and can remove the prop drilling.
Decouple shared state
Currently, state is shared between the visualization components via
React Context. This causes an issue where components are tightly
coupled to their position in the render tree and each component needs
to check whether or not shared props relevant to itself have updated
each time it gets called to re-render. Decoupling this shared state
would give much more flexibility to how components could be laid out,
as well as reduce the amount of memoization checks and logic related
to cloning props to children of children. What I would explore would
be a system of sharing state based on events. This would allow data
processing to happen in a web or service worker instead of on the main
thread. By opening up the ability to do the heavy data work off the
main thread, I could remove hacks being used now such as
queMicrotask calls. Using events would also allow
components to subscribe to only the updates they are interested in,
allowing for easier updates to tooltips, gradual data loading, and
multi-graph filtering.
Visualizations are like onions...
They have layers! One of the most challenging features of ViiksetJS to implement was the custom chart annotations. The reason it was so challenging was because of the above listed poor design choices of tightly coupled state and complex component configurations. To change this, I would try and make it easier for users of the library to think in layers, similar to graphic design software. By layering different elements together, complex visualizations would be much easier to create and debug. Changes to state management and component composition would allow for a layering API to be possible. One of the challenges of a layering API, however, is dealing with the positioning of the elements on the page as well as the medium of which the layers are created. Mixing canvas charts with SVG annotations takes some thought to prevent running into the issue currently faced by the library where elements are highly dependent on their position in the rendering tree.
Looking back
ViiksetJS has been in production for nearly two years and has served my team well. It has good performance, and an easy enough API for visualizations to be created quickly. It's not perfect, but it has been a great learning experience for in learning how to design software for general use cases and diving into the world of data visualization!