Blog

One of the questions we’ve spent a lot of time pondering over the years is a deceptively simple one - How do we visualize billions of data points? To understand why this is a difficult question to answer, let’s look at two popular approaches.

We’ve been writing libraries for Spark at Uncharted for several years, and continuous integration involving the Spark runtime has always been a difficult thing to accomplish. Common approaches, such as creating a Spark context within a standard Scala runtime, can fail to accurately emulate nuances of the distributed Spark environment. I’d like to share a solution we’ve developed for creating a native Spark test environment within TravisCI.