The main purpose of the distribution is to enable handling any data types and formats by combining various technology solutions and data processing architectures.
The Arenadata Hadoop 2.1.4 release comes with Spark 3.1 functionality. Both Spark 2 and Spark 3.1 will be available in Arenadata Hadoop. The two framework versions will run independently of each other.
“We have finished testing the Spark 3 tools and ensured native integration with other components of the platform. From now on our customers will be able to test Spark 3 functionality and gradually migrate to the new framework version.”