Distributed Hash Sorting

Problem

Measure how hash processing and sorting behave as parallelism increases, from shared memory to a distributed cluster. The study compares Hadoop MapReduce and Apache Spark on datasets up to 64 GB, roughly four billion 16-byte BLAKE3 records.

My contribution

I wrote the C++ and OpenMP shared-memory implementations, generated the benchmark data, and deployed the Hadoop and Spark clusters on Chameleon Cloud. I also wrote the Java implementations and Bash orchestration used to configure and run the distributed experiments.

Technical decisions

I compared the frameworks on the same sorting workload and measured scaling from one to eight nodes. That makes memory use, network shuffle, and disk I/O visible as practical constraints on distributed performance.

A single Bash orchestration command handled cluster provisioning. Automating setup made it easier to repeat the experiments with a consistent configuration.

Results

Spark sorted about 2x faster than Hadoop in this workload and cluster configuration. Scaling from one to eight nodes yielded roughly 6 to 7x speedups before network shuffle and disk I/O limited further gains. These results describe this benchmark, not a general speed advantage across workloads.

Automated provisioning reduced the observed setup time from hours to under two minutes in the project's environment.