Project · 2025
MapReduce vs Spark, benchmarked properly.
A big-data systems study: parallel hash tables in C++, Hadoop and Spark clusters deployed on Chameleon Cloud, and rigorous benchmarks on which framework sorts faster at scale.
What it is
This is a big-data systems study with two parts. The first is shared-memory parallelism: hash tables implemented in C++ and parallelized with OpenMP. The second is distributed sorting at scale: Hadoop MapReduce and Apache Spark clusters deployed on Chameleon Cloud, benchmarked against each other on the same sorting workloads.
Why it matters
MapReduce versus Spark is a classic engineering trade-off, and the honest answer depends on workload and scale. Instead of repeating folklore, this project deploys both frameworks on real cloud clusters and measures which one sorts faster under controlled, comparable conditions.
The work covers the full systems stack: writing the C++ and Java implementations, standing up and configuring the clusters on Chameleon Cloud, and designing benchmarks rigorous enough that the results mean something.
More projects
- Local RAG App: An offline retrieval-augmented generation (RAG) document chatbot built by Om Patel with Python, FastAPI, Ollama, and ChromaDB: local LLM answers with source citations and no data leaving the machine.
- Legal One: A real-time legal consultation web platform Om Patel designed, built, and deployed solo: encrypted Socket.IO messaging, role-based access for clients, lawyers, and admins, and GraphQL on Next.js, Prisma, and MongoDB.
- SW Habitation: The production marketing site and blog platform Om Patel built for the SW Habitation development studio with Next.js, TypeScript, and Tailwind CSS, with a scalable content system and performance and SEO work.