Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Map Reduce Design Pattern #3184

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Conversation

hvgh88
Copy link

@hvgh88 hvgh88 commented Jan 31, 2025

Description:
Implemented MapReduce design pattern

Issue: Close #2927

Copy link

github-actions bot commented Jan 31, 2025

PR Summary

This PR implements the MapReduce design pattern, enabling parallel processing of large datasets. It includes the core components: Mapper, Reducer, and Shuffler, along with a MapReduce orchestrator and comprehensive unit tests. The implementation processes text, counting word occurrences. The design is explained in detail within the README, including real-world examples and comparisons to other patterns.

Changes

File Summary
map-reduce/README.md This file provides a comprehensive guide to the MapReduce design pattern. It includes a detailed explanation, real-world examples, a programmatic example in Java, and a discussion of when to use this pattern. It also covers benefits, trade-offs, and related patterns.
map-reduce/etc/map-reduce.png New file: Diagram showing the MapReduce pattern.
map-reduce/etc/map-reduce.urm.puml New file: PlantUML class diagram for the MapReduce implementation.
map-reduce/pom.xml New file: Maven project configuration file for the MapReduce implementation.
map-reduce/src/main/java/com/iluwatar/Main.java This Java class serves as the entry point for the MapReduce program. It sets up the input data and calls the MapReduce class to process it, then prints the results.
map-reduce/src/main/java/com/iluwatar/MapReduce.java This Java class orchestrates the MapReduce process. It takes a list of strings as input, calls the mapper, shuffler, and reducer, and returns the final result.
map-reduce/src/main/java/com/iluwatar/Mapper.java This Java class implements the map function, processing each input string and generating key-value pairs (word, count).
map-reduce/src/main/java/com/iluwatar/Reducer.java This Java class implements the reduce function, aggregating the values for each key (word) to produce the final word counts.
map-reduce/src/main/java/com/iluwatar/Shuffler.java This Java class implements the shuffle function, grouping the key-value pairs generated by the mappers based on the keys.
map-reduce/src/test/java/com/iluwatar/MapReduceTest.java This Java class contains unit tests for the MapReduce class, verifying the correctness of the overall MapReduce process.
map-reduce/src/test/java/com/iluwatar/MapperTest.java This Java class contains unit tests for the Mapper class, ensuring the mapper correctly processes input strings and generates word counts.
map-reduce/src/test/java/com/iluwatar/ReducerTest.java This Java class contains unit tests for the Reducer class, verifying the reducer correctly aggregates word counts.
map-reduce/src/test/java/com/iluwatar/ShufflerTest.java This Java class contains unit tests for the Shuffler class, checking that it correctly groups the key-value pairs.

autogenerated by presubmit.ai

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Review Summary

Commits Considered (2)
Files Processed (13)
  • map-reduce/README.md (1 hunk)
  • map-reduce/etc/map-reduce.png (0 hunks)
  • map-reduce/etc/map-reduce.urm.puml (1 hunk)
  • map-reduce/pom.xml (1 hunk)
  • map-reduce/src/main/java/com/iluwatar/Main.java (1 hunk)
  • map-reduce/src/main/java/com/iluwatar/MapReduce.java (1 hunk)
  • map-reduce/src/main/java/com/iluwatar/Mapper.java (1 hunk)
  • map-reduce/src/main/java/com/iluwatar/Reducer.java (1 hunk)
  • map-reduce/src/main/java/com/iluwatar/Shuffler.java (1 hunk)
  • map-reduce/src/test/java/com/iluwatar/MapReduceTest.java (1 hunk)
  • map-reduce/src/test/java/com/iluwatar/MapperTest.java (1 hunk)
  • map-reduce/src/test/java/com/iluwatar/ReducerTest.java (1 hunk)
  • map-reduce/src/test/java/com/iluwatar/ShufflerTest.java (1 hunk)
Actionable Comments (0)
Skipped Comments (0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

MapReduce design pattern
1 participant