Add advanced functionality to handle processing of big datasets

[iretex]

Adding pyspark module to be triggered in the event that the size of input dataset is bigger than 1mil rows.
This capability should improve the runtime generally and give improved user experience.

[krisblarq]

Hi @risenW i'll like to work on this please.

[risenW]

Sure, please go ahead

[iretex]

Krisblarq, do you want to start writing the test cases, how do you think we can collaborate? Rising, I imagine that you'll like to retain existing code structure, which could mean that the pyspark code will be maintained as a separate module? If we agree on the structure, the development can be accelerated. Very Best! Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Rising Odegua <notifications@github.com> Sent: Saturday, August 1, 2020 9:24:56 AM To: risenW/datasist <datasist@noreply.github.com> Cc: Iretioluwa Olawuyi <iretioluwa.olawuyi@outlook.com>; Author <author@noreply.github.com> Subject: Re: [risenW/datasist] Add advanced functionality to handle processing of big datasets (#55) Sure, please go ahead — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#55 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGC2IF6RTXAWW6MRM3VATATR6PGNRANCNFSM4OZXKWRA>.

[risenW]

Yes @iretex the module should be separate. Then it can be called from other modules when we intend to use it. This should be invisible to the user as well.

[iretex]

Very clear!

…

________________________________ From: Rising Odegua <notifications@github.com> Sent: Saturday, August 1, 2020 1:31:56 PM To: risenW/datasist <datasist@noreply.github.com> Cc: Iretioluwa Olawuyi <iretioluwa.olawuyi@outlook.com>; Mention <mention@noreply.github.com> Subject: Re: [risenW/datasist] Add advanced functionality to handle processing of big datasets (#55) Yes @iretex<https://github.com/iretex> the module should be separate. Then it can be called from other modules when we intend to use it. This should be invisible to the user as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#55 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGC2IFZYRSCOOMDXIGENNKDR6QDLZANCNFSM4OZXKWRA>.