Remove Duplicates – GriddedDataConcatenate

[grkal]

Dear GROOPS Team,

The content of this message concerns a possible error presented after investigating a possible removal of identical points when using the “GriddedDataConcatenate” function.

As an example, I will use two simple “GriddedData” files (each of which contains just 2 data lines) which I try to concatenate into a single “GriddedData” file with the “GriddedDataConcatenate” function after specifying the value of the “removeDuplicates” config element as “keepFirst”, “keepLast” and “< none >”; for all three cases the value of the “sortPoints” config element is set to “no”.

Given that the two data files contain no identical points within the specified margin [1e-5], all three cases described above would be expected to yield the same result. However, the output files of the first two cases differ from the one of the last case (no removal of duplicates); specifically, their data lines are the ones included in the two input files, but they are not in the correct order as it is the case with the output file of the last case.

Why are the data lines of the concatenated file mixed up and not in the correct order when the “removeDuplicates” config element is enabled?

Attached you will find the “RemoveDuplicatesGriddedDataConcatenate.zip” file which includes the two “GriddedData” files [“GT_1.txt” & “GT_2.txt”] and the “RemoveDuplicatesGriddedDataConcatenate.xml” file which is a simple script I created just containing the three cases described above using the “GriddedDataConcatenate” program.

Thank you very much in advance for your time and looking forward to your feedback.

Best regards,
Grigorios Kalimeris
Chair of Astronomical and Physical Geodesy
Technical University of Munich

RemoveDuplicatesGriddedDataConcatenate.zip

[sestras]

Hello Grigorios,

Removing duplicates is implemented by sorting all points and then removing identical subsequent points. Therefore, whenever you use keepFirst or keepLast, the concatenated points are sorted, no matter if you set sortPoints to yes or no.

You can check the code here:

groops/source/programs/griddedData/griddedDataConcatenate.cpp

Lines 147 to 176 in bd72c26

	if(sortPoints || (keep != KEEPALL))
	{
	logStatus<<"sort points"<<Log::endl;
	grid.sort();
	}
	// =====================================================
	// eliminate duplicates
	// --------------------
	if(keep != KEEPALL)
	{
	logStatus<<"eliminate duplicates"<<Log::endl;
	UInt count = std::distance(grid.points.begin(), std::adjacent_find(grid.points.begin(), grid.points.end(),
	[&](const Vector3d &x, const Vector3d &y) {return (x-y).r() <= margin;}));
	if(count < grid.points.size())
	for(UInt i=++count; i<grid.points.size(); i++)
	{
	if((grid.points.at(i)-grid.points.at(count-1)).r() > margin)
	movePoints(i, count++);
	else if(keep == KEEPLAST)
	movePoints(i, count-1);
	}
	logInfo<<" "<<grid.points.size()-count<<" duplicates removed!"<<Log::endl;
	grid.points.resize(count);
	if(grid.areas.size())
	grid.areas.resize(count);
	for(UInt k=0; k<grid.values.size(); k++)
	grid.values.at(k).resize(count);
	}

If you check your logs, it should state "sort points". However, it does not seem like this behavior is documented anywhere, so maybe the program's documentation could be improved.

If you need to retain the order of the points for some reason, this would require a different implementation of the removal algorithm,

Best regards,
Sebastian

[grkal]

Dear Sebastian,

Thank you very much for your detailed response. Indeed, the “sort points” message appears in the logs. In my opinion, it would be nice if the “GriddedDataConcatenate” program’s documentation mentioned that enabling the “removeDuplicates” config element implies that the “sortPoints” config element is activated too.

Ideally, I would like to keep the order of the points even after enabling “removeDuplicates” for two reasons.

Firstly, if for example each input gridded data file is generated from the “Orbit2Groundtracks” program, the data lines of the (unsorted) concatenated gridded data file keep a semblance of time information; each data line is “connected” with the respective epoch of the “inputfileOrbit” (even if the epoch is not included in the concatenated gridded data file). In this case, the user knows that each gridded point corresponds to an epoch before the one of the next gridded point and so on. As a result, he can derive information about the direction of the movement of the examined satellite. With the current version of the program, if the user decides to eliminate possible duplicate points, he would not be able to retrieve any time information from the concatenated output file given that the “sortPoints” option would have been implemented.

Secondly, as I have verified from tests that I have run, by not keeping the order of the points in the concatenated gridded data file, it is no longer possible to get the correct visual result if the “line” config element of the “points” layer is enabled in the “PlotMap” program (as it is expected given the sorting of the points); thus, the only way to get the correct visual result is to enable only the “symbol” config element. However, for some applications where the user might want to also show the movement of the satellite as a line on the map (for example, due to a low spatial sampling of the gridded points), it is not possible if the points are sorted (at least, that’s what I have deduced).

Attached you will find the “RemoveDuplicatesGriddedDataConcatenate2.zip” file which is an updated version of the previously attached file. It includes an updated version of the GROOPS script [“RemoveDuplicatesGriddedDataConcatenate2.xml”] and two “GriddedData” files derived from the GRACE-C satellite for the test dates 2019-04-01 & 2019-04-02 [“GT_2019-04-01_C_04.txt” & “GT_2019-04-02_C_04.txt” respectively]. Again, this script contains the three cases (removeDuplicates: “keepFirst”, “keepLast”, “< none >”) using the “GriddedDataConcatenate” program but now also plots the respective concatenated groundtracks by using the “PlotMap” program with the “points” layer enabled and:

• the “symbol” config element enabled and the “line” config element set to “< none >” (first group)
• the “symbol” config element set to “< none >” and the “line” config element enabled (second group)

I have disabled the programs with the “removeDuplicates” enabled related to the second group, since it takes a lot of time to produce the output plots which are also not the proper ones as I have stated above. I have included in the attached “.zip” file the “GT_2019-04-01_2019-04-02_1_lines.png” file so as to show an example of the resulting plots when “removeDuplicates” is enabled and the “lines” config element is selected.

Best regards,
Greg

PS. I also uploaded as an “Idea” a suggestion I have about including a time column in the structure of the “GriddedData” file type.

RemoveDuplicatesGriddedDataConcatenate2.zip