[dataset]: LA MBON Biodiversity Information 2023

[SandeepJilla]

Contact details

sandeep.jilla@tamucc.edu

Dataset Title

LA MBON Biodiversity Information 2023

Describe your dataset and any specific challenges or blockers you have or anticipate.

GCOOS is working with the LA MBON Oysters data, and the raw data has been made available on the GCOOS ERDDAP. We will be working on aligning this dataset with the Darwin Core (DwC) standard. We may need your guidance on how to handle or incorporate the non-standard variables appropriately within the DwC framework.

Info about "raw" Data Files.

The raw data is available as .csv files through the ERDDAP file system at hhttps://gcoos4.geos.tamu.edu/erddap/files/la_mbon_2023_biodiversityinformation/ . The dataset includes information such as Site_ID, Tray_ID, date_retrieved, date_processed, taxaID, scientificName, scientificNameID, AphiaID, abundance, wet_weight and dry_weight

[MathewBiddle]

Hi @SandeepJilla, this is awesome! Thanks for posting the raw data on ERDDAP. For potential users of the data, and to help with Darwin Core conversion, would it be possible to include the coordinates of each of the sites in latitude and longitude variables?

[SandeepJilla]

Yes, below are the coordinates of each of the sites in the latitude and longitude variables. During the Data Mobilization workshop, we also curated the Darwin Core files for this dataset. Could you please help review them and let us know if they are ready to be published on the IPT?

biodiversityinformation.zip

<style> </style>

Site.ID	Bay	Latitude	Longitude
LUMO3	Terrebonne	29.25615	-90.66753
SL1	Terrebonne	29.26561	-90.92385
TB1	Terrebonne	29.06067	-90.7126
BB1	Barataria	29.24561	-90.01701
BB2	Barataria	29.40507	-90.02205
BB3	Barataria	29.41415	-90.04591

@fgayanilo

[MathewBiddle]

@SandeepJilla this is awesome! Do you mind adding those coordinates to the source data on ERDDAP? In pandas you could use the .merge(on="Site.ID") function. https://gcoos4.geos.tamu.edu/erddap/tabledap/la_mbon_2023_biodiversityinformation.html

I'll take a look at your DwC package soon.

[fgayanilo]

@MathewBiddle Thanks a lot, we want this bunch out of our desk the soonest -- to mean, we want to be able to submit it to IPT. I think we're almost there

[SandeepJilla]

@SandeepJilla this is awesome! Do you mind adding those coordinates to the source data on ERDDAP? In pandas you could use the .merge(on="Site.ID") function. https://gcoos4.geos.tamu.edu/erddap/tabledap/la_mbon_2023_biodiversityinformation.html

I'll take a look at your DwC package soon.

We have added the location coordinates to the source data on ERDDAP https://gcoos4.geos.tamu.edu/erddap/tabledap/la_mbon_2023_biodiversityinformation.html

[MathewBiddle]

I'm poking around now and found that your event file has duplicate eventID values. These should be unique within the event file and match to values listed in the occurrence file.

df_event.loc[df_event.duplicated(subset='eventID', keep=False)].sort_values('eventID')

Row Number	eventID
50	BB1_green_20231030
29	BB1_green_20231030
28	BB2_white_20231030
49	BB2_white_20231030
47	BB3_black_20231030
20	BB3_black_20231030
48	BB3_silver_20231030
26	BB3_silver_20231030
27	BB3_silver_20231030
24	LUMO3_black_20231223
25	LUMO3_black_20231223
23	LUMO3_white_20231223
22	LUMO3_white_20231223
16	TB1_1_20230807
46	TB1_1_20230807
45	TB1_3_20230807
15	TB1_3_20230807
44	TB1_5_20230807
13	TB1_5_20230807
43	TB1_6_20230807
12	TB1_6_20230807

and it looks like your occurrence file has duplicated occurrenceIDs

print(df_occurrence.loc[df_occurrence.duplicated(subset='occurrenceID', keep=False)].sort_values('occurrenceID').to_markdown())

	occurrenceID	eventID	basisOfRecord	scientificName	scientificNameID	taxonID	organismQuantity	organismQuantityType	dateIdentified	occurrenceStatus	occurrenceRemarks
210	BB3_white_20231030_A106765	BB3_white_20231030	MaterialSample	Panopeidae spp.	urn:lsid:marinespecies.org:taxname:106765	106765	100	individuals	12/19/2023	present	nan
214	BB3_white_20231030_A106765	BB3_white_20231030	MaterialSample	Panopeidae spp.	urn:lsid:marinespecies.org:taxname:106765	106765	2	individuals	12/19/2023	present	nan

It looks like those two rows could be merged to make a single row with organismQuantity of 102. Or, you might want to add in organismQuantity values to your occurrenceID building process. Since those values are what sets these two rows apart.

cc: @sformel and @laurabrenskelle if they see something I might be missing.

[MathewBiddle]

Also, I think the emof file doesn't need to have eventID in it. Since the emof is linked to the occurrence via occurrenceID, the eventID can be found there and thus duplicative in the emof file.

Again, @sformel and @laurabrenskelle please add comments if I am misunderstanding this.

[SandeepJilla]

I have updated the event file to remove the duplicates. In the occurrence file, I merged the two rows with the same occurrenceID and updated the organismQuantity to 102. In the eMoF file, I summed the values for DRY WEIGHT and WET WEIGHT for BB3_white_20231030_A106765. Could you please review the updated files and confirm if they are good to be published on IPT?

biodiversityinformation_updated.zip

[fgayanilo]

@MathewBiddle Thanks and are we now ready for IPT submission?

[MathewBiddle]

Looking good! Let's start drafting a dataset in the OBIS-USA IPT.

FYI, I've been working on a simple checker to review files for some minimal darwin core compliance, see the section "Matt's home grown code for checking" in this Gist. Hopefully, this can continue to evolve into a simple online checker to see if you're data files are at least in the ballpark. I used this to find the issues described above.

[fgayanilo]

That's nice, and more are expected -- to mean, tools like that are certainly needed. Yes, I'll start drafting docs for IPT early next week.

@SandeepJilla, thanks for the changes made. We might forget this, but please do not forget to update the files on ERDDAP.