Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gender checks that use descendant concepts + meas unit pairs cleared #524

Merged
merged 8 commits into from
Feb 15, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added male specific procedures, cleaned up concept_plausible_gender_u…
…se_descendants.sql, added missing check_description
Dmitry Dymshyts committed Feb 14, 2024
commit 213de7fcc7f4c57d618ce1c88dda07195c7b69dc
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -46,5 +46,5 @@ Suggests:
R.utils
Remotes:
ohdsi/Eunomia
RoxygenNote: 7.2.2
RoxygenNote: 7.2.3
Encoding: UTF-8
3 changes: 3 additions & 0 deletions inst/csv/OMOP_CDMv5.2_Check_Descriptions.csv
Original file line number Diff line number Diff line change
@@ -23,3 +23,6 @@ FIELD,withinVisitDates,The number and percent of records not within one week on
CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!=''
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you actually please put the concept checks at the end of the file, after the new plausible temporal checks? just so all FIELD are grouped together, and then CONCEPT :)

CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!=''
CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds), '-1' means unit_concept_id can be null).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!=''
FIELD,plausibleAfterBirth,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs prior to birth.",Verification,Plausibility,Temporal,field_plausible_after_birth.sql,plausibleAfterBirth=='Yes'
FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes'
FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes'
1 change: 1 addition & 0 deletions inst/csv/OMOP_CDMv5.2_Concept_Level.csv
Original file line number Diff line number Diff line change
@@ -335,6 +335,7 @@ CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,45772671,Nodular prostate without urin
CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,"4090861, 4025213","Male genitalia finding, Male reproductive finding",,,,,,,,,,Male,,2,,,,,,,,,,,,,
CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,"4095793 , 443343, 4024004 , 4172857, 444094 , 197810, 4158481","Female genitalia finding, Disorder of intrauterine contraceptive device, Menopause finding, Disorder of female genital system, Malignant neoplasm of uterine adnexa, Finding related to pregnancy, Female reproductive finding",,,,,,,,,,Female,,2,,,,,,,,,,,,,
PROCEDURE_OCCURRENCE,PROCEDURE_CONCEPT_ID,4041261,Procedure on female genital system,,,,,,,,,,Female,,2,,,,,,,,,,,,,
PROCEDURE_OCCURRENCE,PROCEDURE_CONCEPT_ID,"4250917, 4077750, 4043199, 4040577","Operation on prostate, Operation on scrotum, Procedure on penis, Procedure on testis",,,,,,,,,,Male,,2,,,,,,,,,,,,,
MEASUREMENT,MEASUREMENT_CONCEPT_ID,3006315,Basophils [#/volume] in Blood,,,,,,,,,,,,,,,,,,,,,,,"8784,8848,8961,9444",5,Approved UNIT_CONCEPT_IDs for the given MEASUREMENT_CONCEPT_ID
MEASUREMENT,MEASUREMENT_CONCEPT_ID,3004410,Hemoglobin A1c/Hemoglobin.total in Blood,,,,,,,,,,,,,,,,,,,,,,,"8554,8737,9225,9579",5,Approved UNIT_CONCEPT_IDs for the given MEASUREMENT_CONCEPT_ID
MEASUREMENT,MEASUREMENT_CONCEPT_ID,40487382,Total lymphocyte count,,,,,,,,,,,,,,,,,,,,,,,"8784,8848,8961",5,Approved UNIT_CONCEPT_IDs for the given MEASUREMENT_CONCEPT_ID
3 changes: 3 additions & 0 deletions inst/csv/OMOP_CDMv5.3_Check_Descriptions.csv
Original file line number Diff line number Diff line change
@@ -23,3 +23,6 @@ FIELD,withinVisitDates,The number and percent of records not within one week on
CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!=''
CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!=''
CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds), '-1' means unit_concept_id can be null).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!=''
FIELD,plausibleAfterBirth,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs prior to birth.",Verification,Plausibility,Temporal,field_plausible_after_birth.sql,plausibleAfterBirth=='Yes'
FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes'
FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes'
1 change: 1 addition & 0 deletions inst/csv/OMOP_CDMv5.3_Concept_Level.csv
Original file line number Diff line number Diff line change
@@ -335,6 +335,7 @@ CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,45772671,Nodular prostate without urin
CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,"4090861, 4025213","Male genitalia finding, Male reproductive finding",,,,,,,,,,Male,,2,,,,,,,,,,,,,
CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,"4095793 , 443343, 4024004 , 4172857, 444094 , 197810, 4158481","Female genitalia finding, Disorder of intrauterine contraceptive device, Menopause finding, Disorder of female genital system, Malignant neoplasm of uterine adnexa, Finding related to pregnancy, Female reproductive finding",,,,,,,,,,Female,,2,,,,,,,,,,,,,
PROCEDURE_OCCURRENCE,PROCEDURE_CONCEPT_ID,4041261,Procedure on female genital system,,,,,,,,,,Female,,2,,,,,,,,,,,,,
PROCEDURE_OCCURRENCE,PROCEDURE_CONCEPT_ID,"4250917, 4077750, 4043199, 4040577","Operation on prostate, Operation on scrotum, Procedure on penis, Procedure on testis",,,,,,,,,,Male,,2,,,,,,,,,,,,,
OBSERVATION,OBSERVATION_CONCEPT_ID,37393449,Plasma total cholesterol level,8840,milligram per deciliter,50,5,,500,5,,,,,,,,,,,,,,,,,,
OBSERVATION,OBSERVATION_CONCEPT_ID,37393449,Plasma total cholesterol level,8753,millimole per liter,1,5,,15,5,,,,,,,,,,,,,,,,,,
OBSERVATION,OBSERVATION_CONCEPT_ID,37397989,Serum total cholesterol level,8840,milligram per deciliter,50,5,,500,5,,,,,,,,,,,,,,,,,,
3 changes: 3 additions & 0 deletions inst/csv/OMOP_CDMv5.4_Check_Descriptions.csv
Original file line number Diff line number Diff line change
@@ -23,3 +23,6 @@ FIELD,withinVisitDates,The number and percent of records not within one week on
CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!=''
CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!=''
CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds), '-1' means unit_concept_id can be null).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!=''
FIELD,plausibleAfterBirth,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs prior to birth.",Verification,Plausibility,Temporal,field_plausible_after_birth.sql,plausibleAfterBirth=='Yes'
FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes'
FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes'
1 change: 1 addition & 0 deletions inst/csv/OMOP_CDMv5.4_Concept_Level.csv
Original file line number Diff line number Diff line change
@@ -335,6 +335,7 @@ CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,45772671,Nodular prostate without urin
CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,"4090861, 4025213","Male genitalia finding, Male reproductive finding",,,,,,,,,,Male,,2,,,,,,,,,,,,,
CONDITION_OCCURRENCE,CONDITION_CONCEPT_ID,"4095793 , 443343, 4024004 , 4172857, 444094 , 197810, 4158481","Female genitalia finding, Disorder of intrauterine contraceptive device, Menopause finding, Disorder of female genital system, Malignant neoplasm of uterine adnexa, Finding related to pregnancy, Female reproductive finding",,,,,,,,,,Female,,2,,,,,,,,,,,,,
PROCEDURE_OCCURRENCE,PROCEDURE_CONCEPT_ID,4041261,Procedure on female genital system,,,,,,,,,,Female,,2,,,,,,,,,,,,,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for not including 4041262 - Procedure on male genital system?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OMOP vocabulary hierarchy is wrong in some concepts here, HCPCS "procedures on prostate OR perineum" has both parents "procedures on prostate" and "procedures on perineum". but it's OR and these codes are used in females.
Once the vocabulary is fixed, we can add these

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not able to find the concepts in the example you are referring to.

In any case, I suggest to not wait for a vocabulary update, and already add a set of more specific concepts for male procedures that do not include prostate. e.g. these three:

  • 4040708 - Procedure on scrotum
  • 4043199 - Procedure on penis
  • 4040577 - Procedure on testis

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, sorry, it was Procedure on scrotum that lead to concept:
2313992 | 93975 | Duplex scan of arterial inflow and venous outflow of abdominal, pelvic, scrotal contents and/or retroperitoneal organs; complete study
Procedure on prostate leads to the 2211770 76872 Ultrasound, transrectal, which can be done for females as well.
So, I ended up with the following list
4250917 - Operation on prostate
4077750 - Operation on scrotum
4043199 - Procedure on penis
4040577 - Procedure on testis

PROCEDURE_OCCURRENCE,PROCEDURE_CONCEPT_ID,"4250917, 4077750, 4043199, 4040577","Operation on prostate, Operation on scrotum, Procedure on penis, Procedure on testis",,,,,,,,,,Male,,2,,,,,,,,,,,,,
OBSERVATION,OBSERVATION_CONCEPT_ID,37393449,Plasma total cholesterol level,8840,milligram per deciliter,50,5,,500,5,,,,,,,,,,,,,,,,,,
OBSERVATION,OBSERVATION_CONCEPT_ID,37393449,Plasma total cholesterol level,8753,millimole per liter,1,5,,15,5,,,,,,,,,,,,,,,,,,
OBSERVATION,OBSERVATION_CONCEPT_ID,37397989,Serum total cholesterol level,8840,milligram per deciliter,50,5,,500,5,,,,,,,,,,,,,,,,,,
18 changes: 9 additions & 9 deletions inst/sql/sql_server/concept_plausible_gender_use_descendants.sql
Original file line number Diff line number Diff line change
@@ -33,14 +33,14 @@ FROM
/*violatedRowsBegin*/
SELECT cdmTable.*
FROM @cdmDatabaseSchema.@cdmTableName cdmTable
INNER JOIN @cdmDatabaseSchema.person p
ON cdmTable.person_id = p.person_id
INNER JOIN @vocabDatabaseSchema.concept_ancestor ca
ON ca.descendant_concept_id = cdmTable.@cdmFieldName
JOIN @cdmDatabaseSchema.person p
ON cdmTable.person_id = p.person_id
JOIN @vocabDatabaseSchema.concept_ancestor ca
ON ca.descendant_concept_id = cdmTable.@cdmFieldName
{@cohort}?{
JOIN @cohortDatabaseSchema.@cohortTableName c
ON cdmTable.person_id = c.subject_id
AND c.cohort_definition_id = @cohortDefinitionId
JOIN @cohortDatabaseSchema.@cohortTableName c
ON cdmTable.person_id = c.subject_id
AND c.cohort_definition_id = @cohortDefinitionId
}
WHERE ca.ancestor_concept_id in (@conceptId)
AND p.gender_concept_id <> {@plausibleGenderUseDescendants == 'Male'} ? {8507} : {8532}
@@ -51,10 +51,10 @@ FROM
SELECT
COUNT_BIG(*) AS num_rows
FROM @cdmDatabaseSchema.@cdmTableName cdmTable
INNER JOIN @vocabDatabaseSchema.concept_ancestor ca
JOIN @vocabDatabaseSchema.concept_ancestor ca
ON ca.descendant_concept_id = cdmTable.@cdmFieldName
{@cohort}?{
JOIN @cohortDatabaseSchema.@cohortTableName c
JOIN @cohortDatabaseSchema.@cohortTableName c
ON cdmTable.person_id = c.subject_id
AND c.cohort_definition_id = @cohortDefinitionId
}

Unchanged files with check annotations Beta

} else if (checkResults[i, ]$checkLevel == "CONCEPT") {
if (is.na(checkResults[i, ]$unitConceptId) &&
grepl(",", checkResults[i, ]$conceptId)) {
thresholdFilter <- sprintf(
"conceptChecks$%s[conceptChecks$cdmTableName == '%s' &
conceptChecks$cdmFieldName == '%s' &
conceptChecks$conceptId == '%s']",
thresholdField,
checkResults[i, ]$cdmTableName,
checkResults[i, ]$cdmFieldName,
checkResults[i, ]$conceptId

Check warning on line 90 in R/evaluateThresholds.R

Codecov / codecov/patch

R/evaluateThresholds.R#L83-L90

Added lines #L83 - L90 were not covered by tests
)
notesFilter <- sprintf(
"conceptChecks$%s[conceptChecks$cdmTableName == '%s' &
conceptChecks$cdmFieldName == '%s' &
conceptChecks$conceptId == '%s']",
notesField,
checkResults[i, ]$cdmTableName,
checkResults[i, ]$cdmFieldName,
checkResults[i, ]$conceptId

Check warning on line 99 in R/evaluateThresholds.R

Codecov / codecov/patch

R/evaluateThresholds.R#L92-L99

Added lines #L92 - L99 were not covered by tests
)
}
else if (is.na(checkResults[i, ]$unitConceptId) &&