Explore the Fecal Calprotectin Data Set

The fcp dataset is the result of a data warehouse lab query for 65 subjects participating in a clinical research study.

Fecal calprotectin is a neutrophil protein that is stable in stool, and can be measured quantitatively with an ELISA assay. It quantitates inflammation in the gastrointestinal tract.

Explore the dataset using functions like glimpse, tabyl, head, etc. to prepare for the following questions:

1. How to extract the FCP value to a new (numeric) column?
2. How to identify/filter for abnormal FCP values?
3. How to store the definition of normal in a column?
4. How to find and store the FCP units into a column?

To get started, do the exercise below to glimpse the variables in FCP. Fill in the dataframe object fcp for glimpse, then try head(), and tabyl() on some of the variables.

glimpse()

glimpse(fcp)
head(fcp)
tabyl(fcp$flag)

Dig into the lab_result_text

Take a look at the first entry for lab_result_text with the code chunk below By adding brackets and 1, i.e. fcp$lab_result_text[1]

Then look at a few more with [2:5]

Take a look at several until you are confident that you have a pattern to find that will consistently include the numerical result for calprotectin.

fcp$lab_result_text[17]

fcp$lab_result_text[1]

fcp$lab_result_text[1:5]

Find the result in lab_result_text

Use str_view to find “Calprotectin, F” and, separately, “mcg/g” in a few rows of lab_result_text, by editing the code chunk below

str_view(string = fcp$lab_result_text, pattern = "55901")

str_view(string = fcp$lab_result_text, pattern = "Calprotectin, F")

str_view(string = fcp$lab_result_text, pattern = "mcg/g")

Extracting the result from lab_result_text

You have now bracketed the numeric result.

Build a pattern using RVerbalExpressions

RVE

that includes

1. starts with “Calprotectin, F” (seek prefix)
2. Result in the middle (anything)
3. Ends with “mcg/g” (seek_suffix)

Set this as your pattern, and see if you are selecting the numeric result with str_view by editing the code chunk below.

Make sure your prefix and suffix are unique and flank the result value

p <- "Test Performed by:"


str_view(string = fcp$lab_result_text, pattern = p)

p <- rx() %>% 
  rx_seek_prefix("Calprotectin, F") %>% 
  rx_anything() %>% 
  rx_seek_suffix("mcg/g")

str_view(string = fcp$lab_result_text, pattern = p)

Extracting a number from lab_result_text

You have a pattern to get your result. You probably have to peel off the letter H from some of these results.

Try this out first with str_extract using the same pattern, then pipe it into parse_number() in the code chunk below

p <- rx() %>% 
  rx_seek_prefix("Calprotectin, F") %>% 
  rx_anything() %>% 
  rx_seek_suffix("mcg/g ")

str_view(string = fcp$lab_result_text, pattern = p)

p <- rx() %>% 
  rx_seek_prefix("Calprotectin, F") %>% 
  rx_anything() %>% 
  rx_seek_suffix("mcg/g ")

str_extract(string = fcp$lab_result_text, 
            pattern = p)

Create the new fcp_value variable

Once you have this working, edit the code chunk below - start with fcp, pipe it into a mutate statement, which should create a new variable called fcp_value, and set it equal to the str_extract/parse_number pipe result.

p <- rx() %>% 
  rx_seek_prefix("Calprotectin, F") %>% 
  rx_anything() %>% 
  rx_seek_suffix("mcg/g ")

fcp %>% 
  mutate(fcp_value = var) %>% 
  select(sbj_id, eff_date, flag, fcp_value)

p <- rx() %>% 
  rx_seek_prefix("Calprotectin, F") %>% 
  rx_anything() %>% 
  rx_seek_suffix("mcg/g ")

fcp %>% 
  mutate(fcp_value = parse_number(str_extract(string = fcp$lab_result_text, pattern = p))) %>% 
  select(sbj_id, eff_date, fcp_value)

More Challenges

2. Identify/filter for abnormal FCP values?
3. Store the definition of normal in a column?
4. Find and store the FCP units into a column?

Exploring the rad dataset

Start with glimpse() and head() to get a feel for this dataset

glimpse(df)

glimpse(rad)
head(rad)

We want to find out which patients had abscesses on their scans.

Exploring the text blobs

The report text is in rad_text. Take a look at several of these entries using rad$rad_text.

You can select which row with [row] Edit the code chunk below to review the rad_text from report 1, then reports 2 through 5.

rad

rad$rad_text[1]
rad$rad_text[2:5]

Cleaning up newlines ()

Notice that there are a lot of newline characters (“”) which create line breaks.

Use str_replace_all to replace all instances of “” with a space " ".

Edit the code chunk below to mutate and create a new rad_text_sp variable, with all of the newline characters replaced by spaces. When you are convinced that it works, use the left assignment arrow to assign the str_replace pipe to rad$rad_text to overwrite the original version.

rad$rad_text

rad$rad_text %>% 
  str_replace("EXAMINATION:", "exam:")

rad$rad_text

rad$rad_text %>% 
  str_replace_all("\n", " ")

rad$rad_text <- rad$rad_text %>% 
  str_replace_all("\n", " ")

Finding abscesses

Since this is a large dataset of 2781 imaging reports, we will start with detecting the word “abscess”.

Use str_detect on rad$rad_text to find out how in many rows the word “abscess” occurs. Pipe this into the sum() function to get the sum

rad$rad_text %>% 
  str_detect("stricture") 

rad$rad_text %>% 
  str_detect("abscess") %>% 
  sum()

Wow. 11156 out of 2781 seems like a lot.

Taking a Look with str_view()

Let’s see what is going on with str_view(). Edit the code block below to see what is going on.

rad$rad_text %>% 
  str_view("fistula") 

rad$rad_text %>% 
  str_view("abscess") 

You can see that we have a negation problem. Just because a word appears in a text blob, it does not mean that the patient has that finding. It can easily be “No abscess”.

This is a good time to use RVE to detect the words before and after “abscess” to help parse its meaning.

Extracting Abscesses

Let’s build a pattern in RVerbalExpressions to detect the entire sentence surrounding the word “abscess”. Or possibly a certain number of words before & after.

This is probably a good time to peek at the RVE website.

RVE

pattern = "abscess"

rad$rad_text %>% 
  str_extract(pattern) 

pattern = rx() %>% 
  rx_find(". ") %>% 
  rx_anything() %>% 
  rx_find("abscess") %>% 
  rx_anything() %>% 
  rx_find(". ")
  
rad$rad_text %>% 
  str_extract_all(pattern) 

pattern = rx() %>% 
  rx_word() %>% 
  rx_space() %>% 
  rx_word() %>% 
  rx_space() %>% 
  rx_word() %>% 
  rx_space() %>% 
  rx_find("abscess") %>% 
  rx_space() %>% 
  rx_word() %>% 
  rx_space() %>% 
  rx_word() 
  
rad$rad_text %>% 
  str_extract_all(pattern) 

Encoding results

Do you have enough of a pattern to mutate a new column variable named abscess_present and to create 1s and 0s for this variable?

Or would you rather extract or str_view the cases that have the word abscess (can filter for these with str_detect), and hand code based on human interpretation?

This is the kind of situation where NLP can benefit from human assistance, kind of like laparoscopy vs. hand-assisted laparoscopy. You can do most of the work with robots, but a little help from humans goes a long way.

Edit the code block below to

1. create a variable in rads named abscess_present, and assign 0 to each row as the default
2. filter the rad dataset with filter(str_select) to filter for only those rows with the word abscess.
3. Review the results, decide which ones truly have abscesses.
4. Hand assign these values to 1, as in rad$abscess_present[patid == 73] <- 1

Challenges - On Your Own

1. Can you extract CT, MR, and XR codes from the variable code, and determine what percentage of these exams are CT vs MR vs X-rays?

rad$code 

rad$code %>% str_detect("^MR") %>% sum()
rad$code %>% str_detect("^CT") %>% sum()
rad$code %>% str_detect("^XR") %>% sum()
length(rad$code)

rad %>% 
  mutate(mr = str_detect(code, "^MR")) %>% 
  mutate(ct = str_detect(code, "^CT")) %>%
  mutate(xr = str_detect(code, "^XR")) %>% 
  pivot_longer(cols = c(mr, ct, xr), names_to = "exam") %>% 
  filter(value == TRUE) %>% 
  select(-value)

2. Can you determine whether more MRs or CTs find abscesses (by percentage of positive exams?)

3. Which doctors (ordering_doc_id) order the highest percentage of MRIs? Which order the highest percentage of CT scans? Which group is more likely to be pediatricians?