Non-standard evaluation

Exercises

1. plot() uses deparse(substitute(x)) to generate labels for the x and y axes. Can you generate input that causes it to display bad labels? Write your own wrapper around plot() that uses expr_label() to compute xlim and ylim.

2. Create a simple implementation of mean() that stops with an informative error message if the argument is not numeric:

   x <- c("a", "b", "c")
   my_mean(x)
   #> Error: `x` is a not a numeric vector.
   my_mean(x == "a")
   #> Error: `x == "a"` is not a numeric vector.
   my_mean("a")
   #> Error: "a" is not a numeric vector.

3. Read the source code for expr_text(). How does it work? What additional arguments to deparse() does it use?

Definition of a formula

Technically, a formula is a “language” object (i.e. an unevaluated expression) with a class of “formula” and an attribute that stores the environment:

f <- ~ x + y + z
typeof(f)
#> [1] "language"
attributes(f)
#> $class
#> [1] "formula"
#> 
#> $.Environment
#> <environment: R_GlobalEnv>

The structure of the underlying object is slightly different depending on whether you have a one-sided or two-sided formula:

• One-sided formulas have length two:

  length(f)
  #> [1] 2
  # The 1st element is always ~
  f[[1]]
  #> `~`
  # The 2nd element is the RHS
  f[[2]]
  #> x + y + z

• Two-sided formulas have length three:

  g <- y ~ x + z
  length(g)
  #> [1] 3
  # The 1st element is still ~
  g[[1]]
  #> `~`
  # But now the 2nd element is the LHS
  g[[2]]
  #> y
  # And the 3rd element is the RHS
  g[[3]]
  #> x + z

To abstract away these differences, lazyeval provides f_rhs() and f_lhs() to access either side of the formula, and f_env() to access its environment:

f_rhs(f)
#> x + y + z
f_lhs(f)
#> NULL
f_env(f)
#> <environment: R_GlobalEnv>

f_rhs(g)
#> x + z
f_lhs(g)
#> y
f_env(g)
#> <environment: R_GlobalEnv>

Evaluating a formula

A formula captures delays the evaluation of an expression so you can later evaluate it with f_eval():

f <- ~ 1 + 2 + 3
f
#> ~1 + 2 + 3
f_eval(f)
#> [1] 6

This allows you to use a formula as a robust way of delaying evaluation, cleanly separating the creation of the formula from its evaluation. Because formulas capture the code and context, you get the correct result even when a formula is created and evaluated in different places. In the following example, note that the value of x inside add_1000() is used:

x <- 1
add_1000 <- function(x) {
  ~ 1000 + x
}

add_1000(3)
#> ~1000 + x
#> <environment: 0x561651a5fee8>
f_eval(add_1000(3))
#> [1] 1003

It can be hard to see what’s going on when looking at a formula because important values are stored in the environment, which is largely opaque. You can use f_unwrap() to replace names with their corresponding values:

f_unwrap(add_1000(3))
#> ~1000 + 3

Non-standard scoping

f_eval() has an optional second argument: a named list (or data frame) that overrides values found in the formula’s environment.

y <- 100
f_eval(~ y)
#> [1] 100
f_eval(~ y, data = list(y = 10))
#> [1] 10

# Can mix variables in environment and data argument
f_eval(~ x + y, data = list(x = 10))
#> [1] 110
# Can even supply functions
f_eval(~ f(y), data = list(f = function(x) x * 3))
#> [1] 300

This makes it very easy to implement non-standard scoping:

f_eval(~ mean(cyl), data = mtcars)
#> [1] 6.1875

One challenge with non-standard scoping is that we’ve introduced some ambiguity. For example, in the code below does x come from mydata or the environment?

f_eval(~ x, data = mydata)

You can’t tell without knowing whether or not mydata has a variable called x. To overcome this problem, f_eval() provides two pronouns:

• .data is bound to the data frame.
• .env is bound to the formula environment.

They both start with . to minimise the chances of clashing with existing variables.

With these pronouns we can rewrite the previous formula to remove the ambiguity:

mydata <- data.frame(x = 100, y = 1)
x <- 10

f_eval(~ .env$x, data = mydata)
#> [1] 10
f_eval(~ .data$x, data = mydata)
#> [1] 100

If the variable or object doesn’t exist, you’ll get an informative error:

f_eval(~ .env$z, data = mydata)
#> Error: Object 'z' not found in environment
f_eval(~ .data$z, data = mydata)
#> Error: Variable 'z' not found in data

Unquoting

f_eval() has one more useful trick up its sleeve: unquoting. Unquoting allows you to write functions where the user supplies part of the formula. For example, the following function allows you to compute the mean of any column (or any function of a column):

df_mean <- function(df, variable) {
  f_eval(~ mean(uq(variable)), data = df)
}

df_mean(mtcars, ~ cyl)
#> [1] 6.1875
df_mean(mtcars, ~ disp * 0.01638)
#> [1] 3.779224
df_mean(mtcars, ~ sqrt(mpg))
#> [1] 4.43477

To see how this works, we can use f_interp() which f_eval() calls internally (you shouldn’t call it in your own code, but it’s useful for debugging). The key is uq(): uq() evaluates its first (and only) argument and inserts the value into the formula:

variable <- ~cyl
f_interp(~ mean(uq(variable)))
#> ~mean(cyl)

variable <- ~ disp * 0.01638
f_interp(~ mean(uq(variable)))
#> ~mean(disp * 0.01638)

Unquoting allows you to create code “templates”, where you write most of the expression, while still allowing the user to control important components. You can even use uq() to change the function being called:

f <- ~ mean
f_interp(~ uq(f)(uq(variable)))
#> ~mean(disp * 0.01638)

Note that uq() only takes the RHS of a formula, which makes it difficult to insert literal formulas into a call:

formula <- y ~ x
f_interp(~ lm(uq(formula), data = df))
#> ~lm(x, data = df)

You can instead use uqf() which uses the whole formula, not just the RHS:

f_interp(~ lm(uqf(formula), data = df))
#> ~lm(y ~ x, data = df)

Unquoting is powerful, but it only allows you to modify a single argument: it doesn’t allow you to add an arbitrary number of arguments. To do that, you’ll need “unquote-splice”, or uqs(). The first (and only) argument to uqs() should be a list of arguments to be spliced into the call:

variable <- ~ x
extra_args <- list(na.rm = TRUE, trim = 0.9)
f_interp(~ mean(uq(variable), uqs(extra_args)))
#> ~mean(x, na.rm = TRUE, trim = 0.9)

Exercises

1. Create a wrapper around lm() that allows the user to supply the response and predictors as two separate formulas.

2. Compare and contrast f_eval() with with().

3. Why does this code work even though f is defined in two places? (And one of them is not a function).

   f <- function(x) x + 1
   f_eval(~ f(10), list(f = "a"))
   #> [1] 11

Programming with sieve()

Imagine that you’ve written some code that looks like this:

sieve(march, ~ x > 100)
sieve(april, ~ x > 50)
sieve(june, ~ x > 45)
sieve(july, ~ x > 17)

(This is a contrived example, but it illustrates all of the important issues you’ll need to consider when writing more useful functions.)

Instead of continuing to copy-and-paste your code, you decide to wrap up the common behaviour in a function:

threshold_x <- function(df, threshold) {
  sieve(df, ~ x > threshold)
}
threshold_x(df, 3)
#>   x y
#> 4 4 2
#> 5 5 1

There are two ways that this function might fail:

1. The data frame might not have a variable called x. This will fail unless there’s a variable called x hanging around in the global environment:

   rm(x)
   df2 <- data.frame(y = 5:1)
   
   # Throws an error
   threshold_x(df2, 3)
   #> Error in eval(expr, data, expr_env): object 'x' not found
   
   # Silently gives the incorrect result!
   x <- 5
   threshold_x(df2, 3)
   #>   y
   #> 1 5
   #> 2 4
   #> 3 3
   #> 4 2
   #> 5 1

2. The data frame might have a variable called threshold:

   df3 <- data.frame(x = 1:5, y = 5:1, threshold = 4)
   threshold_x(df3, 3)
   #>   x y threshold
   #> 5 5 1         4

These failures are partiuclarly pernicious because instead of throwing an error they silently produce the wrong answer. Both failures arise because f_eval() introduces ambiguity by looking in two places for each name: the supplied data and formula environment.

To make threshold_x() more reliable, we need to be more explicit by using the .data and .env pronouns:

threshold_x <- function(df, threshold) {
  sieve(df, ~ .data$x > .env$threshold)
}

threshold_x(df2, 3)
#> Error: Variable 'x' not found in data
threshold_x(df3, 3)
#>   x y threshold
#> 4 4 2         4
#> 5 5 1         4

Here .env is bound to the environment where ~ is evaluated, namely the inside of threshold_x().

Adding arguments

The threshold_x() function is not very useful because it’s bound to a specific variable. It would be more powerful if we could vary both the threshold and the variable it applies to. We can do that by taking an additional argument to specify which variable to use.

One simple approach is to use a string and [[:

threshold <- function(df, variable, threshold) {
  stopifnot(is.character(variable), length(variable) == 1)
  
  sieve(df, ~ .data[[.env$variable]] > .env$threshold)
}
threshold(df, "x", 4)
#>   x y
#> 5 5 1

This is a simple and robust solution, but only allows us to use an existing variable, not an arbitrary expression like sqrt(x).

A more general solution is to allow the user to supply a formula, and use unquoting:

threshold <- function(df, variable = ~x, threshold = 0) {
  sieve(df, ~ uq(variable) > .env$threshold)
}

threshold(df, ~ x, 4)
#>   x y
#> 5 5 1
threshold(df, ~ abs(x - y), 2)
#>   x y
#> 1 1 5
#> 5 5 1

In this case, it’s the responsibility of the user to ensure the variable is specified unambiguously. f_eval() is designed so that .data and .env work even when evaluated inside of uq():

x <- 3
threshold(df, ~ .data$x - .env$x, 0)
#>   x y
#> 4 4 2
#> 5 5 1

Dot-dot-dot

There is one more tool that you might find useful for functions that take .... For example, the code below implements a function similar to dplyr::mutate() or base::transform().

mogrify <- function(`_df`, ...) {
  args <- list(...)
  
  for (nm in names(args)) {
    `_df`[[nm]] <- f_eval(args[[nm]], `_df`)
  }
  
  `_df`
}

(NB: the first argument is a non-syntactic name (i.e. it requires quoting with `) so it doesn’t accidentally match one of the names of the new variables.)

transmogrifty() makes it easy to add new variables to a data frame:

df <- data.frame(x = 1:5, y = sample(5))
mogrify(df, z = ~ x + y, z2 = ~ z * 2)
#>   x y z z2
#> 1 1 4 5 10
#> 2 2 1 3  6
#> 3 3 2 5 10
#> 4 4 5 9 18
#> 5 5 3 8 16

One problem with this implementation is that it’s hard to specify the names of the generated variables. Imagine you want a function where the name and expression are in separate variables. This is awkward because the variable name is supplied as an argument name to mogrify():

add_variable <- function(df, name, expr) {
  do.call("mogrify", c(list(df), setNames(list(expr), name)))
}
add_variable(df, "z", ~ x + y)
#>   x y z
#> 1 1 4 5
#> 2 2 1 3
#> 3 3 2 5
#> 4 4 5 9
#> 5 5 3 8

Lazyeval provides the f_list() function to make writing this sort of function a little easier. It takes a list of formulas and evaluates the LHS of each formula (if present) to rename the elements:

f_list("x" ~ y, z = ~z)
#> $x
#> ~y
#> 
#> $z
#> ~z

If we tweak mogrify() to use f_list() instead of list():

mogrify <- function(`_df`, ...) {
  args <- f_list(...)
  
  for (nm in names(args)) {
    `_df`[[nm]] <- f_eval(args[[nm]], `_df`)
  }
  
  `_df`
}

add_new() becomes much simpler:

add_variable <- function(df, name, expr) {
  mogrify(df, name ~ uq(expr))
}
add_variable(df, "z", ~ x + y)
#>   x y z
#> 1 1 4 5
#> 2 2 1 3
#> 3 3 2 5
#> 4 4 5 9
#> 5 5 3 8

Exercises

1. Write a function that selects all rows of df where variable is greater than its mean. Make the function more general by allowing the user to specify a function to use instead of mean() (e.g. median()).

2. Create a version of mogrify() where the first argument is x? What happens if you try to create a new variable called x?

Dot-dot-dot

If you want a ... function that doesn’t require formulas, I recommend that the SE version take a list of arguments, and the NSE version uses dots_capture() to capture multiple arguments as a list of formulas.

mogrify_ <- function(`_df`, args) {
  args <- as_f_list(args)
  
  for (nm in names(args)) {
    `_df`[[nm]] <- f_eval(args[[nm]], `_df`)
  }
  
  `_df`
}

mogrify <- function(`_df`, ...) {
  mogrify_(`_df`, dots_capture(...))
}

Exercises

1. Recreate subscramble() using base::subset() instead of sieve(). Why does it fail?