Tutorial

Using an API Token when Accessing NEON Data with neonUtilities

Authors: [Claire K. Lunch]

Last Updated: Dec 31, 2024

NEON data can be downloaded from either the NEON Data Portal or the NEON API. When downloading from the Data Portal, you can create a user account. Read about the benefits of an account on the User Account page. You can also use your account to create a token for using the API. Your token is unique to your account, so don’t share it.

Using a token is optional! You can download data without a token, and without a user account. Using a token when downloading data via the API, including when using the neonUtilities package, links your downloads to your user account, as well as enabling faster download speeds. For more information about token usage and benefits, see the NEON API documentation page.

For now, in addition to faster downloads, using a token helps NEON to track data downloads. Using anonymized user information, we can then calculate data access statistics, such as which data products are downloaded most frequently, which data products are downloaded in groups by the same users, and how many users in total are downloading data. This information helps NEON to evaluate the growth and reach of the observatory, and to advocate for training activities, workshops, and software development.

Tokens can be used whenever you use the NEON API. In this tutorial, we’ll focus on using tokens with the neonUtilities R package and the neonutilities Python package. You can follow the tutorial using your preferred programming language.

Objectives

After completing this activity, you will be able to:

Create a NEON API token
Use your token when downloading data with neonUtilities

Things You’ll Need To Complete This Tutorial

A recent version of R (version 4+) or Python (3.9+) installed on your computer.

Install and Load Packages

R

Install the neonUtilities package. You can skip this step if it’s already installed, but remember to update regularly.

install.packages("neonUtilities")

Load the package.

library(neonUtilities)

Python

Install the neonutilities package. You can skip this step if it’s already installed, but remember to update regularly.

# do this in the command line
pip install neonutilities

Load the package.


import neonutilities as nu
import os

Additional Resources

If you’ve never downloaded NEON data using the neonUtilities package before, we recommend starting with the Download and Explore tutorial before proceeding with this tutorial.

In the next sections, we’ll get an API token from the NEON Data Portal, and then use it in neonUtilities when downloading data.

Get a NEON API Token

The first step is create a NEON user account, if you don’t have one. Follow the instructions on the Data Portal User Accounts page. If you do already have an account, go to the NEON Data Portal, sign in, and go to your My Account profile page.

Once you have an account, you can create an API token for yourself. At the bottom of the My Account page, you should see this bar:

Account page on NEON Data Portal showing Get API Token button.

Click the ‘GET API TOKEN’ button. After a moment, you should see this:

Account page on NEON Data Portal showing API token has been created.

Click on the Copy button to copy your API token to the clipboard:

Account page on NEON Data Portal showing API token with Copy button highlighted

Use API token in neonUtilities

In the next section, we’ll walk through saving your token somewhere secure but accessible to your code. But first let’s try out using the token the easy way, by using it as a simple text string.

NEON API tokens are very long, so it would be annoying to keep pasting the entire string into functions. Assign your token an object name:

R

NEON_TOKEN <- "PASTE YOUR TOKEN HERE"

Now we’ll use the loadByProduct() function to download data. Your API token is entered as the optional token input parameter. For this example, we’ll download Plant foliar traits (DP1.10026.001).

foliar <- loadByProduct(dpID="DP1.10026.001", site="all", 
                        package="expanded", check.size=F,
                        token=NEON_TOKEN)

Python


NEON_TOKEN = "PASTE YOUR TOKEN HERE"

Now we’ll use the load_by_product() function to download data. Your API token is entered as the optional token input parameter. For this example, we’ll download Plant foliar traits (DP1.10026.001).

foliar = nu.load_by_product(dpid="DP1.10026.001", site="all", 
                        package="expanded", check_size=False,
                        token=NEON_TOKEN)

You should now have data saved in the foliar object; the API silently used your token. If you’ve downloaded data without a token before, you may notice this is faster!

This format applies to all neonUtilities functions that involve downloading data or otherwise accessing the API; you can use the token input with all of them. For example, when downloading remote sensing data:

Use token to download AOP data

R

chm <- byTileAOP(dpID="DP3.30015.001", site="WREF", 
                 year=2017, check.size=F,
                 easting=c(571000,578000), 
                 northing=c(5079000,5080000), 
                 savepath=getwd(),
                 token=NEON_TOKEN)

Python

chm = nu.by_tile_aop(dpid="DP3.30015.001", site="WREF", 
                 year=2017, check_size=False,
                 easting=[571000,578000], 
                 northing=[5079000,5080000], 
                 savepath=os.getcwd(),
                 token=NEON_TOKEN)

Token management for open code

Your API token is unique to your account, so don’t share it!

If you’re writing code that will be shared with colleagues or available publicly, such as in a GitHub repository or supplemental materials of a published paper, you can’t include the line of code above where we assigned your token to NEON_TOKEN, since your token is fully visible in the code there. Instead, you’ll need to save your token locally on your computer, and pull it into your code without displaying it. There are a few ways to do this, we’ll show two options here.

Option 1: Save the token in a local file, and source() (R) or import (Python) that file at the start of every script. This is fairly simple but requires a line of code in every script.
Option 2: Set the token as an environment variable and you can access it from any script. This is a little harder to set up initially, but once it’s done, it’s done globally, and it will work in every script you run.

Option 1: Save token in a local file

R

Open a new, empty R script (.R). Put a single line of code in the script:

NEON_TOKEN <- "PASTE YOUR TOKEN HERE"

Save this file in a logical place on your machine, somewhere that won’t be visible publicly. Here, let’s call the file neon_token_source.R, and save it to the working directory. Then, at the start of every script where you’re going to use the NEON API, you would run this line of code:

source(paste0(wd, "/neon_token_source.R"))

Now you can use token=NEON_TOKEN when you run neonUtilities functions, and you can share your code without accidentally sharing your token.

Python

Open a new, empty Python script (.py). Put a single line of code in the script:


NEON_TOKEN = "PASTE YOUR TOKEN HERE"

Save this file in a logical place on your machine, somewhere that won’t be visible publicly. Here, let’s call the file neon_token_source.py, and save it to the working directory. Then, at the start of every script where you’re going to use the NEON API, you would run this line of code:

import neon_token_source

Now you can use token=neon_token_source.NEON_TOKEN when you run neonutilities functions, and you can share your code without accidentally sharing your token.

Option 2: Set token as environment variable

R

To create a persistent environment variable in R, we use a .Renviron file. Before creating a file, check which directory R is using as your home directory:

# For Windows:
Sys.getenv("R_USER")
For Mac/Linux:
Sys.getenv("HOME")

Check the home directory to see if you already have a .Renviron file, using the file browse pane in RStudio, or using another file browse method with hidden files shown. Files that begin with . are hidden by default, but RStudio recognizes files that begin with .R and displays them.

File browse pane in RStudio showing .Renviron file. — Screenshot of file browse pane with .Renviron file.

If you already have a .Renviron file, open it and follow the instructions below to add to it. If you don’t have one, create one using File -> New File -> Text File in the RStudio menus.

Add one line to the text file. In this case, there are no quotes around the token value.

NEON_TOKEN=PASTE YOUR TOKEN HERE

Save the file as .Renviron, in the RStudio home directory identified above. Double check the spelling, this will not work if you have a typo. Re-start R to load the environment.

Once your token is assigned to an environment variable, use the function Sys.getenv() to access it. For example, in loadByProduct():

foliar <- loadByProduct(dpID="DP1.10026.001", site="all", 
                        package="expanded", check.size=F,
                        token=Sys.getenv("NEON_TOKEN"))

Python

To create a persistent environment variable in Python, the simplest option is to use the dotenv module. You will still need to load the variables in each script, but it provides a more flexible way to manage enrionment variables.

pip install python-dotenv

Create a file named .env in the project folder. If you’re using GitHub, make sure .env is in your .gitignore to avoid syncing tokens to GitHub.

To add variables to the .env file:

import dotenv
dotenv.set_key(dotenv_path=".env",
key_to_set="NEON_TOKEN",
value_to_set="YOUR TOKEN HERE")

Use the command dotenv.load_dotenv() to load environment variables to the session, then use os.environ.get() to access particular variables. For example, in load_by_product():

dotenv.load_dotenv()

foliar = nu.load_by_product(dpid="DP1.10026.001", site="all", 
                        package="expanded", check_size=False,
                        token=os.environ.get("NEON_TOKEN"))

If dotenv.load_dotenv() returns False, the variables did not load. Try dotenv.load_dotenv(dotenv.find_dotenv(usecwd=True)).

Get Lesson Code

neon-api-tokens-tutorial.R