You can enhance the performance of an LLM by providing context — structured or unstructured data, such as documents, articles, or knowledge bases — tailored to the domain or information you want the model to specialize in. Using techniques like prompt engineering and context injection, you can build an intelligent chatbot capable of navigating extensive datasets, retrieving relevant information, and delivering responses.

Whether it's storing recipes, code documentation, research articles, or answering domain-specific queries, an LLM-based chatbot can adapt to your needs with customization and privacy. You can deploy it locally to create a highly specialized conversational assistant that respects your data.

In this article, you will learn how to build a local Retrieval-Augmented Generation (RAG) application using Ollama and ChromaDB in R. By the end, you'll have a custom conversational assistant with a Shiny interface that efficiently retrieves information while maintaining privacy and customization.

Table of Contents

• What is RAG?

• Project Overview

• Project Setup

• Ollama Installation

• Data Collection and Cleaning

• How to Create Chunks

• How to Generate Sentence Embeddings

• How to Set Up the Vector Database for Embedding Storage

• How to Write the User Input Query Embedding Function

• Tool Calling

• How to Initialize the Chat System, Design Prompts, and Integrate Tools

• How to Interact with Your Chatbot Using a Shiny App

• Complete Code

• Conclusion

What is RAG?

Retrieval-Augmented Generation (RAG) is a method that integrates retrieval systems with generative AI, enabling chatbots to access recent and specific information from external sources.

By using a retrieval pipeline, the chatbot can fetch up-to-date, relevant data and combine it with the generative model’s language capabilities, producing responses that are both accurate and contextually enriched. This makes RAG particularly useful for applications requiring fact-based, real-time knowledge delivery.

Project Overview

Project Setup

Prerequisites

Before you begin, ensure you have installed the latest version of the items listed here:

1. RStudio: The IDE – RStudio is the primary workspace where you'll write and test your R code. Its user-friendly interface, debugging tools, and integrated environment make it ideal for data analysis and chatbot development.

2. R: The Programming Language – R is the backbone of your project. You'll use it to handle data manipulation, apply statistical models, and integrate your recipe chatbot components seamlessly.

3. Python – Some libraries, like the embedding library you'll use for text vectorization, are built on Python. It’s vital to have Python installed to enable these functionalities alongside your R code.

4. Java – Java serves as a foundational element for certain embedding libraries. It ensures efficient processing and compatibility for text embedding tasks required to train your chatbot.

5. Docker Desktop – Docker Desktop allows you to run ChromaDB, the vector database, locally on your machine. This enables fast and reliable storage of embeddings, ensuring your chatbot retrieves relevant information quickly.

6. Ollama – Ollama brings powerful Large Language Models (LLMs) directly to your local computer, removing the need for cloud resources. It lets you access multiple models, customize outputs, and integrate them into your chatbot effortlessly.

Ollama Installation

Ollama is an open-sourced tool you can use to run and manage LLMs on your computer. Once installed, you can access various LLMs as per your needs. You will be using llama3.2:3b-instruct-q4_K_M model to build this chatbot.

A quantized model is a version of a machine learning model that has been optimized to use less memory and computational power by reducing the precision of the numbers it uses. This enables you to use an LLM locally, especially when you don’t have access to a GPU (Graphics Processing Unit – a specialized processor that perform complex computations).

To start, you can download and install the Ollama software here.

Then you can confirm installation by running this command:

ollama --version

Run the following command to start Ollama:

ollama serve

Next, run the following command to pull the Q4_K_M quantization of llama3.2:3b-instruct:

ollama pull llama3.2:3b-instruct-q4_K_M

Then confirm that the model was extracted with this:

ollama list

If the model extraction was successful, a list containing the model’s name, ID, and size will be returned, like so:

Now you can chat with the model:

ollama run llama3.2:3b-instruct-q4_K_M

If successful, you should receive a prompt that you can test by asking a question and getting an answer. For example:

Then you can exit the console by typing /bye or ctrl + D

Data Collection and Cleaning

The chatbot you are building will be a cooking assistant that suggests recipes given your available ingredients, what you want to eat, and how much food a recipe yields.

You first have to get the data to train the model. You will be using a dataset that contains recipes from Kaggle.

To start, load the necessary libraries:

# loading required libraries
library(xml2) #read, parse, and manipulate XML,HTML documents
library(jsonlite) #manipulate JSON objects

library(RKaggle) # download datasets from Kaggle 
library(dplyr)   # data manipulation

Then download and save recipe dataset:

# Download and read the "recipe" dataset from Kaggle
recipes_list <- RKaggle::get_dataset("thedevastator/better-recipes-for-a-better-life")

Inspect the dataframe and extract the first element like this:

# inspect the dataset
class(recipes_list)
str(recipes_list)
head(recipes_list)
# extract the first tibble
recipes_df <- recipes_list[[1]]

A quick inspection of the recipes_list object shows that it contains two objects of type tibble. You will be using only the first element for this project. A tibble is a type of data structure used for storing and manipulating data. It’s similar to a traditional dataframe, but it’s designed to enforce stricter rules and perform fewer automatic actions compared to traditional dataframes.

We’ll use a regular dataframe in this project because more people are likely familiar with it. It can also efficiently handle row indexing, which is crucial for accessing and manipulating specific rows in our recipe dataset.

In the code block below, you’ll convert the tibble to a dataframe and then drop the first column, which is the index column. Then you’ll inspect the newly converted dataframe and drop unnecessary columns.

Unnecessary columns are best removed to streamline the dataset and focus on relevant features. In this project, we’ll drop certain columns that aren’t particularly useful for training the chatbot. This ensures that the model concentrates on meaningful data to improve its accuracy and functionality.

# convert to dataframe and drop the first column
recipes_df <- as.data.frame(recipes_df[, -1])
# inspect the converted dataframe
head(recipes_df)
class(recipes_df)
colnames(recipes_df)
# drop unnecessary columns
cleaned_recipes_df <- subset(recipes_df, select = -c(yield,rating,url,cuisine_path,nutrition,timing,img_src))

Now you need to identify rows with NA (missing) values, which you can do like this:

# Identify rows and columns with NA values
which(is.na(cleaned_recipes_df), arr.ind = TRUE)

# a quick inspection reveals columns [2:4] have missing values
subset_column_names <- colnames(cleaned_recipes_df)[2:4]
subset_column_names

It is important to handle NA values to ensure that your data is complete, to prevent errors, and to preserve context.

Now, replace the NA values and confirm that there are no missing values:

# Replace NA values dynamically based on conditions
cols_to_modify <- c("prep_time", "cook_time", "total_time")
cleaned_recipes_df[cols_to_modify] <- lapply(
  cleaned_recipes_df[cols_to_modify],
  function(x, df) {
    # Replace NA in prep_time and cook_time where both are NA
    replace(x, is.na(df$prep_time) & is.na(df$cook_time), "unknown")
  },
  df = cleaned_recipes_df  # Pass the whole dataframe for conditions
)
cleaned_recipes_df <- cleaned_recipes_df %>%
  mutate(
    prep_time = case_when(
      # If cooktime is present but preptime is NA, replace with "no preparation required"
      !is.na(cook_time) & is.na(prep_time) ~ "no preparation required",
      # Otherwise, retain original value
      TRUE ~ as.character(prep_time)
    ),
    cook_time = case_when(
      # If prep_time is present but cook_time is NA, replace with "no cooking required"
      !is.na(prep_time) & is.na(cook_time) ~ "no cooking required",
      # Otherwise, retain original value
      TRUE ~ as.character(cook_time)
    )
  )
# confirm there are no missing values
any(is.na(cleaned_recipes_df))
)

# confirm the replacing NA logic works by inspecting specific rows
cleaned_recipes_df[1081,]
cleaned_recipes_df[1,]
cleaned_recipes_df[405,]

For this tutorial, we’ll subset the dataframe to the first 250 rows for demo purposes. This saves on time when it comes to generating embeddings.

# recommended for demo/learning purposes
cleaned_recipes_df <- head(cleaned_recipes_df,250)

How to Create Chunks

To understand why chunking is important before embedding, you need to understand what an embedding is.

An embedding is a vectoral representation of a word or a sentence. Machines don’t understand human text – they understand numbers. LLMs work by transforming human text to numerical representations in order to give answers. The process of generating embeddings requires a lot of computation, and breaking down the data to be embedded optimizes the embedding process.

So now we’re going to split the dataframe into smaller chunks of a specified size to enable efficient batch processing and iteration.

# Define the size of each chunk (number of rows per chunk)
chunk_size <- 1

# Get the total number of rows in the dataframe
n <- nrow(cleaned_recipes_df)

# Create a vector of group numbers for chunking
# Each group number repeats for 'chunk_size' rows
# Ensure the vector matches the total number of rows
r <- rep(1:ceiling(n/chunk_size), each = chunk_size)[1:n]

# Split the dataframe into smaller chunks (subsets) based on the group numbers
chunks <- split(cleaned_recipes_df, r)

How to Generate Sentence Embeddings

As previously mentioned, embeddings are vector representations of words or sentences. Embeddings can be generated from both words and sentences. How you choose to generate embeddings depends on your intended application of the LLM.

Word embeddings are numerical representations of individual words in a continuous vector space. They capture semantic relationships between words, allowing similar words to have vectors close to each other.

Word embeddings can be used in search engines as they support word-level queries by matching embeddings to retrieve relevant documents. They can also be used in text classification to classify documents, emails, or tweets based on word-level features (for example, detecting spam emails or sentiment analysis).

Sentence embeddings are numerical representations of entire sentences in a vector space, designed to capture the overall meaning and context of the sentence. They are used in settings where sentences provide better context like question answering systems where user queries are matched to relevant sentences or documents for more precise retrieval.

For our recipe chatbot, sentence embedding is the best choice.

First, create an empty dataframe that has three columns.

#empty dataframe
recipe_sentence_embeddings <-  data.frame(
  recipe = character(),
  recipe_vec_embeddings = I(list()),
  recipe_id = character()
)

The first column will hold the actual recipe in text form, the recipe_vec_embeddings column will hold the generated sentence embeddings, and the recipe_id holds a unique id for each recipe. This will help in indexing and retrieval from the vector database.

Next, it’s helpful to define a progress bar, which you can do like this:

# create a progress bar
pb <- txtProgressBar(min = 1, max = length(chunks), style = 3)

Embedding can take a while, so it’s important to keep track of the progress of the process.

Now it’s time to generate embeddings and populate the dataframe.

Write a for loop that executes the code block as long as the length of the chunks.

for (i in 1:length(chunks)) {}

The recipe field is the text at the chunk that is currently being executed and the unique chunk id is generated by pasting the index of the chunk and the text “chunk”.

for (i in 1:length(chunks)) {
    recipe <- as.character(chunks[i])
    recipe_id <- paste0("recipe",i)
}

The text embed function from the text library generates either sentence or word embeddings. It takes in a character variable or a dataframe and produces a tibble of embeddings. You can read loading instructions here for smooth running of the text library.

The batch_size defines how many rows are embedded at a time from the input. Setting the keep_token_embeddings discards the embeddings for individual tokens after processing, and aggregation_from_layers_to_tokens “concatenates” or combines embeddings from specified layers to create detailed embeddings for each token. A token is the smallest unit of text that a model can process.

for (i in 1:length(chunks)) {
    recipe <- as.character(chunks[i])
    recipe_id <- paste0("recipe",i)
    recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )
}

In order to specify sentence embeddings, you need to set the argument to the aggregation_from_tokens_to_texts parameter as "mean".

aggregation_from_tokens_to_texts = "mean"

The "mean" operation averages the embeddings of all tokens in a sentence to generate a single vector that represents the entire sentence. This sentence-level embedding captures the overall meaning and semantics of the text, regardless of its token length.

# convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

The embedding function returns a tibble object. In order to obtain a vector embedding, you need to first unlist the tibble and drop the row names and then list the result to form a simple vector.

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

Finally, update the empty dataframe after each iteration with the newly generated data.

  # track embedding progress
  setTxtProgressBar(pb, i)

In order to keep track of the embedding progress, you can use the earlier defined progress bar inside the loop. It will update at the end of every iteration.

Complete Code Block:

# load required library
library(text)
# # ensure to read loading instructions here for smooth running of the 'text' library
# # https://www.r-text.org/
# embedding data
for (i in 1:length(chunks)) {
  recipe <- as.character(chunks[i])
  recipe_id <- paste0("recipe",i)
  recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )

  # convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

  # track embedding progress
  setTxtProgressBar(pb, i)

}

How to Set Up the Vector Database for Embedding Storage

A vector database is a special type of database that stores embeddings and allows you to query and retrieve relevant information. There are numerous vector databases available, but for this project, you will use ChromaDB, an open-source option that integrates with the R environment through the rchroma library.

ChromaDB runs locally in a Docker container. Just make sure you have Docker installed and running on your device.

Then load the rchroma library and run your ChromaDB instance:

# load rchroma library
library(rchroma)
# run ChromaDB instance.
chroma_docker_run()

If it was successful, you should see this in the console:

Next, connect to a local ChromaDB instance and check the connection:

# Connect to a local ChromaDB instance
client <- chroma_connect()

# Check the connection
heartbeat(client)
version(client)

Now you’ll need to create a collection and confirm that it was created. Collections in ChromaDB function similarly to tables in conventional databases.

# Create a new collection
create_collection(client, "recipes_collection")

# List all collections
list_collections(client)

Now, add embeddings to the collection. To add embeddings to the recipes_collection, use the add_documents function.

# Add documents to the collection
add_documents(
  client,
  "recipes_collection",
  documents = recipe_sentence_embeddings$recipe,
  ids = recipe_sentence_embeddings$recipe_id,
  embeddings = recipe_sentence_embeddings$recipe_vec_embeddings
)

The add_documents() function is used to add recipe data to the recipes_collection. Here's a breakdown of its arguments and how the corresponding data is accessed:

1. documents: This argument represents the recipe text. It is sourced from the recipe column of the recipe_sentence_embeddings dataframe.

2. ids: This is the unique identifier for each recipe. It is extracted from the recipe_id column of the same dataframe.

3. embeddings: This contains the sentence embeddings, which were previously generated for each recipe. These embeddings are accessed from the recipe_vec_embeddings column of the dataframe.

All three arguments—documents, ids, and embeddings—are obtained by subsetting their respective columns from the recipe_sentence_embeddings dataframe.

How to Write the User Input Query Embedding Function

In order to retrieve information from a vector database, you must first embed your query text. The database compares your query's embedding with its stored embeddings to find and retrieve the most relevant document.

It's important to ensure that the dimensions (rows × columns) of your query embedding match those of the database embeddings. This alignment is achieved by using the same embedding model to generate your query.

Matching embeddings involves calculating the similarity (for example, cosine similarity) between the query and stored embeddings, identifying the closest match for effective retrieval.

Let’s write a function that allows us to embed a query which then queries similar documents using the generated embeddings. Wrapping it in a function makes it reusable.

  #sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents using embeddings
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }

This chunk of code is similar to how we have previously used the text_embed() function. The query() function is added to enable querying the vector database, particularly the recipes' collection, and returns the top two documents that closely match a user’s query.

Our function thus takes in a sentence as an argument and embeds the sentence to generate sentence embeddings. It then queries the database and returns two documents that match the query most.

Tool Calling

To interact with Ollama in R, you will utilize the ellmer library. This library streamlines the use of large language models (LLMs) by offering an interface that enables seamless access to and interaction with a variety of LLM providers.

To enhance the LLM’s usage, we need to provide context to it. You can do this by tool calling. Tool calling allows an LLM to access external resources in order to enhance its functionality.

For this project, we are implementing Retrieval-Augmented Generation (RAG), which combines retrieving relevant information from a vector database and generating responses using an LLM. This approach improves the chatbot's ability to provide accurate and contextually relevant answers.

Now, define a function that links to the LLM to provide context using the tool() function from the ellmer library.

# load ellmer library
library(ellmer)

# function that links to llm to provide context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

The tool() function takes the question function that returns the relevant documents that we’ll use as context as the first argument. We’ll use the documents to help the LLM answer questions accordingly.

The text, "obtains the right context for a given question", is a description of what the tool will be doing.

Finally, the sentence = type_string() defines what type of object the question() function expects.

How to Initialize the Chat System, Design Prompts, and Integrate Tools

Next, you’ll set up a conversational AI system by defining its role and functionality. Using system prompt design, you will shape the assistant’s behavior, tone, and focus as a culinary assistant. You’ll also integrate external tools to extend the chatbot’s capabilities by registering tools. Let’s dive in.

First, you need to initialize a Chat Object:

#  Initialize the chat system with propmpt instructions.
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")

You can do that using the chat_ollama() function. This sets up a conversational agent with the specified system prompt and model.

The system prompt defines the conversational behavior, tone, and focus of the LLM while the model argument specifies the language model (llama3.2:3b-instruct-q4_K_M) that the chat system will use to generate responses.

Next, you need to register a tool.

 #register tool
  chat$register_tool(tool_context)

We need to tell our chat object about our tool_context() function. Do this by registering a tool using the register_tool() function.

How to Interact with Your Chatbot Using a Shiny App

To interact with the chatbot you’ve just created, we’ll use Shiny, a framework for building interactive web applications in R. Shiny provides a user-friendly graphical interface that allows seamless interaction with the chatbot.

For this purpose, we’ll use the shinychat library, which simplifies the process of building a chat interface within a Shiny app. This involves defining two key components:

1. User Interface (UI):

   • Responsible for the visual layout and what the user sees.

   • In this case, chat_ui("chat") is used to create the interactive chat interface.

2. Server Function:

   • Handles the functionality and logic of the application.

   • It connects the chatbot to external tools and manages processes like embedding queries, retrieving relevant responses, and handling user inputs.

# load the required library
library(shinychat)

# wrap the chat code in a Shiny App
ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  # Connect to a local ChromaDB instance running on docker with embeddings loaded
  client <- chroma_connect()

  #sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents using embeddings
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }


  # function that provides context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

  #  Initialize the chat system with the first chunk
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")
  #register tool
  chat$register_tool(tool_context)

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

Alright, let’s understand how this is working:

1. User input monitoring with observeEvent(): The observeEvent() block monitors user inputs from the chat interface (input$chat_user_input). When a user sends a message, the chatbot processes it, retrieves relevant context using the embeddings, and streams the response dynamically to the chat interface.

2. Tool calling for context: The chatbot employs tool calling to interact with external resources (like the vector database) and enhance its functionality. In this project, Retrieval-Augmented Generation (RAG) ensures the chatbot provides accurate and context-rich responses by integrating retrieval and generation seamlessly.

This approach brings the chatbot to life, enabling users to interact with it dynamically through a responsive Shiny app.

Complete Code

The R scripts have been split in two, with data.R containing code that handles data gathering and cleaning, text chunking, sentence embeddings generation, creating a vector database, and loading documents to it.

The chat.R script contains code that handles user input querying, context retrieval, chat initialization, system prompt design, tool integration, and a chat Shiny app.

data.R

# install and load required packages
# install devtools from CRAN
install.packages('devtools')
devtools::install_github("benyamindsmith/RKaggle")

library(text)
library(rchroma)
library(RKaggle)
library(dplyr)

# run ChromaDB instance.
chroma_docker_run()

# Connect to a local ChromaDB instance
client <- chroma_connect()

# Check the connection
heartbeat(client)
version(client)


# Create a new collection
create_collection(client, "recipes_collection")

# List all collections
list_collections(client)

# Download and read the "recipe" dataset from Kaggle
recipes_list <- RKaggle::get_dataset("thedevastator/better-recipes-for-a-better-life")

# extract the first tibble
recipes_df <- recipes_list[[1]]

# convert to dataframe and drop the first column
recipes_df <- as.data.frame(recipes_df[, -1])

# drop unnecessary columns
cleaned_recipes_df <- subset(recipes_df, select = -c(yield,rating,url,cuisine_path,nutrition,timing,img_src))

## Replace NA values dynamically based on conditions
# Replace NA when all columns have NA values
cols_to_modify <- c("prep_time", "cook_time", "total_time")
cleaned_recipes_df[cols_to_modify] <- lapply(
  cleaned_recipes_df[cols_to_modify],
  function(x, df) {
    # Replace NA in prep_time and cook_time where both are NA
    replace(x, is.na(df$prep_time) & is.na(df$cook_time), "unknown")
  },
  df = cleaned_recipes_df  
)

# Replace NA when either or columns have NA values
cleaned_recipes_df <- cleaned_recipes_df %>%
  mutate(
    prep_time = case_when(
      # If cook_time is present but prep_time is NA, replace with "no preparation required"
      !is.na(cook_time) & is.na(prep_time) ~ "no preparation required",
      # Otherwise, retain original value
      TRUE ~ as.character(prep_time)
    ),
    cook_time = case_when(
      # If prep_time is present but cook_time is NA, replace with "no cooking required"
      !is.na(prep_time) & is.na(cook_time) ~ "no cooking required",
      # Otherwise, retain original value
      TRUE ~ as.character(cook_time)
    )
  )

# chunk the dataset
chunk_size <- 1
n <- nrow(cleaned_recipes_df)
r <- rep(1:ceiling(n/chunk_size),each = chunk_size)[1:n]
chunks <- split(cleaned_recipes_df,r)

#empty dataframe
recipe_sentence_embeddings <-  data.frame(
  recipe = character(),
  recipe_vec_embeddings = I(list()),
  recipe_id = character()
)

# create a progress bar
pb <- txtProgressBar(min = 1, max = length(chunks), style = 3)

# embedding data
for (i in 1:length(chunks)) {
  recipe <- as.character(chunks[i])
  recipe_id <- paste0("recipe",i)
  recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )

  # convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

  # track embedding progress
  setTxtProgressBar(pb, i)

}

# Add documents to the collection
add_documents(
  client,
  "recipes_collection",
  documents = recipe_sentence_embeddings$recipe,
  ids = recipe_sentence_embeddings$recipe_id,
  embeddings = recipe_sentence_embeddings$recipe_vec_embeddings
)

chat.R

# Load required packages
library(ellmer)
library(text)
library(rchroma)
library(shinychat)

ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  # Connect to a local ChromaDB instance running on docker with embeddings loaded 
  client <- chroma_connect()

  # sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }


  # function that provides context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

  #  Initialize the chat system 
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")
  #register tool
  chat$register_tool(tool_context)

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

You can find the complete code here.

Conclusion

Building a local Retrieval-Augmented Generation (RAG) application using Ollama and ChromaDB in R programming offers a powerful way to create a specialized conversational assistant.

By leveraging the capabilities of large language models and vector databases, you can efficiently manage and retrieve relevant information from extensive datasets.

This approach not only enhances the performance of language models but also ensures customization and privacy by running the application locally.

Whether you're developing a cooking assistant or any other domain-specific chatbot, this method provides a robust framework for delivering intelligent and contextually aware responses.

In this article, you'll explore how to use popular R libraries for web scraping to extract data from a website. The target website displays different books across multiple pages, requiring navigation between them. You'll learn how to use RVest for data extraction and RSelenium to automate button clicks.

There are a couple of housekeeping rules when it comes to harvesting data on the internet:

• Inspect the robots.txt file: Check the robots.txt file of a website to understand what data you are allowed to extract. You can find this file by appending “/robots.txt” to the website's home URL.

• Review terms and conditions: Before scraping, read the website's terms and conditions to understand the legal expectations regarding data extraction.

• Limit requests: Avoid overloading the server with requests by implementing rate limiting. The polite library in R can help manage request rates effectively.

Let’s dive in!

Table of Contents

• Project Overview

• Project Setup

• How to Understand and Inspect a Webpage

• How to Extract Data Using RVest

• How to Mimic Human Behaviour Using RSelenium

• How to Combine RSelenium & RVest and Save to CSV

• Bringing it All Together

• Conclusion

Project Overview

Here’s what we’re going to be building:

This approach to web scraping allows you to see the browser in action as it navigates and extracts data from the website. Unlike headless browsing, where everything runs in the background without a visible interface, this method provides a graphical UI, making it easier to monitor and debug the process.

To practice your data mining skills, you will be scraping data from a website built specifically for that: Books To Scrape. You are going to be using a driver to drive a browser which will then open your target website. It’ll navigate from the first page, mimicking human behaviour (clicking the next button) while collecting data about the books, right to the last page.

Project Setup

Prerequisites:

To follow along with this tutorial, you will need:

• R programming knowledge

• HTML knowledge

• R Studio installed

Note that I’m building this tutorial on a Windows machine.

Setup and Install Chrome Driver

First, you’ll want to check to make sure you have Java installed on your computer by running this terminal command:

java -version

If it’s not present, download and install Java here.

Next, install the Chrome browser if you don’t already have it. Once it’s installed, check for your browser version in the settings section.

Then you can download the Browser Driver that corresponds to your Browser Version here. Check where other browser drivers are stored on your device by running this in RStudio terminal:

# install and load wdman and binman packages
install.packages("wdman")
library(wdman)

install.packages("binman")
library(binman)

# check drivers already installed
binman::list_versions(appname = "chromedriver")

# check browser driver locations
wdman::selenium(retcommand = TRUE, check = FALSE)

Extract the driver “.exe“ and store it at the specified folder location. This is usually the following location:

"C:\Users\YourName\AppData\Local\binman\binman_chromedriver\win32\version\chromedriver.exe"

Now, add the drivers to your system path by specifying the folder path excluding the application. Confirm installation by running the following terminal command.

# Chromedriver SYSTEMS PATH: "C:\Users\YourName\AppData\Local\binman\binman_chromedriver\win32\version\"
# check chromedriver installation
chromedriver -version

How to Understand and Inspect a Webpage

A webpage is a visual representation of an HTML document that is available on the internet and accessed through a web browser. The components of a webpage, called elements, are structured hierarchically in a HTML DOM (Document Object Model) tree. Each element can be located using specific paths called selectors or locators, which you can read more about here.

Developer Tools are a set of tools available in your browser. They’re helpful for inspecting and analyzing a webpage’s structure. The feature “Inspect“ helps examine the structure and styling of a specific element. You can access this feature by selecting the element you would like to inspect, right clicking on it, and clicking “Inspect”.

How to Extract Data Using RVest

RVest is an R package that contains a set of functions that enables you to extract data from HTML and XML web pages

We are interested in extracting the following information about books from every page on the website’s catalogue:

• Book Title

• Book Rating

• Book Price

• Individual Book Link

• Cover Image Link

Let’s go through the steps for using RVest to extract this data.

Step 1: Load the webpage

To load the first page of your target website and parse the HTML document using the RVest package in R, follow these steps:

1. Install and load the RVest package: If you haven't already installed the RVest package, you can do so by running the following command in R:

    install.packages("rvest")
   

   Then, load the package:

    library(rvest)
   

2. Load the webpage and parse the HTML: Use the read_html() function from the RVest package to fetch and parse the HTML content of the webpage. Here's an example of how to do this:

    # Specify the URL of the target website
    url <- "https://books.toscrape.com/"
   
    # Fetch and parse the HTML content
    webpage <- read_html(url)
   

This code will download the HTML content of the specified webpage and convert it into an XML document, making it easier to structure and organize the data for further processing or storage.

Step 2: Identify the target elements

The target elements are the HTML elements that contain the specific data you intend to extract.

A quick inspection of the webpage using developer tools shows that the each book’s information is contained in an article tag and forms part of an ordered list. It’s important to specify the <ol> tag in the path, as there are other lists in the tree.

The pipe %>% operator facilitates chaining operations, making it easier to extract elements step by step. html_element() returns the first matching element while html_elements() returns all the elements that match the defined path.

# define the path from which other details will be extracted
book <- books %>% html_element("ol")  %>% html_elements("li") %>% html_element("article")

# extracting details using css locators.
# title
title <- book %>% 
  html_element("h3 a") %>% 
  html_attr("title")

# rating
rating <- book %>% 
  html_element("p") %>% 
  html_attr("class")

# price
price <- book %>% 
  html_element(".product_price p") %>% 
  html_text2()

#link to book page
book_link <- book %>% 
  html_element("h3 a") %>% 
  html_attr("href")

# cover page image link
cover_page_link <- book %>% 
  html_element(".image_container a img") %>% 
  html_attr("src")

# inspect right format by selecting the first element of each detail
title[[1]]
rating[[1]]
price[[1]]
book_link[[1]]
cover_page_link[[1]]

Step 3: Clean the “rating” data

To clean the "star-rating" data, you can use the stringr package in R to remove the unnecessary text and trim any whitespace. Here's how you can do it:

library(stringr)

# Example of extracted rating data
rating_data <- "star-rating Three"

# Remove "star-rating " and trim whitespace
cleaned_rating <- str_trim(str_replace(rating_data, "star-rating ", ""))

# Output the cleaned rating
cleaned_rating

This code will output "Three", effectively removing the "star-rating" prefix and any leading or trailing whitespace.

How to Mimic Human Behaviour Using RSelenium

How Selenium Works

Selenium is a tool that allows you to simulate user actions on a website, usually for testing purposes. RSelenium is an R library that allows you to access this functionality.

We need a script, a browser, and browser driver to mimic user behaviour. The code you write that contains the instructions detailing the actions you would like to automate is the script. The browser driver acts as a bridge between your script and the browser and performs your desired actions by translating the script into actions.

The script, when run, is the client which requests and receives info from the browser driver’s server.

When you run a script, the script is converted to JSON format data which is then transferred to the browser driver via the JSON Wire Protocol. A protocol is simply a set of rules that define how data should be managed and handle during transfer across devices.

The driver receives and validates the received data. If successful, it communicates the actions defined in the script to the browser. If it’s unsuccessful, an error is sent to the client.

On browser initialization, the driver performs the actions step by step. This carries on to completion or until an error is encountered (missing elements, server errors, and so on). The bidirectional communication between the driver and browser is via HTTP. Finally, the results are sent back to the client and the browser is shut down.

Automating Page Navigation and Data Collection with RSelenium

# install and load RSelenium
install.packages("RSelenium")
library(RSelenium)

# initialize and run the chrome driver
rD <- rsDriver(browser = "chrome", port = 4567L)

# extract and assign the client
remDr <- rD[["client"]]

Running rsDriver() starts a Selenium server that launches ChromeDriver. Extract and assign the rD[["client"]] to a variable. This variable allows you to control and interact with the browser.

Sometimes, starting the driver may fail due to reasons such as permission restrictions, missing dependencies, or incorrect setup. If that happens, you can manually launch ChromeDriver by adding the following block of code right after loading the libraries at the top of the script. It is important to ensure the port numbers match.

cDrv <- chrome(verbose = FALSE, check = FALSE, port = 4567L)
cDrv$process

Now, navigate to the target webpage:

# naivigate to the target site
remDr$navigate("https://books.toscrape.com/")

#maximize Chrome Window Size
remDr$maxWindowSize()

And scroll to the bottom of the page:

# scroll to the bottom of the page
webElem <- remDr$findElement("css", "body")
webElem$sendKeysToElement(list(key = "end"))

The above code locates the body element and simulates pressing the down key to the end of the page.

Now, click Next to navigate to the next page:

# locate next button and click next
nextPage <-  remDr$findElement(using = "css selector",
                               value = ".next > a")
nextPage$clickElement()

Find the element that contains the link to the next page and click on it to redirect you.

Now we’re going to write a while loop that navigates through all the pages, up to page 50, and then closes the browser once it’s done.

A while loop executes a piece of code as long as a specific condition is met. Once the condition is not met, the loop exits.

while(condition is TRUE){
    #DO SOMETHING
}

Write a loop that ensures the next page button is clicked as long as the element containing the link to the next page is visible in the HTML DOM.

First, locate the next button element. Its presence in the open webpage makes sure that the loop runs.

The last page does not have a next button, so the loop will exit when it reaches that page (and Selenium will throw an error due to the missing element).

nextPage <- remDr$findElement(using = "css selector", value = ".next > a")

Wrap the nextPage element search in a tryCatch() block. This prevents the script from crashing if the 'Next' button is missing. If an error occurs, tryCatch() returns NULL, signaling that there are no more pages to navigate.

An if block then checks for a NULL value. If encountered, a message is displayed to inform the client that no 'Next' button was found, and the break statement exits the loop.

Finally, close the browser once the driver navigates to the last page (page 50 in the catalogue) to free up system resources using remDr$close().

while (TRUE) {  
  # Try to find and click "Next" button
  nextPage <- tryCatch({
    remDr$findElement(using = "css selector", value = ".next > a")
  }, error = function(e) {
    return(NULL)  # No more pages
  })

  if (is.null(nextPage)) {
    message("No 'Next' button found. Exiting loop.")
    break
  }

  nextPage$clickElement()
  Sys.sleep(3)  # Allow next page to load

}
print("finished scraping")
remDr$close()

How to Combine RSelenium & RVest and Save to CSV

Now that we’ve extracted data from specific HTML elements using RVest and automated user actions using RSelenium, let’s combine the two to scrape data from all the pages in the website.

Create a scrape books function

You will be saving the scraped books information in a CSV file. First, create an empty dataframe to hold the scraped data:

# install and load dplyr for dataframe manipulation
install.packages("dplyr")
library(dplyr)

# create a dataframe to hold book information
Books <-  data.frame()

Retrieve and parse the webpage

For Rvest to work with RSelenium, you have to retrieve the HTML source of the currently loaded webpage within the Selenium-controlled browser using remDr$getPageSource()[[1]] to extract the HMTL content.

page <- remDr$getPageSource()[[1]]

Convert the HTML content to XML using read_html() like this:

 # define the path from which other details will be extracted
    books <- read_html(page)  %>% html_element("ol")  %>% html_elements("li") %>% html_element("article")

Extract each book’s details using CSS selectors with rvest functions. The scraped objects returned are XML objects and lists. They need to be formatted to character strings, preventing unexpected data type issues when working with the data. Do this by piping as.character() at the very end of each extracted detail.

    # title
    title <- book %>% 
      html_element("h3 a") %>% 
      html_attr("title") %>% 
      as.character()

Wrap the block of code used to extract details from HTML elements in a function and return a dataframe whose column values are the book details. This makes the code reusable and modular.

scrape_books <- function() {
    page <- remDr$getPageSource()[[1]]

    # define the path from which other details will be extracted
    books <- read_html(page)  %>% html_element("ol")  %>% html_elements("li") %>% html_element("article")

    # extracting details using css locators.
    # title
    title <- book %>% 
      html_element("h3 a") %>% 
      html_attr("title") %>% 
      as.character() 

    # rating
    rating <- book %>% 
      html_element("p") %>% 
      html_attr("class") %>% 
      as.character() 

    cleaned_rating <- str_trim(gsub("star-rating", "", rating))

    # price
    price <- book %>% 
      html_element(".product_price p") %>% 
      html_text2() %>% 
      as.character() 

    #link to book page
    book_link <- book %>% 
      html_element("h3 a") %>% 
      html_attr("href") %>% 
      as.character() 

    # image link
    cover_page_link <- book %>% 
      html_element(".image_container a img") %>% 
      html_attr("src") %>% 
      as.character() 

    return(data.frame(title,cleaned_rating,price,book_link,cover_page_link, stringsAsFactors = FALSE))
}

Write to CSV

Save the dataframe to a CSV file saved as “books.csv“:

write.csv(Books, file = "./books.csv", fileEncoding = "UTF-8")

Bringing it All Together

Let’s review what we’ve done so far: First, the script to scrape book data begins by loading the browser, maximizing the window size, and navigating to the Books To Scrape Page.

Then we created an empty dataframe to hold the scraped data. We then scraped the data from the first page, saved it to the dataframe, and located the ‘Next‘ button in order to navigate to the next page – from which we scraped data and stored it.

The process of scraping, adding to the dataframe, and clicking the next page button continues until the ‘Next’ button is no longer available in the HTML DOM.

Once the last page has been reached, the code exits the loop and saves the data to CSV. Finally, it closes the driver to free up system resources.

# load libraries
library(wdman)
library(binman)
library(rvest)
library(stringr)
library(RSelenium)
library(dplyr)


cDrv <- chrome(verbose = FALSE, check = FALSE, port = 4450L)
cDrv$process

rD <- rsDriver(browser = "chrome", port = 4450L)
remDr <- rD[["client"]]


remDr$navigate("https://books.toscrape.com/")
remDr$maxWindowSize()

page <- remDr$getPageSource()[[1]]
webElem <- remDr$findElement("css", "body")
webElem$sendKeysToElement(list(key = "end"))

nextPage <-  remDr$findElement(using = "css selector",
                               value = ".next > a")
nextPage$clickElement()


# converting the lists containg the scraped data into a dataframe 
Books <-  data.frame(title = character(), rating = character(), stringsAsFactors = FALSE)

scrape_books <- function() {
    page <- remDr$getPageSource()[[1]]

    # define the path from which other details will be extracted
    books <- read_html(page)  %>% html_element("ol")  %>% html_elements("li") %>% html_element("article")

    # extracting details using css locators.
    # title
    title <- book %>% 
      html_element("h3 a") %>% 
      html_attr("title") %>% 
      as.character() 

    # rating
    rating <- book %>% 
      html_element("p") %>% 
      html_attr("class") %>% 
      as.character() 

    cleaned_rating <- str_trim(gsub("star-rating", "", rating))

    # price
    price <- book %>% 
      html_element(".product_price p") %>% 
      html_text2() %>% 
      as.character() 

    #link to book page
    book_link <- book %>% 
      html_element("h3 a") %>% 
      html_attr("href") %>% 
      as.character() 

    # image link
    cover_page_link <- book %>% 
      html_element(".image_container a img") %>% 
      html_attr("src") %>% 
      as.character() 

    return(data.frame(title,cleaned_rating,price,book_link,cover_page_link, stringsAsFactors = FALSE))
}

# scrape first page
Books <- rbind(Books, scrape_books())

while (TRUE) {
  # scrape current page
  Books <- rbind(Books, scrape_books())

  # find and click "next" button
  nextPage <- tryCatch({
    remDr$findElement(using = "css selector", value = ".next > a")
  }, error = function(e) {
    return(NULL)  # No more pages
  })

  # exit loop if "next" button is missing
  if (is.null(nextPage)) {
    message("No 'Next' button found. Exiting loop.")
    break
  }

  nextPage$clickElement()
  # Allow next page to load
  Sys.sleep(3)  

}

write.csv(Books, file = "./books.csv", fileEncoding = "UTF-8")
print("finished scraping")
remDr$close()

Conclusion

In this tutorial, you learned how to effectively combine RSelenium and RVest to scrape data from a website. By leveraging RSelenium, you can automate user interactions and navigate through web pages, while RVest allows you to extract specific data from HTML elements.

This approach provides a powerful and flexible method for web scraping, enabling you to handle dynamic content and mimic human behavior. By following the steps outlined here, you can successfully scrape data from multiple pages and save it to a CSV file for further analysis.

When you think of R, you probably imagine someone wearing chunky thick prescription glasses and devouring a book. You know, a statistician dealing with complex models, an insane amount of mathematical equations, and copious amounts of data.

But R is far more than just a tool for statistics. It shines when you need to turn raw data into actionable insights and present those insights in a clear, engaging way.

With frameworks like Shiny, R takes this one step further, enabling you to create fully interactive web apps without having to worry about frontends, backends, or learning an entirely new programming language.

In this tutorial, you will create a simple weather app that fetches data from an API and displays the results in a good-looking app.

Table of Contents

1. Project Overview

2. Project Setup

3. API Keys: Storage and Retrieval

4. How to Make Your First API Call

5. How to Build the Shiny App

6. Conclusion

Project Overview

Here’s what we’re going to be building:

For the weather app to work, you will need to make two separate API calls. We’ll use the One Call API 3.0 to update weather data and the OpenWeather API for geocoding. You can get your API Key here. Just keep in mind that if this is your first time signing up for an API key, activation may take up to 24 hours.

The weather app will take the location/city from user input. The input will then be geocoded by making the call to OpenWeather API. Then, from its response, the coordinates (latitude and longitude) will be extracted. The coordinates will be used as query arguments for the One Call API call to obtain the weather data in JSON format.

Prerequisites:

To follow along with this tutorial, you will need:

• R programming knowledge

• HTML and a bit of JavaScript knowledge

• R Studio installed

Project Setup

Create a folder in your desired directory. Set and confirm the project folder as the working directory using the following command in the R console:

setwd("path/to/your/project/file")
getwd()

Create a project in the set path using the following command:

#create R project
usethis::create_project(path = ".", open = FALSE)

You should have a folder structure that looks like this.

Create an R file in the root directory and save it as app.R. All your R code will be contained here.

Install and load the following libraries that you are going to work with:

library(shiny)
library(bslib)
library(shinyjs)
library(httr2)
library(lubridate)
library(shiny.semantic)

API Keys: Storage and Retrieval

Storing your credentials in a location separate from your scripts and global environment is a good practice. This ensures security, scalability, and flexibility, especially when working in shared or production environments. The .Renviron file best serves that purpose.

Open and edit your .Renviron file in the following way:

#open and edit .Renviron
usethis::edit_r_environ(scope=c("project")

The scope argument set to project sets up the .Renviron specifically to your project. In the newly opened file, add your API key as follows:

OPENWEATHERAPIKEY="yourapikey"

How to Make Your First API Call

You will be using the httr2 library (built based on httr) to obtain data from the API. It grants you more control over how you make requests to the web.

Make the API Key accessible in the script

First, you’ll need to securely access and store the API key in the script without hardcoding it. You can do that like this:

#access API keys in script
readenviron(".Renviron")
api_key = Sys.getenv("OPENWEATHERAPIKEY")

Define the Geocoding Function

You will create a function that takes a location and an API key as inputs, sends a request to the OpenWeather geocoding API, and returns the coordinates of the specified location.

Start by creating a request. The pipe (|>) operator facilitates the chaining of HTTP requests step by step in a clear and readable manner. The geocoding URL takes two parameters: location, denoted by q, and the API key, denoted by app_id. The req_url_query() function appends these parameters to the query.

Chain the query to perform the request and fetch action, and finally obtain the response in JSON format using the second to last line.

# Geocoding URL
geocoding_url <- "https://api.openweathermap.org/data/2.5/weather"
geocode <- function(location, api_key) {
  request(geocoding_url) |> 
    req_url_query(`q` = location, `appid` = api_key) |> 
    req_perform() |> 
    resp_body_json() |>
    coordinates()
}

Define the coordinate-extracting function

The coordinates() function is a helper function that extracts the latitude and longitude values from the JSON response. A quick inspection of the JSON response reveals the coordinate's position. The JSON object is simply a long list of lists and you can access elements by subsetting it.

A blank data body would imply that the city/location is unavailable, and you’d get the message "No such city exists!". If the JSON contains an element, the length would be more than 0 – it is a list after all.

coordinates <- function(body) {
  if(length(body) != 0) { 
    lat <- body$coord$lat
    lng <- body$coord$lon
    town <- body$name
    c(lat, lng, town)
  } else {
    "No such city exists!"
  }
}

Define the weather-update function

You will create a function that sends a request to the OpenWeather API with specified query parameters, handles errors using a predefined function, and returns the parsed JSON response containing the weather data.

As implemented in the geocoding function, start by creating a request and adding the necessary query parameters using the req_url_query() function. The openweather_json() function accepts two main arguments:

• api_key: This is a required argument used for authentication with the OpenWeather API matched by position.

• ...: This represents optional keyword arguments that you can use to customize the query. You can pass as many additional parameters as needed, provided they are specified as named arguments.

openweather_json <- function(api_key, ...) { 
  request(current_weather_url) |> 
    req_url_query(..., `appid` = api_key, `units` = "metric") |> 
    req_error(body = openweather_error_body) |>
    req_perform() |> 
    resp_body_json()
}

Error Handling: Extracting and Managing Status Codes

You will create an error-handling function that extracts non-200 status codes from a response and defines how to manage them. The structure of this function depends on how the API reports errors and where the relevant information is stored.

Define the weather-update error body

The req_error() in openweather_json() introduces a new concept: error handling. API requests may throw exceptions, and getting the status codes helps you know what message to show the user and how to resolve it.

Create an error body which is a function that captures the error code if the status code is not 200 (which means everything is OK).

The function takes a response and extracts the status response stored in the JSON response at the $message sublist. The underscore (_)is a placeholder for the JSON object.

openweather_error_body <- function(resp) {
  resp |> resp_body_json() |> _$message 
}

Define the geocode error body

This error body function will prove useful in the Shiny App. This is a simple walkthrough.

The req_error() function allows you to customize how response errors are handled. Its is_error argument determines whether a given response should be considered an error. By setting is_error to \(resp) FALSE (an anonymous function that always returns FALSE), all responses, regardless of the status code, are treated as successful. This prevents the app from exiting due to non-200 status codes.

With this setup, you can extract the status code from the response body and pipe it into the resp_status() function to retrieve the exact code.

openstreetmap_error_body <- function(location, api_key) {
  resp <- request(geocoding_url) |> 
    req_url_query(`q` = location, `appid` = api_key) |> 
    req_error(is_error = \(resp) FALSE) |>
    req_perform() |>  resp_status()
  resp
}

How to Build the Shiny App

Now that you have nailed down how to obtain data from the API, it’s time to render the results in an interpretable and interactive format. For this, you will use Shiny. Shiny is a framework that allows you to create interactive web apps.

A Shiny App is made up of two components:

• The UI: what the user interacts with. It defines the layout and appearance of the app.

• The server: contains the app’s logic and behaviour.

Building the Shiny UI

Shiny UI provides a collection of elements that allow users to input data, make selections, and trigger events seamlessly.

You will include a textInput element that takes in the location and the weather data will be fetched and rendered upon submission. The input_task_button button prevents the user from clicking when an API call is in progress. The other elements are output elements where the weather data will be displayed and a mode-switching button.

Styling the Shiny app

You can use shiny.semantic, a library built on top of Fomantic-UI, to style your Shiny dashboard. Fomantic-UI is a front-end framework that provides a rich collection of pre-styled HTML components like buttons, modals, form inputs, and more. It simplifies UI design by allowing developers to create visually appealing and responsive interfaces without needing extensive custom CSS or HTML knowledge.

Fomantic-UI styling is applied by wrapping elements in their corresponding classes, which define their behavior and appearance.

A grid in Fomantic-UI is a flexible layout system used to organize content. It acts as a canvas that divides the layout into rows (horizontally aligned) and columns (vertically aligned). A root grid can contain up to 16 columns, making it ideal for creating structured and responsive designs.

To specify a column's width, you append classes like wide and the size (a number from 1 to 16) to represent its span. The total width of all columns in a row should sum up to 16.

A segment groups related content, while a card displays detailed, content-rich items, such as a user's social media profile. Dividers are visual elements used to separate sections or content within a layout.

For the weather app, first create a div of class grid within which you’ll nest the various elements.

Search bar section

Divide the grid into sixteen columns and create a segment that groups elements in the search bar section. Add a theme toggle button, location input that takes in user input, a search button for submitting the location to the API, and a notification button, defining their width by the column size.

div(class = "sixteen wide column",
          div(class = "ui segment",
              div(class = "ui grid",
                  div(class = "two wide column",
                      button(
                        class = "ui button icon basic",
                        input_id = "darkmode",
                        label = NULL,
                        icon = icon("moon icon")
                      )
                  ),
                  div(class = "ten wide column",
                      textInput(
                        "location",
                        label = NULL,
                        placeholder = "Search for your preferred city"
                      )
                  ),
                  div(class = "two wide column",
                      tags$div(
                        class = "ui button",
                        id = "my-custom-button",
                        input_task_button("search", label = "Search", icon = icon("search"))
                      )
                  ),
                  div(class = "two wide column",
                      actionButton("show_alert", label = icon("bell"), class = "bell-no-alert"),
                      textOutput("alert_message")
                  )
              )
          )
      )

Location and current weather section

Divide the grid into sixteen columns and nest another grid within the partitions that will host two columns.

Within the grid, define two columns. The first column is for time, location, and date data, and the second column will hold current weather data.

Then create card elements to hold each weather parameter, its unit of measurement, and the corresponding icon.

div(class = "sixteen wide column",
          div(class = "ui equal-height-grid grid",
              div(class = "left floated center aligned four wide column",
                  div(class = "ui raised equal-height-two-segment segment",
                      style = "flex: 1;",
                      div(class = "column center aligned",
                          div(class = "ui hidden section divider"),
                          span(class = "ui large text", textOutput("city")),
                          div(class = "ui hidden section divider"),
                          span(class = "ui big text", textOutput("currentTime")),
                          div(class = "ui hidden section divider"),
                          span(class = "ui large text", textOutput("currentDate")),
                          div(class = "ui hidden section divider")
                      )
                  )
              ),
              div(class = "right floated center aligned twelve wide column",
                  div(class = "ui raised segment",
                      div(class = "ui horizontal equal width segments",
                          div(class = "ui equal-height-two-segment segment",
                              style = "flex: 3;",
                              div(class = "column",
                                  span(class = "ui big text centered", textOutput("currentTemp")),
                                  textOutput("feelsLike"),
                                  card(
                                    class = "ui mini",
                                    div(class = "content", icon(class = "large sun"),
                                        div(class = "sub header", "Sunrise"),
                                        div(class = "description", textOutput("sunriseTime"))
                                    )
                                  ),
                                  card(
                                    class = "ui mini",
                                    div(class = "content", icon(class = "large moon"),
                                        div(class = "sub header", "Sunset"),
                                        div(class = "description", textOutput("sunsetTime"))
                                    )
                                  )
                              )
                          ),
                          div(class = "ui segment",
                              style = "flex: 3;",
                              div(
                                class = "column center aligned",
                                div(class = "ui hidden divider"),
                                htmlOutput("currentWeatherIcon"),
                                span(class = "ui large text", textOutput("currentWeatherDescription"))
                              )
                          ),
                          div(class = "ui segment",
                              style = "flex: 3;",
                              div(class = "column",
                                  card(
                                    class = "ui tiny",
                                    div(class = "content", icon(class = "big tint"),
                                        div(class = "sub header", "Humidity"),
                                        div(class = "description", textOutput("currentHumidity"))
                                    )
                                  ),
                                  card(
                                    class = "ui tiny",
                                    div(class = "content", icon(class = "big tachometer alternate"),
                                        div(class = "sub header", "Pressure"),
                                        div(class = "description", textOutput("currentPressure"))
                                    )
                                  )
                              )
                          ),
                          div(class = "ui segment",
                              style = "flex: 3;",
                              div(class = "column center aligned",
                                  card(
                                    class = "ui tiny",
                                    div(class = "content", icon(class = "big wind"),
                                        div(class = "sub header", "Wind Speed"),
                                        div(class = "description", textOutput("currentWindSpeed"))
                                    )
                                  ),
                                  card(
                                    class = "ui tiny",
                                    div(class = "content", icon(class = "big umbrella"),
                                        div(class = "sub header", "UV Index"),
                                        div(class = "description", textOutput("currentUV"))
                                    )
                                  )
                              )
                          )
                      )
                  )
              )
          )
      )

Forecast section

This section holds the forecasted data. Divide the grid into sixteen columns and nest another grid within the partitions hosting two columns.

Within the grid, define two columns. The first column holds the 5-Day Forecast data. Separate the elements containing different values using rows. The second column contains Hourly Forecast data. Separate the elements containing different values using columns.

      # Forecast section
      div(class = "sixteen wide column",
          div(class = "ui grid equal-height-grid",
              div(class = "left floated center aligned six wide column",
                  div(class = "ui raised segment special-segment equal-height-segment",
                      h4("5 Days Forecast:"),
                      div(class = "ui three column special-column grid",
                          # Day forecasts
                          div(class = "row",
                              div(class = "five wide column", textOutput("dailyDtOne")),
                              div(class = "three wide column", textOutput("dailyTempOne")),
                              div(class = "three wide column", htmlOutput("dailyIconOne"))
                          ),
                          div(class = "row",
                              div(class = "five wide column", textOutput("dailyDtTwo")),
                              div(class = "three wide column", textOutput("dailyTempTwo")),
                              div(class = "three wide column", htmlOutput("dailyIconTwo"))
                          ),
                          div(class = "row",
                              div(class = "five wide column", textOutput("dailyDtThree")),
                              div(class = "three wide column", textOutput("dailyTempThree")),
                              div(class = "three wide column", htmlOutput("dailyIconThree"))
                          ),
                          div(class = "row",
                              div(class = "five wide column", textOutput("dailyDtFour")),
                              div(class = "three wide column", textOutput("dailyTempFour")),
                              div(class = "three wide column", htmlOutput("dailyIconFour"))
                          ),
                          div(class = "row",
                              div(class = "five wide column", textOutput("dailyDtFive")),
                              div(class = "three wide column", textOutput("dailyTempFive")),
                              div(class = "three wide column", htmlOutput("dailyIconFive"))
                          )
                      )
                  )
              ),
              div(class = "right floated center aligned ten wide column",
                  div(class = "ui raised segment special-segment equal-height-segment",
                      h4("Hourly Forecast:"),
                      div(
                        class = "ui grid",
                        style = "display: flex; flex-direction: row; align-items: center; justify-content: space-around; flex-wrap: wrap; height: 100%;",
                        # Hourly forecasts
                        div(class = "column",
                            textOutput("hourlyDtOne"),
                            htmlOutput("hourlyIconOne"),
                            textOutput("hourlyTempOne")
                        ),
                        div(class = "column",
                            textOutput("hourlyDtTwo"),
                            htmlOutput("hourlyIconTwo"),
                            textOutput("hourlyTempTwo")
                        ),
                        div(class = "column",
                            textOutput("hourlyDtThree"),
                            htmlOutput("hourlyIconThree"),
                            textOutput("hourlyTempThree")
                        ),
                        div(class = "column",
                            textOutput("hourlyDtFour"),
                            htmlOutput("hourlyIconFour"),
                            textOutput("hourlyTempFour")
                        ),
                        div(class = "column",
                            textOutput("hourlyDtFive"),
                            htmlOutput("hourlyIconFive"),
                            textOutput("hourlyTempFive")
                        )
                      )
                  )
              )
          )
      )
  )

Building the Shiny Server

Each element in the UI section has an ID (unique identifier) that is used to manipulate what data/information will be displayed to it.

The render*() set of functions defines the visualization type while the output$* functions subset elements. These two are used to link the visual to the logic. Most elements will have data extracted from the JSON list, except for the weather icons (for which an external link as a source will be referenced).

Reactivity

Reactivity is what makes Shiny apps dynamic—outputs automatically update when their dependencies change.

Two key components of reactivity are reactives and observers. A reactive computes and returns a value based on its dependencies, while an observer monitors reactive values and runs code that causes side effects, like logging or updating a database.

To control reactivity, you can use bindEvent() to delay execution until a specific event occurs or observeEvent() to listen for a user action and trigger a code block. Together, these tools provide flexibility for managing app behavior.

The Server Code

1. location reactive

The location reactive includes an if-else conditional block that defines what message to display depending on the status code. The query variable contains the city/location that will be geocoded to obtain coordinates. The flow is piped to bindEvent(). This ensures the geocoding API call is completed before another call can be made, which reduces unnecessary requests.

location <- reactive({
    query <- input$location
    if(openstreetmap_error_body(query, api_key) == "404"){
      validate("No such city/town exists. Check your spelling!")
    }
    else if(openstreetmap_error_body(query, api_key) == "400"){
      validate("Bad request")
    }
    coords <- geocode(query, api_key)
  }) %>% bindEvent(input$search)

2. weather_data reactive

The weather reactive combines a geocoding API call and a weather update API call using coordinates obtained and extracted from location():

  weather_data <- reactive({
    loc <- location()
    openweather_json(api_key, lat = loc[1], lon = loc[2])
  })

To access the JSON objects returned by the API call, you call the reactive as if it were a function. The specific values to be extracted can then be accessed by subsetting the JSON value.

# subsetting weather data.
  output$city <- renderText({
    location()[3]
  })

  output$currentWeatherDescription <- renderText({
    weather_data()$current$weather[[1]]$description
  })

3. Create a Parse Date function

All the time data in the JSON response, forecasted or current, is provided in UNIX format. To make this information user-friendly, it needs to be converted into a human-readable format. You can do this by creating a function that takes the time data as input and uses functions from the lubridate package to handle the conversion.

First, convert the timestamp element to a datetime object. Format the time item to a 12-hour clock system and a date item to include the day of the week, the date, and the month.

• %I: Displays the hour in a 12-hour clock format (01-12).

• %M: Displays the minutes (00-59).

• %p: Adds the AM/PM indicator.

The paste function concatenates the values. The function returns a vector containing date and time values to be extracted by subsetting.

parse_date <- function(timestamp) {
  datetime <- as_datetime(timestamp) 
  date <- paste(weekdays(datetime), ",", day(datetime), months(datetime))
  time <- format(as.POSIXct(datetime), format = "%I:%M %p")
  c(date, time)
}

4. Add a modal to display error messages

The location reactive provides a way to handle errors. You can incorporate a modal to enhance the user experience by overlaying the page and disabling its content until the user completes a specified action whenever an error occurs.

You’ll add JavaScript to control when and how the modal shows.

Add two modals in the UI section, each featuring an explanation of the error (header) and an outline of the required action (content). The action class includes a button that enables the user to close the modal.

# modals - UI
  div(id = "notFound", class = "ui modal",
      div(class = "header", "Location Not Found"),
      div(class = "content", "No such city/town exists. Check your spelling!"),
      div(class = "actions",
          div(class = "ui button", id = "closeNotFound", "OK"))
  ),
  div(id = "badRequest", class = "ui modal",
      div(class = "header", "Invalid Request"),
      div(class = "content", "Bad request. Please try again with valid details."),
      div(class = "actions",
          div(class = "ui button", id = "closeBadRequest", "OK"))
  )

Slightly adjust the location reactive to incorporate the modal. The commented-out code will be replaced with the JavaScript lines. The runjs function shows the modal depending on the error encountered. req(FALSE) terminates the reactive flow.

# show and hide modals  - Server
location <- reactive({
    query <- input$location
    if(openstreetmap_error_body(query, api_key) == "404"){
      #validate("No such city/town exists. Check your spelling!")
      runjs("$('#notFound').modal('show');")
      req(FALSE)
    }
    else if(openstreetmap_error_body(query, api_key) == "400"){
      #validate("Bad request")
      runjs("$('#badRequest').modal('show');")
      req(FALSE)
    }
    coords <- geocode(query, api_key)
  }) %>% bindEvent(input$search)

# listens for button click on modals to hide modal
observeEvent(input$closeNotFound, {
    runjs("$('#notFound').modal('hide');")
  })

observeEvent(input$closeBadRequest, {
    runjs("$('#badRequest').modal('hide');")
  })

Conclusion

In this tutorial, you have built a weather app using Shiny that retrieves weather data from an API and displays it in an interactive and visually appealing way.

To do this, you used the following libraries:

• httr2 for making API requests and handling responses

• shiny.semantic for styling the app

• lubridate for working with and formatting time data

• shinyjs for integrating JavaScript features into the app

This combination of tools allowed you to create a functional, user-friendly weather app.

You can find the complete code for the project here.

La Fin!