The platform contains a ton of datasets and notebooks that you can use to learn and practice your data science and machine learning skills. They even have competitions you can participate in.

Kaggle offers a 100% free platform for all users – but there are some restrictions depending on the resources you're using.

For example, you can use their CPU system for an unlimited amount of time. But there are strict limitations on GPU and TPU usage. You can use their GPU for 30 hours and TPU for 20 hours in a week. It gets resets each week, and then you get a fresh 30 hours GPU usage and 20 hours TPU usage at the start of the new week.

Kaggle Website

Alongside Kaggle, there are another popular platforms for machine learning engineers and data scientists – like Google Colaboratory, or Google Colab for short.

In Google Colab, you can also use their CPU and GPU, but the free versions have more limitations than the free Kaggle account. In Google Colab, you can not get any GPU computational power until they allocate it from their free units. You don't know how many hours you can use, and you don't even know if you have any chance to get units over the next few days.

In order to get all the features, you need to subscribe to their pro plans which are quite expensive.

But sometimes you still may want to use Colab, in most cases for short tasks. In Colab, you can directly connect your Google Drive and use your datasets from there. You can also store your output from the notebook to Google Drive if you want.

When you're working on a project, though, sometimes you'll want to use datasets from Kaggle in Google Colab. So you'll need to download the dataset from Kaggle and upload that to Colab's temporary storage or your Google Drive.

You can probably guess that this is a very time-consuming process.

But there is a way that you can directly download a Kaggle dataset using an API call in the Google Colab's notebook! In this article, I am going to show you how you can do that.

Table of Contents

I've broken this tutorial down into separate parts for better understanding. You can get a clear overview of the entire article here:

• Types of Kaggle datasets
• Prerequisites
• Setup Google Colab for using Kaggle API
• Install Kaggle library
• Mount Google Drive to Colab
• Add the Kaggle API Token to Colab Notebook
• Download Kaggle dataset
• Download Kaggle Competition dataset
• Download Specifc file from Kaggle Competition dataset
• Conclusion

Video

If you would like to watch all of the steps from a video, you're in luck – I made this video just for you:

Types of Kaggle Datasets

Normally Kaggle provides two types of datasets: typical datasets that anyone can upload, and competition datasets. In the competition datasets, the competition organizers typically add/upload the datasets.

Even though you can download a Kaggle dataset easily, you can't download a competition dataset if you don't participate in that competition. But some competitions remain open, and you can access their datasets via "Late Submission". So just make sure to check.

Prerequisites

To go through this tutorial and get the most ouf of it, you'll need a Kaggle account, and that is completely free. Simply head over to the official website of Kaggle, and create an account if you don't have one already.

You'll also need Kaggle's API. Head over to the settings of your Kaggle account. Go to the API section, and click "Create New Token". Keep in mind that Kaggle does not allow you to keep multiple tokens. You can use only one active token for your Kaggle account.

Kaggle API Token

This will give you a kaggle.json file. Keep it safe, as you'll need to use it later.

You also need a Google account if you want to use Google Colab. You may already have one, but if you don't, go ahead and create a new account in Google.

Now, you can store your Kaggle JSON in your Google drive. I prefer to create a new folder and keep my JSON file there so that I can call that in Colab whenever I want.

How to Setup Google Colab to Use the Kaggle API

You can simply open any Colab notebook where you want to use the Kaggle API to download the dataset.

Google Colab

Install the Kaggle library

You need to install the Kaggle Python library before you start working with Kaggle. You can simply install it in the colab notebook using the command ! pip install kaggle.

Install Kaggle library in colab

Mount Google Drive to Colab

Now you need to mount your Google Drive to the Colab notebook, since you've uploaded your kaggle.json file inside your Google drive.

You can simply do that by using the two lines of code given below:

from google.colab import drive
drive.mount('/content/drive')

Make sure to give it permission to access your Google Drive:

Give access to Google Drive

Mount Google Drive

If you refresh the mounted folder icon, you will see your Google Drive and all of the content in the notebook.

Find MyDrive in Notebook

Add the Kaggle API Token to the Colab Notebook

Now you need to add the Kaggle API token to the notebook. But before that, you can simply create a temporary directory for Kaggle at the temporary instance location on the Colab drive by using the command ! mkdir ~/.kaggle.

Now you need to copy your uploaded JSON file to that temporary Kaggle directory. You need the URL where you uploaded your JSON file earlier. You can grab that link directly from the drive folder in the notebook.

Copy JSON file location

You can get the path directly like this.

Then you can use the copy command like below:

! cp kaggle_json_path ~/.kaggle/

For example, my JSON file is located at "/content/drive/MyDrive/Kaggle_API/kaggle.json", so my command would be:

! cp /content/drive/MyDrive/Kaggle_API/kaggle.json ~/.kaggle/

Copy JSON file

Now you need to change the file permissions to read/write to the owner only for safety.

You can use the command below to achive that:

! chmod 600 ~/.kaggle/kaggle.json

Change file permission of kaggle.json file

How to Download the Kaggle Dataset

For downloading a typical Kaggle dataset, you have to find the dataset on Kaggle first.

Let's say I want to download the following dataset from Kaggle:

Sample dataset

Check the complete URL of the dataset, which in this case is:

https://www.kaggle.com/datasets/mdfahimbinamin/fastsurfer-processed-3d-brain-mri-from-adni

We need the "account_name_of_the_dataset_owner/dataset_path" string. From the URL, the account name of the dataset owner is mdfahimbinamin. The dataset path is fastsurfer-processed-3d-brain-mri-from-adni.

So to download this exact dataset from Kaggle to your Google colab, your command would be:

! kaggle datasets download mdfahimbinamin/fastsurfer-processed-3d-brain-mri-from-adni

Downloading the Kaggle dataset to your Colab notebook

The entire process happens on Google's Cloud PC. So the downloading speed should be quite fast.

By default, the datasets come as .zip file. So if you need to unzip that, you can simply use the command below:

! unzip dataset-path.zip

For example, my dataset name/path was "fastsurfer-processed-3d-brain-mri-from-adni". So I will use the following command:

! unzip fastsurfer-processed-3d-brain-mri-from-adni.zip

Unzip Kaggle Dataset

That's it! 😊

How to Download a Kaggle Competition Dataset

Before downloading a Competition dataset, you need to make sure that either you have joined that competition or that you've selected "Late Submission" using the same Kaggle account that you're using for Kaggle API token.

Suppose I'm joining the ConnectX competition on Kaggle.

Connect X competition

I need to click "Join Competition" to get access to their dataset.

But if I want to download a dataset from a past competition, I need to join their "Late Submission" to gain their dataset.

Join a past competition

After clicking on "Late Submission", I need to grab the URL. This time, I'm using the Binary Classification with a Bank Churn Dataset. The complete URL is: https://www.kaggle.com/competitions/playground-series-s4e1/overview

From the URL, I can see that the dataset is located at "playground-series-s4e1". So I will use the following command to download the dataset to my Google Colab notebook:

! kaggle competitions download playground-series-s4e1

Download dataset

That's it! 😊

How to Download a Specific File from a Kaggle Competition Dataset

Let's say, I want to download a specific file from a Kaggle competition dataset. I can also do that.

dataset

In the dataset used above, you can see that there are 3 files. Let's say I want to download the test.csv file only.

To do this, the command would be strucutred like this: ! kaggle competitions download dataset-path -f file_name_with_extension.

So my command would be:

! kaggle competitions download playground-series-s4e1 -f test.csv

Download specific file

That's it! 😊

Conclusion

I hope you have gained some valuable insights from the article.

If you have enjoyed the procedures step-by-step, then don't forget to let me know on Twitter/X or LinkedIn.

You can follow me on GitHub as well if you are interested in open source. Make sure to check my website (https://fahimbinamin.com/) as well!

If you like to watch programming and technology-related videos, then you can check my YouTube channel, too. You can also check my other writings on Dev.to.

All the best for your programming and development journey. 😊

You can do it! Don't give up, never! ❤️

You can write text, write code, run that code, and see the output – all in line in the same notebook.

Benefits of Google Colab

Sharing notebooks is as easy as sharing any Google document. You can also get the app and run the code from your phone.

You can use the powerful and popular Python language in your Google Drive, and the set-up will take less than five minutes.

Because Python runs on a server (and not in your local browser or on your local computer) you can easily use it to interact with an online database and analyze data in situations where you need to keep the code private.

How to Add Colab to Your Google Drive

1. On your computer, in your Google Drive, click the “+ new” button.

2. Click “more”, then click “connect more apps” at the bottom of that new menu.

3. In the Google Workspace Marketplace, type “colab” into the search box.

4. Click to add Google Colaboratory.

5. Now you have Colab in your list of available apps.

How to Use Google Colab

Now when you click the “+new” button and click “more” (at the bottom of that first list) you will see “Google Colaboratory” on the next list.

Click that to open a new Colab notebook. Give your notebook a name, like you would with any Google document or spreadsheet. This notebook is in dark mode:

Type a simple command as a test. To run the code in any cell, you can click the run button on the left side of the code cell (looks like a “play” button with a triangle in a circle) or you can click [shift] + [enter]. The output will appear right below the code cell.

You can import many popular libraries without having to install them first.

If you import a library or define a function in one cell, that will still be available in other cells for at least a few minutes. The runtime will disconnect if you go 30 minutes without running a cell or if you have the notebook open for 12 hours.

Explore Colab

If you were following along and you now have Google Colaboratory installed, you are ready to build your own projects.

If you click on the “< >” symbol on the bottom-left side of the notebook, you will find code snippets that you can use. Google also has many resources for you, some of which are at https://colab.research.google.com/ (just close out of the pop-up window that appears).

At freeCodeCamp, we are building a curriculum to show you how to use Python to solve math problems. You now have the tools at your fingertips. Happy coding!

Google Colab and VS Code are two popular editor tools that many Python developers use. They're great for developing efficient tech solutions or systems especially in the areas of Machine Learning and Data Science.

If you're a Python developer or data scientist, you might already know how to use Google Colab. But did you know that you can set up VS Code on Google Colab and use it as an editor the same way as in your local machine?

In this article, you will learn:

1. How to install the colabcode Python package.
2. How to start VS Code (code server).
3. How to access the online VS Code.
4. How to open the terminal.
5. How to run a Python file.

How to Use Google Colab with VS Code

Open Colab Notebook

The first step is to launch a new colab notebook in your Google Colab. You can rename the file as you want.

For example, run_vscode.ipynb.

Install colabcode Python package.

To use Google Colab with VS Code (code server), you need to install the colabcode Python package. This is an awesome open-source Python package developed by Abhishek Thakur.

To install the package, run the following command in your notebook cell:

 !pip install colabcode

Import ColabCode

The next step is to import the ColabCode class from the package.

from colabcode import ColabCode

Create an instance of ColabCode

After importing ColabCode, you need to create an instance of ColabCode and set the following arguments:

• port – The port you want to run the code-server on. For example port=10000
• password – You can set a password to protect your code-server from unauthorized access. This is an optional argument.
• mount_drive – If you want to use your Google drive. This is a Boolean argument which means you can set it to True or False. This is an optional argument.

ColabCode(port=10000)

Start the Code Server

After running the ColabCode instance, it will start the server and show the link to access the code server.

You need to click the link and it will open in a new tab.

Now you can take advantage of a full-fledged code editor and run different experiments on the Colab VM.

Note: If you check on your Colab Notebook, you will see that the cell that runs the ColabCode instance is continuously running. Don't close your Colab notebook unless you want to close the code server that runs VS Code.

Tips to use VS Code on Google Colab

After launching the code server, use the following tips to help you start using VS Code on Google Colab.

Step 1: Open Terminal

To open the terminal on VS Code that runs on Google Colab, use the following shortcut command:

Ctrl + Shift + `

Step 2: Change the Theme if You Want

You can change the theme of the editor by clicking the setting icon (bottom-left corner) and then click "Color Theme". It will open a popup window with different theme options you can select.

Step 3: Run a Python File

You can create a Python file by clicking the "File" section on the sidebar and then select a "New File" tab.

In the following example, you will see how to run a simple Python file that trains a machine-learning algorithm to classify iris flowers into three species (setosa, versicolor, or virginica) and then make a prediction.

Final Thoughts on Using Google Colab with VS Code

Congratulations 👏👏, you have made it to the end of this article! I hope you have learned something new. You can set up VS Code on Google Colab and take your coding to the next level.

You can also use the colabcode Python package on the Kaggle platform to run VS Code. You just need to follow the same steps mentioned above.

If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post!

You can also find me on Twitter @Davis_McDavid.

And you can read more articles like this here.

Let’s continue our journey to explore the best machine learning frameworks in computer vision.

In the first article we explored object detection with the official Tensorflow APIs. The second article was dedicated to an excellent framework for instance segmentation, Matterport Mask R-CNN based on Keras.

In this article we examine Keras implementation of RetinaNet object detection developed by Fizyr. RetinaNet, as described in Focal Loss for Dense Object Detection, is the state of the art for object detection.
The object to detect with the trained model will be my little goat Rosa.

Object detection with Fizyr

The colab notebook and dataset are available in my Github repo.

In this article, we go through all the steps in a single Google Colab netebook to train a model starting from a custom dataset.

We will keep in mind these principles:

• illustrate how to make the annotation dataset
• describe all the steps in a single Notebook
• use free software, Google Colab and Google Drive, so it’s based exclusively on free cloud resources

At the end of the article you will be surprised by the simplicity of use and the good results we will obtain through this object detection framework.

Despite its ease of use, Fizyr is a great framework, also used by the winner of the Kaggle competition “RSNA Pneumonia Detection Challenge”.

Making the dataset

We start by creating annotations for the training and validation dataset, using the tool LabelImg. This excellent annotation tool let you quickly annotate the bounding boxes of the objects to train the machine learning model.

LabelImg annotation tool

LabelImg creates annotations in PascalVoc format, so we need to convert annotations to Fizyr format:

• create a zip file containing training dataset images and annotations with the same filename (check my example dataset in Github)

objdet_dataset.zip|- img1.jpg|- img1.xml|- img2.jpg|- img2.xml...

• Upload zip file in Google Drive, get Drive file id, and substitute the DATASET_DRIVEID value
• Run cell that iterates over the xml files and creates annotations.csv file

Note: you can see my answer on Stackoverflow to get the Drive file id.

Model training

Model training is the core of the notebook. Fizyr offers various parameters, described in Github, to run and optimize this step.

It’s a good option to start from a pretrained model instead of training a model from scratch. Fizyr released a model based on ResNet50 architecture, pretrained on Coco dataset.

URL_MODEL = 'https://github.com/fizyr/keras-retinanet/releases/download/0.5.0/resnet50_coco_best_v2.1.0.h5'

We can even use our pretrained model, and continue the training from it. This option is particularly useful to train for some epochs, so save it in Google Drive, and later restart the training from the saved model. In this way we can bypass the 12-hour execution limit in Colab, and we can train the model for many epochs.

From my tests, a high value of batch_size and steps offers better results, but they greatly increase the execution time of each epoch.

Tensorboard training charts

We can start training from our custom dataset with:

!keras_retinanet/bin/train.py --freeze-backbone --random-transform --weights {PRETRAINED_MODEL} --batch-size 8 --steps 500 --epochs 10 csv annotations.csv classes.csv

Let’s analyze each argument passed to the script train.py.

• freeze-backbone: freeze the backbone layers, particularly useful when we use a small dataset, to avoid overfitting
• random-transform: randomly transform the dataset to get data augmentation
• weights: initialize the model with a pretrained model (your own model or one released by Fizyr)
• batch-size: training batch size, higher value gives smoother learning curve
• steps: number of steps for epochs
• epochs: number of epochs to train
• csv: annotations files generated by the script above

The training process output contains a description of layers and loss metrics during training, and as you can see, loss metrics decrease during each epoch:

Using TensorFlow backend....Layer (type)                    Output Shape         Param #     Connected toinput_1 (InputLayer)            (None, None, None, 3 0padding_conv1 (ZeroPadding2D)   (None, None, None, 3 0           input_1[0][0]                    ...Total params: 36,382,957Trainable params: 12,821,805Non-trainable params: 23,561,152NoneEpoch 1/10500/500 [==============================] - 1314s 3s/step - loss: 1.0659 - regression_loss: 0.6996 - classification_loss: 0.3663Epoch 2/10500/500 [==============================] - 1296s 3s/step - loss: 0.6747 - regression_loss: 0.5698 - classification_loss: 0.1048Epoch 3/10500/500 [==============================] - 1304s 3s/step - loss: 0.5763 - regression_loss: 0.5010 - classification_loss: 0.0753

Epoch 3/10500/500 [==============================] - 1257s 3s/step - loss: 0.5705 - regression_loss: 0.4974 - classification_loss: 0.0732

Inference

The last step performs inference of test images with the trained model.
The Fizyr framework allows us to perform inference using CPU, even if you trained the model with GPU. This feature is important in typical production environments, where people usually opt for less expensive hardware infrastructures for inference, without GPUs.

Let’s examine the following lines in detail:

model_path = os.path.join('snapshots', sorted(os.listdir('snapshots'), reverse=True)[0])print(model_path)

# load retinanet modelmodel = models.load_model(model_path, backbone_name='resnet50')model = models.convert_model(model)

The first line sets the model file as the last model generated by the training process in /snapshots directory. Then the model is loaded from the filesystem and converted to run inference.

You can change the values of THRES_SCORE, which represents the confidence threshold to show an object detection.

Object detection inference

Conclusions

We went through the complete journey to make object detection with Fizyr implementation of RetinaNet. We created a dataset, trained a model, and ran inference (here is my Github repo for the notebook and dataset).

I was impressed by the following aspects of this excellent framework:

• this framework is easy to use to get good inference, even without much customization
• it was simple to transform annotations to Fizyr’s dataset format, compared to other frameworks.

In general Fizyr is a good choice to start an object detection project, in particular if you need to quickly get good results.

If you enjoyed this article, leave a few claps, it will encourage me to explore further machine learning opportunities :)