We can use TensorFlow to train simple to complex neural networks using large sets of data.

TensorFlow is used in a variety of applications, from image and speech recognition to natural language processing and robotics. TensorFlow enables us to quickly and easily build powerful AI models with high accuracy and performance.

TensorFlow also works with GPUs and TPUs, which are types of computer chips built to extend TensorFlow’s capabilities. These chips make TensorFlow run faster, which is helpful when you have a lot of data to work with.

In this article, we will learn about tensors and how to work with tensors using TensorFlow. Let’s dive right in.

What is a Tensor?

A simple explanation would be that a tensor is a multi-dimensional array.

Scalar, Vector, Matrix and Tensor

A scalar is a single number. A vector is an array of numbers. A matrix is a 2-dimensional array. A tensor is an n-dimensional array.

In TensorFlow, everything can be considered a tensor including a scalar. A scalar would be a tensor of dimension 0, a vector of dimension 1, and a matrix of dimension 2.

Now, this is useful because we are not limited to working with complex datasets in TensorFlow. TensorFlow can handle any type of data and feed it to machine learning models.

What is TensorFlow?

TensorFlow is an open-source software library for building neural networks. Google Brain team was the one who built it and it is the most popular deep learning library in the market today.

You can use TensorFlow to build AI models including image and speech recognition, natural language processing, and predictive modeling.

Classification neural network

TensorFlow uses a dataflow graph to represent computations. To put it simply, TensorFlow has made it easy to build complex machine learning models.

TensorFlow takes care of a lot of work behind the scenes which makes it useful while building and training any type of machine learning model. TensorFlow also manages the computation, including parallelization and optimization, on the user’s behalf.

TensorFlow and Keras

Tensorflow and Keras

TensorFlow has a high-level API called Keras. Keras was a standalone project which is now available within the TensorFlow library. Keras makes it easy to define and train models while TensorFlow provides more control over the computation.

TensorFlow supports a wide range of hardware, including CPUs, GPUs, and TPUs. TPUs are Tensor processing Unites, built specifically to work with Tensors and TensorFlow.

We can also run TensorFlow on mobile devices and IoT devices using TensorFlow Lite. TensorFlow also has a large community of developers, and it is updated with new features and capabilities.

How to Build Tensors with TensorFlow

Let’s start writing some code. If you don't have TensorFlow installed, you can use a Google colab notebook to follow along.

Let’s start by importing TensorFlow and printing out the version.

import tensorflow as tf
print(tf.__version__)

OUTPUT:
2.9.2

Let’s first create a scalar using tf.constant. We use tf.constant to create a new constant value. We can also use tf.variable to create a variable value. We will then print the value and also check the dimension of the scalar using the ndim property. Its dimension will be zero because it is a single value.

scalar = tf.constant(7)
print(scalar)
print(scalar.ndim)

OUTPUT:
tf.Tensor(7, shape=(), dtype=int32)
0

Now let’s create a vector and print its dimensions. You can see that the dimension is 1.

vector = tf.constant([10,10])
print(vector)
print(vector.ndim)

OUTPUT:
tf.Tensor([10 10], shape=(2,), dtype=int32)
1

Now let’s try creating a matrix and printing its dimensions.

matrix = tf.constant([
    [10,11],
    [12,13]
])
print(matrix)
print(matrix.ndim)

OUTPUT:
tf.Tensor(
[[10 11]
 [12 13]], shape=(2, 2), dtype=int32)
2

You will see that the dimension is now 2. You can also see that the shape of the matrix is 2 by 2.

Shapes and dimensions are useful when working with TensorFlow because we will often change them while using these data to train neural networks.

We have seen that these tensors have a default datatype of int32. What if we want to create a dataset with a custom datatype?

tf.constant provides us with the dtype argument. Let’s create the same matrix again with float16 as the data type.

tensor_1 = tf.constant([
    [
        [1,2,3]
    ],
    [
        [4,5,6]
    ],
    [
        [7,8,9]
    ]
],dtype='float32')
print(tensor_1)

OUTPUT:
tf.Tensor(
[[[1. 2. 3.]]

 [[4. 5. 6.]]

 [[7. 8. 9.]]], shape=(3, 1, 3), dtype=float32)

Now let’s create a tensor. We will input a 3-dimensional array to tf.constant. We will also print its dimensions.

tensor = tf.constant([
    [
        [1,2,3]
    ],
    [
        [4,5,6]
    ],
    [
        [7,8,9]
    ]
])
print(tensor)
print(tensor.ndim)

OUTPUT:
tf.Tensor(
[[[1 2 3]]
 [[4 5 6]]
 [[7 8 9]]], shape=(3, 1, 3), dtype=int32)
3

Now we have a tensor of dimension 3 and shape 3 by 1 by 3. This is the simplest tensor you can create. In real-world scenarios, we will be dealing with tensors of higher dimensions and bigger shapes.

Now let’s look at how to create a variable tensor. We won’t be using variable tensors very often compared to constant tensors, but it is good to know that we have an option.

We will use tf.Variable to create a variable tensor. The difference between the constant tensor and variable tensor is that you can change the data in a variable tensor, but you can’t change the values in a constant tensor. Let’s create a variable tensor and print the dimensions.

var_tensor = tf.Variable([
    [
        [1,2,3]
    ],
    [
        [4,5,6]
    ],
    [
        [7,8,9]
    ]
])
print(var_tensor)

OUTPUT:
<tf.Variable 'Variable:0' shape=(3, 1, 3) dtype=int32, numpy=
array([[[1, 2, 3]],
       [[4, 5, 6]],
       [[7, 8, 9]]], dtype=int32)>

How to Generate and Load Tensors

Let’s look at how to generate tensors. In most cases, you won’t be creating tensors from scratch. You will either load a dataset, convert other datasets like NumPy arrays to tensors, or generate tensors. First, let’s look at how to generate tensors.

Let’s create a tensor with random values. There are two common ways you can do this: generate a normal distribution of data or a uniform distribution of data.

Normal distribution

The normal distribution is a bell-shaped curve that represents the distribution of data. Most of the data will be close to the average and fewer data will be away from the average. This means the probability of getting a value near the average is higher.

Uniform distribution

The uniform distribution is a straight line that represents the distribution of data. All the values in a uniform distribution will have an equal probability of occurring within a given range.

Before we generate random values, you must understand what a seed is. If we use a seed value, we can regenerate the same set of data multiple times. This will be useful when we want to test our machine-learning model against the same data after we tweak its performance.

Let’s create two arrays of random tensors. We will first set a seed and generate the random values using that seed.

seed = tf.random.Generator.from_seed(42)

Now we will create a normal and uniform distribution with the shape of 3 by 2.

normal_tensor = seed.normal(shape=(3,2))
print(normal_tensor)
uniform_tensor = seed.uniform(shape=(3,2))
print(uniform_tensor)

OUTPUT:
tf.Tensor( [[-0.7565803  -0.06854702]  [ 0.07595026 -1.2573844 ]  [-0.23193765 -1.8107855 ]], shape=(3, 2), dtype=float32)
tf.Tensor( [[0.7647915  0.03845465]  [0.8506975  0.20781887]  [0.711869   0.8843919 ]], shape=(3, 2), dtype=float32)

We have two tensors created, one with a normal distribution of random numbers and the other with a uniform distribution of random numbers.

Next, we will create a tensor with zeros and ones. In TensorFlow, tensors filled with zeros or ones are often used as a starting point for creating other tensors. They can also be placeholders for inputs in a computational graph.

To create a tensor of zeroes, use the tf.zeros function with a shape as the input argument. To create a tensor with ones, we use tf.ones with the shape as input argument.

zeros = tf.zeros(shape=(3,2))
print(zeros)
ones = tf.ones(shape=(3,2))
print(ones)

OUTPUT:
tf.Tensor(
[[0. 0.]
 [0. 0.]
 [0. 0.]], shape=(3, 2), dtype=float32)
tf.Tensor(
[[1. 1.]
 [1. 1.]
 [1. 1.]], shape=(3, 2), dtype=float32)

Now, let’s look at converting NumPy arrays into tensors. If you don’t know what NumPy is, it is a Python library for numerical computing. It helps us handle large datasets and perform a variety of computations on them.

Let’s import NumPy and create a NumPy array using NumPy’s arrange function.

import numpy as np
numpy_arr = np.arange(1,25,dtype=np.int32)

Now, we can create a tensor using the tf.constant function with the NumPy array as input. TensorFlow has built-in support to handle NumPy arrays, so it is just a matter of importing a NumPy array and setting a shape.

print(numpy_arr)
numpy_tensor = tf.constant(numpy_arr,shape=[2,4,3])
print(numpy_tensor)

OUTPUT:
[ 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24]
tf.Tensor(
[[[ 1  2  3]
  [ 4  5  6]
  [ 7  8  9]
  [10 11 12]]
 [[13 14 15]
  [16 17 18]
  [19 20 21]
  [22 23 24]]], shape=(2, 4, 3), dtype=int32)

You can see both the NumPy array as well as our tensor. The original NumPy array was 1x12 but our tensor is 2x4x3. This is called re-shaping a tensor which we will often do while training deep neural networks.

Basic Operations using Tensorflow

We have learned how tensors are created in TensorFlow. Now let’s look at some basic operations using tensors.

We will start by getting some information on our tensors. Let’s create a 4D tensor with 0 values with the shape 2x3x4x5.

rank4_tensor = tf.zeros([2,3,4,5])
print(rank4_tensor)

OUTPUT:
tf.Tensor(
[[[[0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]]
  [[0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]]
  [[0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]]]
 [[[0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]]
  [[0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]]
  [[0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]
   [0. 0. 0. 0. 0.]]]], shape=(2, 3, 4, 5), dtype=float32)

We have created our rank 4 tensor. Now let's get some information about the size, shape (number of values), and the dimension of the tensor.

We will use tf.size function to get the size. The shape and ndim properties will give us the shape and dimensions of the tensor.

print("Size",tf.size(rank4_tensor))
print("shape",rank4_tensor.shape)
print("Dimension",rank4_tensor.ndim)

OUTPUT: 

Size tf.Tensor(120, shape=(), dtype=int32)
shape (2, 3, 4, 5)
Dimension 4

Let’s look at some simple calculations using the tensor. I will create a new basic tensor.

basic_tensor = tf.constant([[10,11],[12,13]])
print(basic_tensor)

OUTPUT: 

tf.Tensor(
[[10 11]
 [12 13]], shape=(2, 2), dtype=int32)

Let’s try some simple operations. We can add, subtract, multiply, and divide every value in a tensor using the basic operators.

print(basic_tensor + 10)
print(basic_tensor - 10)
print(basic_tensor * 10)
print(basic_tensor / 10)

OUTPUT:
tf.Tensor(
[[20 21]
 [22 23]], shape=(2, 2), dtype=int32)
tf.Tensor(
[[0 1]
 [2 3]], shape=(2, 2), dtype=int32)
tf.Tensor(
[[100 110]
 [120 130]], shape=(2, 2), dtype=int32)
tf.Tensor(
[[1.  1.1]
 [1.2 1.3]], shape=(2, 2), dtype=float64)

Now let’s try matrix multiplication. I will create two simple tensors tensor_011 and tensor_012.

tensor_011 = tf.constant([[2,2],[4,4]])
tensor_012 = tf.constant([[2,3],[4,5]])

Keep in mind that in matrix multiplication, the inner dimensions should match. For example, a (3, 5) (3, 5) multiplication won’t work but (3, 5) (5, 3) will work.

The final shape of the resulting matrix will be its outer dimension. so, a 3x5 tensor multiplied by a 5x3 tensor will give us a 5x5 tensor. We will use the tf.matmul function to perform matrix multiplication.

print(tf.matmul(tensor_011,tensor_012))

OUTPUT:
tf.Tensor(
[[12 16]
 [24 32]], shape=(2, 2), dtype=int32)

Next, let’s look at reshaping and transposing a matrix. As we saw before, we will often use reshaping to change our matrix structure while training neural networks.

For example, an image pixel matrix of 28x28 will be converted into a 1-dimensional 784-pixel array for an image classification neural network.

To reshape, we use the tf.reshape function. To transpose, we use the tf.transpose function. If you don't know what a transpose is, it's converting rows into columns and columns into rows.

print(tf.reshape(tensor_011,[4,1]))
print(tf.transpose(tensor_011))

OUTPUT:
tf.Tensor(
[[2]
 [2]
 [4]
 [4]], shape=(4, 1), dtype=int32)
tf.Tensor(
[[2 4]
 [2 4]], shape=(2, 2), dtype=int32)

Finally, let’s look at some aggregate operations like min, max, standard deviation, square and square root.

To find the minimum and maximum values, we use the tf.reduce_min and tf.reduce_max functions. And to find the sum of the array, we use the tf.reduce_sum function.

tensor_013 = tf.constant([
    [1,2,3],
    [4,5,6],
    [7,8,9]
],dtype='float32')
print(tf.reduce_min(tensor_013))
print(tf.reduce_max(tensor_013))
print(tf.reduce_sum(tensor_013))

OUTPUT:
tf.Tensor(1.0, shape=(), dtype=float32)
tf.Tensor(9.0, shape=(), dtype=float32)
tf.Tensor(45.0, shape=(), dtype=float32)

Now for the standard deviation and variance, we use the tf.math.reduce_std function and tf.math.reduce_variance function.

print(tf.math.reduce_std(tensor_013))
print(tf.math.reduce_variance(tensor_013))

OUTPUT:
tf.Tensor(2.5819888, shape=(), dtype=float32)
tf.Tensor(6.6666665, shape=(), dtype=float32)

Let’s find the square, square root, and log of each value in a tensor.

print(tf.sqrt(tensor_013))
print(tf.square(tensor_013))
print(tf.math.log(tensor_013))

OUTPUT:
tf.Tensor(
[[1.        1.4142135 1.7320508]
 [2.        2.236068  2.4494898]
 [2.6457512 2.828427  3.       ]], shape=(3, 3), dtype=float32)
tf.Tensor(
[[ 1.  4.  9.]
 [16. 25. 36.]
 [49. 64. 81.]], shape=(3, 3), dtype=float32)
tf.Tensor(
[[0.        0.6931472 1.0986123]
 [1.3862944 1.609438  1.7917595]
 [1.9459102 2.0794415 2.1972246]], shape=(3, 3), dtype=float32)

We have learned the basics of TensorFlow in this article. You are now equipped to work with TensorFlow and use it to model data.

If you want to start using this knowledge and build a project, you can check out my course on building a handwriting recognition neural network using TensorFlow. You can also learn advanced TensorFlow concepts using the official documentation.

Conclusion

Tensorflow is a powerful library to build deep-learning models. It has all the tools we need to construct neural networks to solve problems like image classification, sentiment analysis, stock market predictions, etc.

With the advent of technologies like ChatGPT, learning TensorFlow will give you a head start in the current job market.

Hope you liked this article. You can learn more about me and my articles/videos at manishmshiva.com.

PyTorch is an open-source Python-based library. It provides high flexibility and speed while building, training, and deploying deep learning models.

At its core, PyTorch involves operations involving tensors. A tensor is a number, vector, matrix, or any n-dimensional array.

In this article, we will see different ways of creating tensors using PyTorch tensor methods (functions).

Topics we'll cover

• tensor
• zeros
• ones
• full
• arange
• linspace
• rand
• randint
• eye
• complex

The tensor() method

This method returns a tensor when data is passed to it. data can be a scalar, tuple, a list or a NumPy array.

In the above example, a NumPy array that was created using np.arange() was passed to the tensor() method, resulting in a 1-D tensor.

We can create a multi-dimensional tensor by passing a tuple of tuples, a list of lists, or a multi-dimensional NumPy array.

When an empty tuple or list is passed into tensor(), it creates an empty tensor.

The zeros() method

This method returns a tensor where all elements are zeros, of specified size (shape). The size can be given as a tuple or a list or neither.

We could have passed 3, 2 inside a tuple or a list as well. It is self-explainable that passing negative numbers or a float would result in a run time error.

Passing an empty tuple or an empty list gives a tensor of size (dimension) 0, having 0 as its only element, whose data type is float.

The ones() method

Similar to zeros(), ones() returns a tensor where all elements are 1, of specified size (shape). The size can be given as a tuple or a list or neither.

Like zeros(), passing an empty tuple or list gives a tensor of 0 dimension, having 1 as the sole element, whose data type is float.

The full() method

What if you want all the elements of a tensor to be equal to some value but not only 0 and 1? Maybe 2.9?

full() returns a tensor of a shape given by the size argument, with all its elements equal to the fill_value.

Here, we have created a tensor of shape 3, 2 with the fill_value as 3. Here again, passing an empty tuple or list creates a scalar tensor of zero dimension.

While using full, it is necessary to give size as a tuple or a list.

The arange() method

This method returns a 1-D tensor, with elements from start (inclusive) to end (exclusive) with a common difference step. The default value for start is 0 while that for step is 1.

The elements of the tensor can be said to be in Arithmetic Progression, with step as common difference.

Here, we created a tensor which starts from 2 and goes until 20 with a step (common difference) of 2.

All the three parameters, start, end and step can be positive, negative or float.

While choosing start, end, and step, we need to ensure that start and end are consistent with the step sign.

Since step is set as -2, there is no way -42 can reach -22 (exclusive). Hence, it gives an error.

The linspace() method

This method returns a 1-D dimensional tensor, with elements from start (inclusive) to end (inclusive). However, unlike arange(), here, steps isn't the common difference but the number of elements to be in the tensor.

PyTorch automatically decides the common difference based on the steps given.

Not providing a value for steps is deprecated. For backwards compatibility, not providing a value for steps creates a tensor with 100 elements. According to the official documentation, in a future PyTorch release, failing to provide a value for steps will throw a runtime error.

Unlike arange(), linspace can have a start greater than end since the common difference is automatically calculated.

Since steps here is not a common difference, but the number of elements, it can only be a non-negative integer.

The rand() method

This method returns a tensor filled with random numbers from a uniform distribution on the interval 0 (inclusive) to 1 (exclusive). The shape is given by the size argument. The size argument can be given as a tuple or list or neither.

Passing an empty tuple or list creates a scalar tensor of zero dimension.

The randint() method

This method returns a tensor filled with random integers generated uniformly between low (inclusive) and high (exclusive). The shape is given by the size argument. The default value for low is 0.

When only one int argument is passed, low gets the value 0, by default, and high gets the passed value.

The size argument only takes a tuple or a list. An empty tuple or list creates a tensor with zero dimension.

The eye() method

This method returns a 2-D tensor with ones on the diagonal and zeros elsewhere. The number of rows is given by n and columns is given by m.

The default value for m is the value of n. When only n is passed, it creates a tensor in the form of an identity matrix. An identity matrix has its diagonal elements as 1 and all others as 0.

The complex() method

This method returns a complex tensor with its real part equal to real and its imaginary part equal to imag. Both real and imag are tensors.

The data type of both the real and imag tensors should be either float or double.

Also, the size of both tensors, real and imag, should be the same, since the corresponding elements of the two matrices form a complex number.

Conclusion

We've covered ten different ways to create tensors using PyTorch methods. You can go through the official documentation to know more about other PyTorch methods.

You can click here to go to the Jupyter notebook where you can play around with these methods.

If you want to learn more about PyTorch, check out this amazing course on freeCodeCamp's YouTube channel.

Stay safe!

Table of Contents

• What this is about
• The reference code base
• 1. Fetch and print values within Session.run
• 2. Use the tf.Print operation
• 3. Use Tensorboard visualization for monitoring
• a) clean the graph with proper names and name scopes
• b) Add tf.summaries
• c) Add a tf.summary.FileWriter to create log files
• d) Start the tensorboard server from your terminal
• 4. Use the Tensorboard debugger
• 5. Use the TensorFlow debugger
• Conclusion

What this is about

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. — BRIAN W. KERNIGHAN

Debugging in general can be a tedious and challenging task. Nevertheless, you must be comfortable going through the written code and identifying problems. Normally there are many guides, and the process of debugging is often well documented for many languages and frameworks.

When it comes to TensorFlow, however, some new challenges arise because of the way it works.

As the official documentation states:

A TensorFlow Core program consists of two discrete sections:

1. Building the computational graph (a tf.Graph).
2. Running the computational graph (using a tf.Session).

_Source and credit to [https://www.tensorflow.org/guide/graphs](https://www.tensorflow.org/guide/graphs" rel="noopener" target="blank" title=")

The actual computation is done with session.run(), which means that we need to find a way to inspect values inside this function.

The reference code base

As a reference, I will provide my Github repository with the corresponding code here.

We will use a basic neural network to classify handwritten digits from the MNIST dataset, using:

• tf.nn.softmax_cross_entropy_with_logits_v2 as the TF classification operation for defining the loss
• tf.train.GradientDescentOptimizer for minimizing the loss

Running this small neural network shows that it can already achieve an accuracy of ~92%:

The process of debugging

Now for debugging, there are basically 5 (pragmatic) ways to achieve this.

As a side note: It is often useful to assert shapes to ensure that everything works together as intended.

1. Fetch and print values within Session.run

This is probably the fastest and easiest way to get the information you need.

• easy and fast
• any evaluation can be fetched from everywhere
• it’s necessary to hold the reference to the tensor, which is bad in complex models

In essence, you run the session in a print statement and feed it the dictionary, like so: print( f"The bias parameter is: {sess.run(b, feed_dict={x: mnist.test.images, y_: mnist.test.labels})}" )

If the code gets more complex, the partial_run execution of a session could be used. But since this is an experimental feature I will not implement this for demonstration.

Additionally, don’t forget the [.eval()](https://www.tensorflow.org/api_docs/python/tf/Tensor#eval) method for evaluating tensors specifically.

See the full code here on Github.

2. Use the tf.Print operation

The tf.Print method comes in handy during run-time evaluation when we don’t want to explicitly fetch the code with session.run(). It is an identity op that prints data when evaluating.

• it allows us to see the development of values during evaluation
• it has limited configuration and therefore can easily clog the terminal

Yufeng G created a fantastic video and article about how to use the tf.Print statement. And as he points out, it is vital to structure the print node the way that it is used further. As he says:

It is vitally important that you actually use this returned node, because if you don’t, it will be dangling.

In my code, I added a print statement that fetches the values within the session to illustrate how both methods perform differently in execution.

With runtime evaluation comes the possibility of runtime assertion with tf.Assert .

See the full code here.

3. Use Tensorboard visualization for monitoring

Before diving into this debugging method, be aware that there is the Tensorboard and the Tensorboard debugger!

The TF website offers a great tutorial for implementing and using the board.

A key for the usage is the serializing of the data. TensorFlow provides the summary operations, which allow you to export condensed information about the model. They are like anchors telling the visualization board what to plot.

a) Clean the graph with proper names and name scopes

First we need to organize all the variables and operations with the [scope](https://www.tensorflow.org/guide/graph_viz#name_scoping_and_nodes) methods that TF provides.

with tf.name_scope("variables_scope"):    x = tf.placeholder(tf.float32, shape=[None, 784], name="x_placeholder")    y_ = tf.placeholder(tf.float32, shape=[None, 10], name="y_placeholder")

b) Add tf.summaries

For example:

with tf.name_scope("weights_scope"):    W = tf.Variable(tf.zeros([784, 10]), name="weights_variable")    tf.summary.histogram("weight_histogram", W)

c) Add a tf.summary.FileWriter to create log files

Tip: Make sure to create sub folders for each log to avoid accumulation of graphs.

d) Start the tensorboard server from your terminal

For example: tensorboard --logdir=./tfb_logs/ --port=8090 --host=127.0.0.1

Navigating to the tensorboard server (in this case http://127.0.0.1:8090) shows the following:

The distributions tab of tensorboard

Now the full power and use of tensorboard becomes clear. It allows you very easily to spot errors in your machine learning model. My code example is a very simple one. Imagine a model with multiple layers and more variables and operations!

See full code here on Github.

4. Use the Tensorboard debugger

As the Tensorboard Github repository states:

This dashboard is in its alpha release. Some features are not yet fully functional.

However, it can still be used and provides cool debugging features. Please check out the Github repository to get an adequate overview. Also, see their video to get a deeper understanding. They have done a great job.

To accomplish this, there are 3 things to add to our previous example:

1. Import from tensorflow.python import debug as tf_debug
2. Add your session with tf_debug.TensorBoardDebugWrapsperSession
3. Add to your tensorboard server the debugger_port

Now you have the option to debug the whole visualized model like with any other debugger, but with a beautiful map. You are able to select certain nodes and inspect them, control execution with the “step” and “continue” buttons, and visualize tensors and their values.

The debugger feature of tensorboard in action

There is much more to talk about regarding this unique feature of Tensorflow, but I will probably dedicate another article to that.

See my full code here on Github.

5. Use the TensorFlow debugger

The last method, but also very powerful, is the CLI TensorFlow debugger.

This debugger focuses on the command-line interface (CLI) of tfdbg, as opposed to the graphical user interface (GUI) of tfdbg, that is the TensorBoard Debugger Plugin.

You simply wrap the session with tf_debug.LocalCLIDebugWrapperSession(sess) and then you start the debugging with executing the file (maybe it's necessary to add the --debug flag).

It basically allows you to run and step through the execution of your model, while providing evaluation metrics.

I think the official documention could be improved, but they have also created a video which introduces the feature in a good way.

So the key features here are the commands invoke_stepper and then pressing s to step through each operation. It is the basic debugger functionality of a debugger but in the CLI. It looks like this:

Evaluation metrics while debugging with the CLI

See the full code here on Github.

Conclusion

As shown, there are many ways to debug a TensorFlow application. Each method has its own strengths and weaknesses. I didn’t mention the Python debugger, because it is not TensorFlow specific, but keep in mind that the simple Python debugger already provides some good insights!

There is a great presentation by Wookayin who talks about these concepts as well but also goes over some general debugging advise. That advice is:

• name tensors properly
• check and sanitize input
• logging
• assertions
• proper use of exceptions
• failing fast -> immediately abort if something is wrong
• don’t repeat yourself
• organize your modules and code

I am really excited for all the features that TensorFlow has to offer for people who are building machine learning systems. They are doing a great job! Looking forward to further developments! :)

Thanks for reading my article! Feel free to leave any feedback!

Daniel is a LL.M. student in business law, working as a software engineer and organizer of tech related events in Vienna. His current personal learning efforts focus on machine learning.

Connect on:

• LinkedIn
• Github
• Medium
• Twitter
• Steemit
• Hashnode